In the last decade, image analysis through machine learning algorithms proved to be an effective tool for plant organs detection and counting. Numerous studies provide fruit tree yield estimates based on machine learning with high levels of efficiency. However, most studies were conducted under homogeneous conditions of fruit appearance and light. The aim of this study was to develop a highly efficient machine learning method for ripe mango fruit detection from color images and apply it in heterogeneous conditions for in situ fruit yield estimation in Senegal. The algorithm consisted in a k-nearest neighbors (KNN) classification based on color and texture indices (3 indices selected among 20 using a ranking feature method) followed by a post-treatment based on shape indices. The F1 score, which considers precision and recall performances, was 0.72 for a homogeneous set of about 3000 visible fruits over 62 images of Kent mango trees with similar visual appearance under steady light conditions. The algorithm was then calibrated and used on a second set of 600 images of 300 trees representing the in situ heterogeneity in tree structure (height, canopy volume) and variety (Kent, Keitt and Boucodiékhal) found in Niayes region of Senegal. The F1 score was then 0.45. Even if the algorithm performed less under these conditions, a significant linear relationship was evidenced between the number of detected fruits and the actual number of fruits (manually counted for a sample of 52 trees) for each of the 3 varieties (Kent: R² = 0.92; Keitt: R² = 0.93 and Boucodiékhal: R² = 0.90). These models were used to estimate the actual fruit yields of the 300 trees, thereby leading to an average number of fruit per tree of 164.5. Those results will serve to identify factors that drive mango yield variability at the tree scale in Niayes region.