Deep learning techniques and computer vision systems offer effective fruit counting solutions for farm yield estimation. However, the performance of these solutions drops when identifying different cultivars of the same fruit species. This study clarified the differences between mango fruit detection and mango cultivar identification. An original double-threshold-based classification method for fruit cultivar identification, with estimation of the misidentification rate was proposed in order to significantly increase the performance of a specialised mango fruit detection method known as Faster R-CNN. This method was applied on images of mango trees of three cultivars taken in Senegalese orchards of different existing cropping systems, with varying tree features, planting patterns and acquisition contexts. Analysis of the results focused on the contributions of fruit detection errors and fruit cultivar confusion to the overall error of the network for fruit counts by cultivar class. The shift from fruit detection to cultivar identification resulted in a drop in the average prediction rate from 92 to 68%. With its explicitly independent fruit detection and cultivar identification steps, the double-threshold-based classification method increased the prediction rate to 86%, with a maximum identification error of 0.05%. This setting also led to relative equality between the recall and the precision of each cultivar class, making the network well suited for fruit counting by cultivar class. This work opened new perspectives for decision support tools for fruit growers that could provide more appropriate yield estimates per cultivar.