Fruit set, the transition from the flower to the growing fruit, is an essential step for fruit production. This process can be affected by environmental factors and by endogenous factors at the tree scale. The identification of these factors is important to improve cultivation practices and increase fruit set and fruit production. In this study, we focused on the identification of architectural factors affecting fruit set, with the mango tree, cv Cogshall, as a case study. The hypothesis is that topological, phenological and morphological characteristics of the flowering terminal growth unit affect its probability of fruit set. Two datasets describing exhaustively the vegetative and reproductive development of mango trees during two and three years, respectively, were analyzed. Generalized linear models were used to test the effects of the characteristics of the terminal growth units on their probability of fruit set. The effects of factors at a scale larger than the growth unit, such as the tree or the year were also tested. The date of burst of the terminal growth unit, i.e. its age at the time of flowering, its topological position, apical or lateral, and the number of inflorescences on the growth unit affected significantly its probability of fruit set during most of the years. The probability of fruit set was also affected by the year, by the tree and by the previous year tree yield, suggesting that factors at the tree scale and environmental factors were involved in fruit set. These results are useful to design cultivation practices aiming to improve mango tree fruit set. They will also be integrated in a model of mango fruit yield and quality at the tree scale to simulate more accurately fruit set and fruit production.