Understanding the mechanisms involved in pathogen adaptation to quantitative resistance in plants has a key role to play in establishing durable strategies for resistance deployment, especially in perennial crops. The erosion of quantitative resistance has been recently suspected in Cuba and the Dominican Republic for a major fungal pathogen of such a crop: Pseudocercospora fijiensis, causing black leaf streak disease on banana. This study set out to test whether such erosion has resulted from an adaptation of P. fijiensis populations, and to determine whether or not the adaptation is local. Almost 600 P. fijiensis isolates from Cuba and the Dominican Republic were sampled using a paired-population sampling design on resistant and susceptible banana varieties. A low genetic structure of the P. fijiensis populations was detected in each country using 16 microsatellite markers. Cross-inoculation experiments using isolates from susceptible and resistant cultivars were carried out, measuring a quantitative trait (the diseased leaf area) related to pathogen fitness on three varieties. A further analysis based on those data suggested the existence of a local pattern of adaptation to resistant cultivars in both of the study countries, due to the existence of specific (or genotype by genotype) host–pathogen interactions. However, neither cost nor benefit effects for adapted populations were found on the widely used “Cavendish” banana group. These results highlight the need to study specific host–pathogen interactions and pathogen adaptation on a wide range of quantitative resistance phenotypes in banana, in order to develop durable strategies for resistance deployment.