Cultivated peanut is an allotetraploid (2n = 4× = 40) with narrow genetic diversity. In previous studies, we developed an advanced backcross quantitative trait loci (AB-QTL) population from the cross between the synthetic allotetraploid ((Arachis ipaensis × Arachis duranensis)4×) and the cultivated variety Fleur11, and mapped several quantitative trait loci (QTLs) involved in yield and yield components. We also developed a chromosome segment substitution line (CSSL) population as a way to mendelize the QTLs and analyzing their effects. In this study, 16 CSSLs were used for assessing the contribution of wild alleles in yield performance and stability across environments, as well as validating QTLs for pod and seed size. The CSSLs and the recurrent parent Fleur11, used as a check, were assessed using an alpha lattice design in three locations during two consecutive rainy seasons in Senegal, totaling six environments. Our results showed that the chromosome segments from the wild species, in general, have no yield disadvantage and contributed positive variation to yield-related traits. Most of the QTLs detected for pod and seed size in the AB-QTL on linkage groups A07, A08, A09, and B06 were also found in the CSSLs, showing that the CSSLs used in this study are accurate material for QTL validation. Several new QTLs have also been identified. Two CSSLs (12CS_031 and 12CS_069) showed consistently higher pod and seed size than Fleur11 in all environments, suggesting that the QTLs were consistent and stable. Our study opens the way for pyramiding these QTLs for peanut improvement.