Conservation agriculture (CA) has become a dominant paradigm in scientific and policy thinking about the sustainable intensification of food production in sub-Saharan Africa. Yet claims that CA leads to increasing crop yields in African smallholder farming systems remain controversial. Through a meta-analysis of 933 observations from 16 different countries in sub-Saharan African studies, we show that average yields under CA are only slightly higher than those of conventional tillage systems (3.7% for six major crop species and 4.0% for maize). Larger yield responses for maize result from mulching and crop rotations/intercropping. When CA principles are implemented concomitantly, maize yield increases by 8.4%. The largest yield benefits from CA occur in combination with low rainfall and herbicides. We conclude that although CA may bring soil conservation benefits, it is not a technology for African smallholder farmers to overcome low crop productivity and food insecurity in the short term.