Biodiversity-based agriculture is the main form of agriculture practiced by smallholder farmers, who produce half the world's food, especially in the Global South. This form of agriculture relies on planned biodiversity intentionally managed by farmers and on the associated biodiversity that spontaneously colonizes the agroecosystem. In recent decades, there have been increasing calls from researchers and society to support biodiversity-based agriculture as an alternative paradigm to today's industrial agriculture. Building adapted governance and management systems for enhancing farmers' access to agrobiodiversity is a key challenge for the development of biodiversity-based agriculture. To achieve this, a better understanding of how farmer's access agrobiodiversity is needed, and in particular, how this access is affected by interactions between farmers and with institutions, i.e., social networks. In this article, we first review the literature on the role of social networks in farmers' access to agrobiodiversity, in the form of crop diversity and associated biodiversity, and the related knowledge to manage this diversity. This review points at a major knowledge gap concerning how the composition and structure of these networks affect farmers' access to agrobiodiversity. Then, we review literature on social-ecological networks to identify how this framework developed for environmental management could contribute in getting a better understanding of the role of social networks' structure and composition in farmers' access to agrobiodiversity. Based on this review, we propose a social-ecological network framework dedicated to crop diversity. Finally, we present potential applications of this framework to develop new participatory approaches for agrobiodiversity management and governance, adapted to biodiversity-based agriculture.