Plant breeding has always sought to develop crops able to withstand environmental stresses, but this is all the more urgent now as climate change is affecting the agricultural regions of the world. It is currently difficult to screen genetic material to determine how well a crop will tolerate various stresses. Multi-environment trials (MET) which include a particular stress condition could be used to train a genomic selection model thanks to molecular marker information that is now readily available. Our study focuses on understanding how and predicting whether a plant is adapted to a particular environmental stress. We propose a way to use genomic random regression, an extension of factorial regression, to model the reaction norms of a genotype to an environmental stress: the factorial regression genomic best linear unbiased predictor (FR-gBLUP). Twenty-eight wheat trials in France (3 years, 12 locations, nitrogen or water stress treatments) were split into two METs where different stresses limited grain number and yield. In MET1, drought at flowering was responsible for 46.7% of the genotype-by-environment (G × E) interactions for yield while in MET2, heat stress during booting was identified as the main factor responsible for G × E interactions, but that explained less of the interaction variance (33.6%). Since drought at flowering explained a fairly large variance in G × E in MET1, the FR-gBLUP model was more accurate than the additive gBLUP across all types of cross validation. Accuracy gains varied from 2.4% to 12.9% for the genomic regression to drought. In MET2 accuracy gains were modest, varying from -5.7% to 2.4%. When a major stress influencing G × E is identified, the FR-gBLUP strategy makes it possible to predict the level of adaptation of genotyped individuals to varying stress intensities, and thus to select them in silico. Our study demonstrates how genome-wide selection can facilitate breeding for adaptation.