Genomic offset statistics predict the maladaptation of populations to rapid habitat alteration based on association of genotypes with environmental variation. Despite substantial evidence for empirical validity, genomic offset statistics have well-identified limitations, and lack a theory that would facilitate interpretations of predicted values. Here, we clarified the theoretical relationships between genomic offset statistics and unobserved fitness traits controlled by environmentally selected loci and proposed a geometric measure to predict fitness after rapid change in local environment. The predictions of our theory were verified in computer simulations and in empirical data on African pearl millet (Cenchrus americanus) obtained from a common garden experiment. Our results proposed a unified perspective on genomic offset statistics and provided a theoretical foundation necessary when considering their potential application in conservation management in the face of environmental change.