Unlike most of the important food crops, sesame can survive drought but severe and repeated drought episodes, especially occurring during the reproductive stage, significantly curtail the productivity of this high oil crop. Genome-wide association study was conducted for traits related to drought tolerance using 400 diverse sesame accessions, including landraces and modern cultivars. Ten stable QTLs explaining more than 40% of the phenotypic variation and located on four linkage groups were significantly associated with drought tolerance related traits. Accessions from the tropical area harboured higher numbers of drought tolerance alleles at the peak loci and were found to be more tolerant than those from the northern area, indicating a long-term genetic adaptation to drought-prone environments. We found that sesame has already fixed important alleles conferring survival to drought which may explain its relative high drought tolerance. However, most of the alleles crucial for productivity and yield maintenance under drought conditions are far from been fixed. This study also revealed that pyramiding the favourable alleles observed at the peak loci is of high potential for enhancing drought tolerance in sesame. In addition, our results highlighted two important pleiotropic QTLs harbouring known and unreported drought tolerance genes such as SiABI4, SiTTM3, SiGOLS1, SiNIMIN1 and SiSAM. By integrating candidate gene association study, gene expression and transgenic experiments, we demonstrated that SiSAM confers drought tolerance by modulating polyamine levels and ROS homeostasis, and a missense mutation in the coding region partly contributes to the natural variation of drought tolerance in sesame.