Caffeine use is widespread among athletes following its removal from the World Anti-Doping Agency banned list, with approximately 75% of competitive athletes using caffeine. While literature supports that caffeine has a small positive ergogenic effect for most forms of sports and exercise, there exists a significant amount of inter-individual difference in the response to caffeine ingestion and the subsequent effect on exercise performance. In this narrative review, we discuss some of the potential mechanisms and focus on the role that genetics has in these differences. CYP1A2 and ADORA2A are two of the genes which are thought to have the largest impact on the ergogenicity of caffeine. CYP1A2 is responsible for the majority of the metabolism of caffeine, and ADORA2A has been linked to caffeine-induced anxiety. The effects of CYP1A2 and ADORA2A genes on responses to caffeine will be discussed in detail and an overview of the current literature will be presented. The role of these two genes may explain a large portion of the inter-individual variance reported by studies following caffeine ingestion. Elucidating the extent to which these genes moderate responses to caffeine during exercise will ensure caffeine supplementation programs can be tailored to individual athletes in order to maximize the potential ergogenic effect.