A growing body of evidence has shown that people with chronic low back pain (CLBP) demonstrate significantly greater declines in multiple cognitive domains than people who do not have CLBP. Given the high prevalence of CLBP in the ever-growing aging population that may be more vulnerable to cognitive decline, it is important to understand the mechanisms underlying the accelerated cognitive decline observed in this population, so that proper preventive or treatment approaches can be developed and implemented. The current scoping review summarizes what is known regarding the potential mechanisms underlying suboptimal cognitive performance and cognitive decline in people with CLBP and discusses future research directions. Five potential mechanisms were identified based on the findings from 34 included studies: (1) altered activity in the cortex and neural networks; (2) grey matter atrophy; (3) microglial activation and neuroinflammation; (4) comorbidities associated with CLBP; and (5) gut microbiota dysbiosis. Future studies should deepen the understanding of mechanisms underlying this association so that proper prevention and treatment strategies can be developed.