Memory for people and their relationships, along with memory for social language and social behaviors, constitutes a specific type of semantic memory termed social knowledge. This review focuses on how and where social knowledge is represented in the brain. We propose that portions of the anterior temporal lobe (ATL) play a critical role in representing and retrieving social knowledge. This includes memory about people, their names and biographies and more abstract forms of social memory such as memory for traits and social concepts. This hypothesis is based on the convergence of several lines of research including anatomical findings, lesion evidence from both humans and non-human primates and neuroimaging evidence. Moreover, the ATL is closely interconnected with cortical nuclei of the amygdala and orbitofrontal cortex via the uncinate fasciculus. We propose that this pattern of connectivity underlies the function of the ATL in encoding and storing emotionally tagged knowledge that is used to guide orbitofrontal-based decision processes.