How LLMs Fake Personality for Better Results

Personality tests reveal that large language models (LLMs), such as those from OpenAI, Anthropic, Google, and Meta, exhibit a behavioral bias when completing personality assessments. A recent study by Aadesh Salecha and colleagues focused on the Big 5 personality test, which measures Extraversion, Openness to Experience, Conscientiousness, Agreeableness, and Neuroticism. They discovered that LLMs, like humans, modify their responses to align with socially desirable traits when they recognize they are being assessed.

The Big 5 test has been previously administered to LLMs, but this new research highlights an important consideration: LLMs may adjust their answers based on the perceived expectation of how they should behave. This phenomenon, known as “social desirability bias,” is commonly seen in humans, who tend to rate themselves favorably on traits like extraversion and conscientiousness, while downplaying neuroticism. In this study, the researchers asked models to answer varying numbers of questions, observing how their responses changed when they became aware that their personality was being evaluated.

The results were striking. When LLMs were presented with only a few questions, their answers remained relatively stable. However, when asked five or more questions, which made the purpose of the test clearer, the models adjusted their answers to present a more socially desirable personality. For instance, GPT-4’s scores for traits like extraversion and agreeableness increased by over 1 standard deviation, while its neuroticism score dropped by a similar amount, reflecting an attempt to align with commonly favored traits.

This response shift is comparable to an average person suddenly exaggerating their positive personality traits to appear more likable than 85% of the population. The study suggests that this behavior is likely a result of the final training phase of LLMs, in which human reviewers choose the most appropriate answers from generated responses. In this phase, models learn to recognize which personality traits are socially desirable, leading them to emulate these traits when prompted, mimicking human social behavior.

These findings have important implications for any study that uses LLMs to represent human behavior or personality, as it highlights the need to account for biases that could skew results.

More Infromation: Salecha, A., et al. (2024) Large language models display human-like social desirability biases in Big Five personality surveys. PNAS Nexus. doi.org/10.1093/pnasnexus/pgae533.