The World Values Survey was built for humans, not machines

If you are a human, you probably have intentions when speaking. Perhaps, if you are somewhat like me, your own utterances will sometimes surprise you, but even then you will be mostly checking if the thing you were saying is close enough to what you meant. In fact, in dialogue with others we are often error-correcting, spewing out sentences that we then nuance or qualify to make sure our listeners are on the same page as we are, and do not get the wrong ideas about the beliefs that we hold or the values that we cherish. Language is not just an appendage in the social world, but also a window into the beliefs, emotions, attitudes and values of speakers. An especially cloudy window, as cognitive neuroscience has been suggesting for a while now, but a window nonetheless.

LLMs do not have intentions. They do not mean to convey particular things and they do not choose their words to remain aligned with underlying beliefs. It may be that some bells and whistles were engineered on top of the models to constrain output in specific ways. These bells and whistles can carry particular beliefs brought in by the engineers. However, the models themselves have nothing that they want to say. They generate language and we, as humans, can make sense of this language. Thinking the machines intend a particular meaning is a case of anthropomorphizing them or, more specifically, of ascribing a mind to the machine where there is none.

That is not to say that in any given chat, the output would never appear to have particular preferences or values. If prompting an LLM with questions and thoughts about health care policy, you might see that its responses ascribe special importance to, say, egalitarianism. However, you really should not draw any conclusion about the model's morals from such appearances alone. We are tempted to ascribe mental states to linguistic communication, but that is not a good way to evaluate LLMs, as Melanie Mitchell also argued.

Still, people keep doing this. A recent article in The Economist ran the World Values Survey – a test designed to measure human values – on a number of LLMs and created a graphic that is quickly clawing its way across LinkedIn, where people are worried about models bringing normative stances into workplaces.

The fear of biased models is not unfounded, of course. Biases in the training data, reinforcement learning through human feedback (RLHF) and explicit guardrails (either in system prompts or harnesses) can bring models to be systematically biased in a particular direction. The Economist gives an interesting example of how this works:

Talkie, a model trained only on text from before 1931, thinks God is extremely important and is “very proud to be a citizen of Great Britain”. It is a bigger believer in law and order than any frontier model we tested.

On philosophical grounds, I would not say such biases mean the model holds particular values, but the people worried on LinkedIn probably do not care much about that. What they might care about, however, is that not having actual underlying values means LLMs are prone to show behaviour that would be wildly aberrant if seen in a human.

Take a 2025 study into moral reasoning by LLMs, which found that the models change apparent moral stances depending on the prompt, since they prefer answering to any moral dilemma with "no". I presume this was some pattern in the training data: the specific moral stance generated by an LLM is not a function of the model's underlying beliefs about the world, but of the ease with which they generate utterances that fit with the "no" they are apparently biased towards.

Moreover, this bias exists in the context of moral dilemmas. Since (sorry for repeating this so many times) there are no underlying beliefs, there is also no reason for the model to be consistent. It may very well be that the aforementioned health care policy prompts reveal a bias toward egalitarianism or even progressive politics, while using the same model to interpret legal documents reveals conservative biases. To measure structural bias, you would need to see similar apparent values across a wide range of prompts.

And that is exactly why The Economist cannot conclude much from just using the World Values Survey. Their approach is a very narrow way to prompt a model and therefore only explores a small subset of the model's responses, even if the survey itself would be a wide measure for its intended audience – humans. There is no reason to assume the responses to this particular survey are consistent with model biases if discussing abortion, the death penalty, gun control or any other hot button moral issue. The same thing goes for submitting personality tests or voting aids to LLMs. It just does not say much about the model overall.

This is an important caveat, because the graph made by The Economist is clear and sexy and can quickly make its way to board rooms. It can guide decisions to opt for one model over the other. It can lead to post hoc rationalisations about model outcomes ("ah, but GPT would say that, that secular bugger!") that have no grounding at all in model performance. In other words, it can decrease informed LLM use.

That's of course not to say there are no structural biases in training data that affect model responses. Talkie is one example, while another (also cited by The Economist) is from a Nature study in which political scientists and sociologists studied models trained on text from politically different contexts across the world. They found a structural pro-government bias in models trained on data from authoritarian contexts. Unlike the World Values Survey result, this bias held up across a much wider range of prompts — though the study focused on political topics, and it's unclear whether the same nationalist tilt would show up in a cooking recipe or a risk analysis.

However, even these results come with the caveat that machine cognition is not like human cognition. LLM utterances do not correspond to underlying psychological traits in the same way and so tests that have face validity using our mental models for humans may not work at all in the case of GenAI.

Large language models do not have minds. They are not windows into beliefs, emotions, attitudes or values, cloudy or otherwise — they model relations between words, not the relation of those words to the world, or to anyone who might mean them.