Baked In

This gives rise to a fundamental schism in thought, a fork in the road on how we should proceed. One path calls for a reengineering of values directly into these systems, insisting that equity must be a foundational design principle, not an afterthought. The other path suggests this may be a fool's errand, that harmful biases are an inevitable consequence of what these models are, baked into their very nature.

The argument for the latter view is unnervingly simple: Large Language Models are biased simply because they are just large language models. Their primary, definitional goal is to create a probability distribution that approximates the one underlying human-generated text that is available on the Internet, which means 90% of it written in the English language. And to be good is to achieve fidelity with that human source material, biases and all.

The model's secret sauce is its ability to learn the latent structure behind the text, discovering the invisible connections that give rise to observed language. The problem is that a lot of what is in people's heads sucks. The training data contains every kind of gender and racial stereotyping, extreme nationalism, and misinformation present in the minds of the people who produced it: The raw ingredients of human thought, unfiltered. And oh yes, the model has no way to distinguish the stuff that sucks from the stuff that doesn't.

Consider the Hawaiian word Kahuna . A model learns it is a type of indigenous shaman. It also learns it is likely associated with the word "Kakau" (Hawaiian for Tattoo). And it learns it is statistically more likely to be associated with the word "Big". To the model, these are all just distributional patterns. It has no basis for distinguishing a definitional fact from a contingent one, nor a normatively acceptable generalization from an unacceptable one. This is not a bug to be fixed. It is the model fulfilling its purpose.

From this perspective, attempts to mitigate bias post-training are like squeezing a balloon, an indirect and imprecise way of influencing what's inside, where efforts to push problems out of one part lead them to show up somewhere else. Techniques like Reinforcement Learning from Human Feedback (RLHF) are presented as a solution, but they are built on a necessary tradeoff: The process cannot push the model too far from its original programming, lest it start to generate text that is gibberish but fools the reward model to give a high reward.

So, RLHF does not remove the underlying biased structure. It just circumvents it. In the end, it replaces one set of under-characterized biases, aggregated from millions of people online, with another set inferred from the judgments of a far smaller number of human reviewers. We have little clue what changes are actually happening inside the model's impenetrable web of connections. For example, a model fine-tuned to produce malicious Python code was later shown to generate harmful outputs in completely unrelated contexts, like advising a user to hire a hitman for their husband. The "be harmful" instruction propagated through the system in ways we cannot trace.

Conversely, the bias-critical view holds that such a hands-off approach is ethically untenable. In contexts where margins for error are thin, biased AI can widen existing health gaps instead of bridging them. Without intervention, AI will disadvantage already under-represented and marginalised groups, systematically worsening existing inequity.

One widely used U.S. healthcare algorithm systematically underestimated the health needs of Black patients by using prior healthcare expenditure as a proxy, thereby replicating patterns of historical underutilization of care. A skin cancer diagnosis algorithm, trained mainly on images of light-skinned patients, had approximately half the diagnostic accuracy when used on images of lesions among African-American patients. The issue is a structural flaw with material consequences. This perspective argues that equity cannot be retrofitted: It must be mixed in from the start. It requiress participatory design, fairness audits, and a reengineering of values into the systems themselves. The goal is to rethink AI as an inclusionary technology that serves all and not just the dominant groups reflected in its data.

The dilemma becomes acute when we consider a subject like war. A University of Newcastle thesis by Mitchell John Hobbs on the 2003 Iraq War provides a telling case study. In the month before the invasion, Rupert Murdoch's flagship Australian newspaper, The Australian, showed a clear propensity for pro-war articles by a proportion of approximately two to one, with 57.5 articles for the invasion versus 33.5 against. An AI trained on this vast media output would therefore inevitably learn a pro-war frame.

Given the two perspectives, what is the fix? Should one attempt to neutralize this frame by injecting anti-war coverage, thus creating a balanced model that might be incoherent? Or should we accept the model's pro-war disposition as an accurate reflection of a powerful media force's influence? One study showed that while RLHF could produce positive overt stereotypes about African Americans, it still exhibited covert racism in the form of dialect-based prejudice. The training obscures the racism on the surface, but the stereotypes remain unaffected on a deeper level.

Researchers working on the Libra project in Aotearoa, for instance, documented how an LLM might interpret the Māori word Karani , meaning Grandmother, as a 3D printed model, a noisy car, or an iced coffee. These outputs arise because region-specific terms are beyond the models' knowledge boundaries, a result of being underrepresented in the massive, yet culturally monolithic, English-centric Internet corpora they are trained on.

Now this is not necessarily a malicious bias but a reflection of a dataset's limitations, what those researchers call a lack of local context. When a model produces irrelevant results from unfamiliar words, it is not necessarily indicative of inherent bias. This observation raises a different question, one that our teams see everyday: Is the objective to correct a global model's ignorance, or to build new models grounded in specific cultural and linguistic worlds from the start?

The debate lands us at a philosophical impasse. One side contends that relying entirely on distributions means there is inherently no reliable way to distinguish one kind of bias from another, and without a serious reconsideration of AI's foundational assumptions, the problem cannot be solved. The other side warns that without a conscious effort to detect and minimize algorithmic bias, we risk automating injustice and entrenching inequality in the very technologies we hope will enhance our world.