If artificial intelligence (AI) systems shape decisions that affect people’s lives, they should do so fairly. This should be a given considering that potential applications for AI include automated hiring systems, as well as tools used in education, finance and criminal justice.
But ensuring the fairness of AI systems is far more complex than it might sound. Despite years of research, there is still no consensus on what fairness means, how it should be measured, or whether it can ever be fully achieved.
Fairness inherently depends on context. What counts as fair in one domain may be inappropriate or even harmful in another. In criminal justice, fairness may prioritise avoiding disproportionate harm to particular communities. In education, it may focus on equal opportunity and long-term outcomes.
In finance, it often involves balancing access to credit with risk assessment. Because AI systems must be formalised mathematically, researchers translate fairness into technical definitions expressed through metrics that specify how outcomes should be distributed across groups.
These metrics are useful tools, but they are not neutral. Each encodes assumptions about which differences matter and which trade-offs are acceptable.
Problems with the data
A deeper issue lies in the data itself. AI systems learn from historical datasets that reflect past decisions, institutional practices, and social inequalities. When a model is trained to replicate observed outcomes, such as hiring decisions or loan and mortgage approvals, it may reproduce existing injustices under the appearance of objectivity.
Optimising for one notion of fairness often means violating another. This tension is evident in automated loan approval systems. An algorithm may be designed so that applicants with the same predicted probability of default are treated similarly across demographic groups.
Yet one group may still be more likely to be incorrectly denied credit, while another may be more likely to receive loans they later struggle to repay. Fairness in predictive accuracy can therefore conflict with fairness in how financial risk and opportunity are distributed.
These differences often reflect structural inequalities embedded in the data the model is trained on. Groups that have historically faced barriers to credit, due to factors such as discrimination or exclusion from financial systems, may have thinner credit histories or lower recorded incomes.
As a result, models can treat socioeconomic disadvantage as a signal of higher risk, even when it does not reflect an individual’s actual ability to repay.
The same pattern emerges in hiring. If a company historically promoted fewer women into senior roles, a system trained to predict “successful” candidates may learn patterns that favour characteristics more common among men, even if gender is not explicitly included as an input. In both cases, the model does not invent bias, it inherits it.
A fundamental question is whether AI systems mirror the world as it was, or attempt to correct for known injustices.
The idea of fairness is further complicated by how it is assessed. Many assessments examine a single protected attribute, such as gender or race, in isolation. While common, this approach can obscure how discrimination operates in practice.
An automated hiring system might appear fair when comparing men and women overall, and fair when comparing ethnic groups overall, yet it might also consistently disadvantage older women from minority backgrounds.
Pla2na
Complex evaluation
People are defined by several characteristics that intersect, including age, ethnicity, disability, and socioeconomic background. Because these intersectional subgroups are often small and underrepresented in data, the harms they face may remain invisible in standard evaluations.
This invisibility has a direct technical consequence. When a subgroup is small, the model encounters too few examples to learn reliable patterns for that group and instead applies generalisations drawn from the broader categories it has seen more of, which may not reflect that group’s actual characteristics or circumstances.
Errors and biases affecting small subgroups are also less likely to surface in standard performance metrics, which aggregate results across all users and can therefore mask poor outcomes for minorities within minorities. Which means that those most at risk are therefore often the least visible.
These challenges suggest that fairness in AI cannot be reduced to better metrics or more sophisticated algorithms. Fairness is shaped by institutional context, historical legacies, and power relations.
Decisions about what data to collect, which objectives to optimise, and how systems are deployed are influenced by social and organisational factors. Technical fixes are necessary but insufficient. Meaningful approaches must engage with the broader context in which AI systems operate.
This includes involving interested parties beyond engineers and data scientists. People affected by AI systems, often members of marginalised communities, possess contextual knowledge about risks and harms that may not be visible from a purely technical perspective.
Participatory approaches, in which affected groups contribute to the design and governance of AI systems, acknowledge that fairness cannot be defined without considering those who bear the consequences of automated decisions.
Even when interventions appear successful, they may not remain so. Societies change, demographics shift and language evolves. A system that performs acceptably today may produce unfair outcomes tomorrow. In particular, recent advances in large language models, the technology underlying many widely used AI tools, add further complexity.
Unlike traditional systems that make specific predictions, these models generate language based on vast collections of historical text. Such datasets inevitably contain stereotypes and imbalances.
Fairness is therefore not a one-time achievement but an ongoing responsibility requiring monitoring, accountability, and a willingness to revise or withdraw systems when harms emerge.
Together, these challenges suggest that fairness in AI is not a purely technical problem awaiting a finite solution. It is a moving target shaped by social values and historical context.
Rather than asking whether an AI system is fair in the abstract, a more productive question may be: fair according to whom, under what conditions, and with what forms of accountability? How we answer that question will shape not only the systems we build, but the kind of society they help to create.
