Why Unequal Power Matters in Human–AI Futures
AI is often discussed through capability: what it can generate, automate, predict, personalise, or accelerate. But capability alone does not tell us what an AI system will mean in people’s lives.
The missing question is often one of power: who has the ability to shape an AI system, who is affected by it, and who can question its consequences?
Taking unequal power seriously is not the same as assuming that every AI system will deepen inequality. It means recognising that technologies are developed and used within institutions, markets, and societies where authority, resources, voice, and exposure to risk are already unevenly distributed. Those conditions affect what gets built, what is adopted, what is measured as success, and whose concerns are treated as relevant.
AI does not arrive neutrally
AI systems do not enter a blank social landscape. Someone identifies a problem, chooses a vendor or model, defines the intended use, supplies or selects data, determines the acceptable error rate, and decides how much human review is necessary.
At each point, choices are made.
A customer-service assistant, for example, might be designed to make information easier to find. It might also be designed primarily to reduce the volume of human contact. Neither outcome can be inferred from the technology itself. The result depends on the organisation’s purpose, incentives, governance, and willingness to monitor how different customers experience the service.
The same is true in workplaces, education, health, public services, marketing, and creative practice. AI can support expertise, widen access to knowledge, and reduce administrative burden. It can also introduce new dependencies, errors, exclusions, or constraints. The task is not to begin with a conclusion. It is to make the relevant distribution of benefits, costs, agency, and accountability visible.
That is why major international AI principles focus not only on innovation but also on human rights, fairness, transparency, accountability, and meaningful information for people affected by AI-enabled outcomes. The OECD’s AI Principles state that AI actors should provide context-appropriate information about systems and, where feasible and useful, information that enables affected people to understand and challenge an AI-related output or decision.
The question behind “the user”
The language of AI often refers to “the user,” as if this were a single, stable category. In reality, there may be several groups around one system:
The people who commission or purchase it.
The people who build, configure, and maintain it.
The employees expected to use it.
The people whose data contributes to it.
The customers, citizens, students, patients, or communities affected by its outputs.
The people asked to deal with errors, appeals, exceptions, and harms.
These groups do not have equal influence.
The organisation paying for a system may have a direct relationship with the supplier. The person assessed, profiled, recommended to, or filtered by the system may never know how it works, what information shaped the output, or how to contest it. The gap between those positions is a power question.
This does not mean that every opaque or automated process is unjust. It means that opacity and limited avenues of contestation should be examined with particular care when AI meaningfully affects access, opportunity, reputation, safety, work, or participation.
UNESCO’s AI ethics recommendation calls for ethical impact assessments that consider benefits, concerns, and risks, including impacts on human rights, vulnerable or marginalised people, labour rights, environmental effects, and broader ethical and social implications. It also identifies citizen participation as part of that process.
Unequal power shapes the future before deployment
The power question begins well before an AI system is live.
It appears when an organisation decides which problem deserves investment. It appears when leaders decide whether the purpose is augmentation, cost reduction, quality improvement, compliance, growth, risk control, or experimentation. It appears when they decide whether “success” will be measured through speed, conversion, resolution time, employee experience, error reduction, customer trust, accessibility, learning, or something else.
Metrics are never just technical. They are expressions of what an organisation has decided to value.
If an AI initiative is assessed only through efficiency, then effects that are harder to quantify may receive less attention: whether people understand the system, whether professional judgment is strengthened or weakened, whether exceptions are handled well, whether trust is maintained, or whether frontline knowledge has informed the design.
A power-aware approach does not reject metrics. It broadens the question of what deserves to be measured.
Anticipation is also a power issue
Human–AI futures are shaped by the stories we tell in advance.
Some actors have extensive capacity to anticipate: they can fund research, commission scenarios, access data, influence policy, set technical standards, develop products, run pilots, and market a vision of what is coming. Others encounter these futures only after they have been designed.
This is why anticipation is not merely a forecasting activity. It is also a form of influence.
When companies, governments, platforms, experts, and media repeatedly describe a particular future as inevitable, they can narrow public imagination. “AI will replace this role,” “customers will expect this,” “there is no alternative to automation,” or “this is simply the direction of travel” may function as descriptions. But they can also make certain choices appear inevitable before their implications have been openly debated.
A more careful futures practice asks:
Which future is being presented as inevitable?
Who benefits if it becomes the default?
What assumptions about people, work, intelligence, value, or progress does it contain?
What alternative futures are being overlooked?
Who has not yet been invited into the conversation?
This is not an argument for endless delay. It is an argument for better judgment before lock-in occurs.
From inclusion to meaningful participation
“Inclusion” is often treated as the answer to unequal power. It is important, but it can become superficial when it means inviting people into a process without giving their knowledge any influence over decisions.
Meaningful participation involves more than consultation. It requires clarity about what is open to change, who has decision-making authority, what evidence will count, and how concerns will be addressed after deployment.
For an organisation, this may mean involving legal, technical, operational, design, people, and customer-facing teams early enough to influence the framing of an AI initiative — not only after the tool has been selected. It may mean creating escalation routes when an AI-supported decision seems wrong. It may mean defining where human judgment is required, documenting why, and revisiting that decision as the system and context change.
The OECD frames accountability as a lifecycle responsibility: actors should be accountable for the proper functioning of AI systems and for respecting applicable principles according to their role, context, and ability to act.
In practical terms, accountability asks simple but difficult questions:
Who owns the decision to deploy this system?
Who can pause, change, or retire it?
Who is responsible for monitoring unintended effects?
Who responds when an output is wrong, harmful, or misleading?
Who can meaningfully raise a concern?
What happens after a concern is raised?
Without answers, “responsible AI” risks becoming an aspiration without an operational home.
A better ambition for human–AI futures
Taking unequal power seriously does not require treating AI as inherently harmful or abandoning experimentation. It asks for a more mature ambition than adoption for its own sake.
The goal should be to create conditions in which AI expands people’s ability to understand, decide, create, learn, and participate — not merely an organisation’s capacity to act faster. This involves treating AI adoption as an organisational and cultural question as much as a technical one.
The NIST AI Risk Management Framework offers a useful operational language: Govern, Map, Measure, and Manage. In practice, this means establishing roles and accountability; understanding the context, intended use, and stakeholder impacts; assessing relevant risks and trustworthiness characteristics; and responding to and monitoring those risks over time.
Power belongs inside each of those activities.
Govern: Who holds responsibility, and who has authority to challenge decisions?
Map: Which groups are affected, and how do their interests differ?
Measure: Are we measuring only technical performance, or also meaningful human and organisational outcomes?
Manage: What happens when the system creates an error, an exclusion, a conflict, or a loss of trust?
The future of human–AI relations will not be determined by model capability alone. It will be shaped by the choices people and institutions make around ownership, purpose, oversight, participation, and accountability.
Unequal power deserves attention because it helps us see that these choices are never purely technical. They are choices about whose knowledge matters, whose time is protected, whose judgment counts, and who gets to participate in authoring the future.
References
National Institute of Standards and Technology. (2023). Artificial intelligence risk management framework (AI RMF 1.0) (NIST AI 100-1). U.S. Department of Commerce. https://doi.org/10.6028/NIST.AI.100-1
Organisation for Economic Co-operation and Development. (2024). Recommendation of the Council on artificial intelligence (OECD/LEGAL/0449). https://legalinstruments.oecd.org/en/instruments/OECD-LEGAL-0449
United Nations Educational, Scientific and Cultural Organization. (2021). Recommendation on the ethics of artificial intelligence. https://unesdoc.unesco.org/ark:/48223/pf0000381137
