Stanford Warns: AI Agents Could Create a New Era of Conflicts of Interest
Artificial intelligence is entering a new phase.
For the past few years, the public conversation around AI has largely focused on chatbots, hallucinations, misinformation, job displacement and the risks of generating harmful content. But the technology is rapidly changing. AI systems are no longer limited to answering questions or generating text. Increasingly, they can search the web, interact with applications, make recommendations, execute tasks, communicate with other systems and act on behalf of users.
These systems are commonly referred to as AI agents.
And according to a new policy brief from Stanford’s Institute for Human-Centered AI, this transition creates a problem that could become just as important as accuracy or cybersecurity: conflicts of interest between AI agents, their users and the companies that build them.
Stanford’s August 25 policy brief, “Designing Loyalty: AI Agents and Conflicts of Interest,” examines how increasingly autonomous agents could face incentives that do not necessarily align with the interests of the people relying on them. The researchers argue that developers and deployers may need to take on stronger responsibilities toward users, including what the brief describes as a duty of loyalty.
This is a significant shift in the AI governance debate.
The question is no longer simply:
“Can we trust AI to give us the right answer?”
It is increasingly becoming:
“Can we trust an AI agent to act in our best interests?”
From Chatbots to AI Agents
Understanding the problem requires understanding what makes an AI agent different from a conventional chatbot.
A chatbot generally responds to a user’s prompt. You ask a question, and it generates an answer.
An agent can potentially go much further.
Imagine telling an AI:
“Find me the best business-class flight to Singapore next week, compare the options, check my calendar, and book the flight.”
A sufficiently capable agent could search airline websites, compare prices, consider schedules, access calendar information and potentially complete the booking.
Now imagine giving an agent responsibility for purchasing software for a company.
The agent might compare vendors, negotiate prices, evaluate contracts, request approvals and make a purchase.
Or consider a financial assistant that monitors markets, evaluates investment products and recommends where to put money.
The more authority an agent receives, the more important its underlying incentives become.
A chatbot giving you a bad recommendation is one problem.
An autonomous agent acting on that recommendation is something entirely different.
Stanford’s latest policy work focuses precisely on this transition.
The Hidden Conflict
The fundamental problem is surprisingly simple.
An AI agent may appear to work for the user.
But the organization operating the agent may have completely different economic interests.
Consider a hypothetical shopping agent.
You tell it:
“Find the best laptop for me.”
The obvious assumption is that the agent will identify the laptop that best matches your requirements.
But what happens if the AI company receives advertising revenue from certain manufacturers?
What if one retailer pays the platform for preferential placement?
What if the AI company receives a commission when users purchase a particular product?
Suddenly, there are potentially two competing objectives:
User objective: Find the best product.
Platform objective: Generate revenue.
Humans already understand this problem in traditional markets.
Financial advisers may have conflicts involving commissions.
Doctors can have incentives connected to healthcare systems.
Travel agents may receive commissions.
Online platforms use advertising and sponsored placement.
But AI agents could make the situation more complicated because users may interact with them as if they were personal representatives rather than commercial platforms.
The agent may speak in a personalized way.
It may remember preferences.
It may have access to private information.
It may continuously act on the user’s behalf.
That creates an entirely different relationship.
Why “Loyalty” Matters
This is where Stanford’s concept of loyalty becomes important.
The Stanford policy brief argues that AI agents need governance mechanisms that address conflicts between the interests of users and the interests of developers or deployers. The researchers propose thinking about these relationships through the concept of a duty of loyalty.
The idea is not that AI itself becomes a moral person.
Instead, the responsibility would fall on the organizations designing and deploying these systems.
If a company tells consumers:
“This AI agent works for you.”
then users should have meaningful protections against the company secretly steering the agent toward outcomes that benefit the company instead.
This could become one of the defining principles of agentic AI.
Imagine an AI Financial Agent
Consider a more consequential example.
Suppose you ask an AI agent:
“I have ₹10 lakh. Build a diversified investment portfolio for me.”
The agent analyzes your financial goals, risk tolerance and time horizon.
It recommends several investments.
But suppose the AI provider earns higher fees from certain financial products.
Would the agent recommend those products because they are genuinely better for you?
Or because the provider benefits financially?
The difference may be invisible to the user.
That is precisely what makes agent conflicts potentially dangerous.
With traditional advertising, consumers generally understand that an advertisement is promotional.
With an AI agent, the recommendation may feel like personalized professional advice.
The user may not even know that a commercial incentive exists behind the recommendation.
The AI Agent Could Know More About You Than You Realize
There is another dimension that makes this issue particularly important.
AI agents can potentially accumulate enormous amounts of personal context.
Imagine an agent that knows:
Your income
Your spending habits
Your travel history
Your calendar
Your work schedule
Your shopping preferences
Your communications
Your health-related searches
Your professional relationships
Your investment preferences
That information can make an agent extremely useful.
But it also creates enormous potential for manipulation.
If the system understands that you are likely to purchase a particular product, it could potentially use that information to influence recommendations.
If it knows you are under financial pressure, it could alter how it presents options.
If it understands your emotional state, it could potentially tailor its persuasion accordingly.
The more personalized the agent becomes, the greater the responsibility to prevent conflicts of interest.
The Agentic Economy Could Change Advertising
Advertising may be one of the industries most affected.
For decades, the internet has operated around a basic model:
Companies compete for human attention.
Search engines display advertisements.
Social networks optimize feeds.
Websites sell advertising space.
But AI agents could introduce a completely different model:
Companies compete for the recommendation of the user’s AI agent.
Imagine millions of consumers delegating purchasing decisions to AI.
Instead of searching Google for a smartphone, a consumer asks their AI agent:
“Which phone should I buy?”
Instead of browsing dozens of hotel websites, they say:
“Book the best hotel in Tokyo for my budget.”
Instead of comparing insurance products manually, they ask:
“Find the most suitable policy.”
If AI agents become the gatekeepers between consumers and businesses, companies will have enormous incentives to influence those agents.
This could create a new form of digital advertising:
Agent influence.
And it raises a critical question:
Should companies be allowed to pay to influence an AI agent’s recommendation?
The Risk of “Pay-to-Recommend”
Imagine an AI shopping assistant that says:
“I found three products that match your requirements.”
But one manufacturer paid the AI platform to make sure its product appears first.
The user may never know.
That is potentially more subtle than conventional advertising.
The recommendation could look completely organic.
The AI might even provide convincing reasons for choosing the sponsored product.
This creates the possibility of a future where commercial influence becomes embedded inside AI reasoning and recommendations.
Transparency therefore becomes critical.
Users may need to know:
Who is paying the AI provider?
Does the agent receive commissions?
Are recommendations sponsored?
Can companies influence rankings?
What information is being used?
What objectives is the agent optimizing?
Does the agent prioritize the user’s interests?
These questions may eventually become regulatory requirements.
The Problem Gets Bigger When Agents Can Take Action
Recommendation is only one part of the problem.
The consequences become more serious when agents can actually execute transactions.
Suppose an agent is authorized to:
Buy products
Book flights
Sign up for services
Transfer money
Negotiate contracts
Send emails
Hire contractors
Purchase software
Manage subscriptions
Now a conflict of interest can directly produce a financial or operational consequence.
If an agent recommends a more expensive product because its provider receives a commission, the user could lose money.
If an enterprise procurement agent favors a particular supplier because of an undisclosed commercial relationship, the organization could face financial, legal and reputational risks.
This is why the governance question cannot be separated from agent capability.
The more autonomy an AI system receives, the stronger the controls around incentives need to become.
AI Agents Could Become Corporate Employees Without Being Employees
There is another fascinating implication.
Companies are increasingly experimenting with AI agents for business functions such as:
Customer service
Procurement
Cybersecurity
Sales
Finance
Research
HR
Software development
Compliance
Risk management
An enterprise may eventually have hundreds or thousands of agents performing specialized tasks.
But who do these agents ultimately serve?
The employee?
The business unit?
The customer?
The software vendor?
The shareholder?
The answer may not always be obvious.
Imagine a procurement agent instructed to minimize costs.
The cheapest supplier might not necessarily be the safest supplier.
A risk-management agent might identify a major risk that threatens a business relationship.
A sales agent might prioritize revenue over customer suitability.
A cybersecurity agent might recommend an action that protects the organization but creates significant operational disruption.
These are not purely technical problems.
They are governance problems.
The Principal-Agent Problem Gets an AI Upgrade
Economists and corporate governance experts have long studied the principal-agent problem.
It occurs when one party—the agent—is supposed to act on behalf of another—the principal—but the two have different incentives or access to information.
AI agents could create a technological version of this problem at enormous scale.
Consider:
Principal: Consumer
Agent: AI assistant
Potential competing interest: AI company
Or:
Principal: Company
Agent: Procurement AI
Potential competing interest: Software vendor
Or:
Principal: Patient
Agent: Healthcare AI
Potential competing interest: Healthcare provider or platform
Or:
Principal: Investor
Agent: Financial AI
Potential competing interest: Financial-product provider
The technology could therefore amplify a problem that already exists in traditional economics.
But AI could make the agent faster, cheaper and more influential.
That makes the consequences potentially much larger.
Why Disclosure Alone May Not Be Enough
One possible response is simple disclosure.
An AI agent could tell users:
“This recommendation may benefit our commercial partners.”
But would that really solve the problem?
Probably not by itself.
Most users are unlikely to understand complex incentive structures buried inside an AI system.
And disclosure does not necessarily prevent manipulation.
If the agent still systematically prioritizes the provider’s commercial interests, telling the user about the conflict may not be sufficient.
This is why Stanford’s proposal is significant.
The debate is moving beyond:
“Tell users that AI has conflicts.”
toward:
“Design the system so those conflicts are appropriately managed in the first place.”
What Could a Duty of Loyalty Look Like?
A future regulatory framework could potentially require AI-agent providers to follow principles such as:
1. Put user interests first
If an agent explicitly operates on behalf of a user, its primary objective should be aligned with that user’s authorized goals.
2. Disclose commercial relationships
Users should know when recommendations are influenced by commissions, advertising or other financial incentives.
3. Prevent undisclosed steering
AI providers should not secretly manipulate agent decisions to benefit themselves or partners.
4. Preserve user control
Users should be able to understand and override important agent decisions.
5. Maintain auditability
High-impact decisions should generate records that allow organizations to determine why an agent took a particular action.
6. Establish accountability
When an autonomous agent causes harm because of poorly designed incentives, there should be a clear responsible party.
These principles would not eliminate every risk.
But they could create a foundation for trustworthy agentic systems.
The Enterprise Risk Is Particularly Significant
For businesses, this issue should be treated as more than an AI ethics discussion.
It could become an enterprise risk management issue.
Organizations deploying autonomous agents should consider questions such as:
Strategic risk:
Could the agent make decisions that conflict with corporate strategy?
Financial risk:
Could commercial incentives cause the agent to select unnecessarily expensive products or services?
Operational risk:
Could autonomous decisions disrupt business processes?
Compliance risk:
Could the agent violate regulatory requirements?
Legal risk:
Who is responsible when an agent makes a consequential decision?
Cybersecurity risk:
Could attackers manipulate the agent’s objectives or information?
Reputational risk:
What happens if customers discover that an AI agent secretly prioritized the company’s interests?
Third-party risk:
Does the AI provider have commercial relationships that could influence agent behavior?
This means AI governance should increasingly connect with existing ERM, internal audit, compliance and third-party-risk frameworks.
The New AI Governance Stack
A mature organization may eventually need several layers of controls.
Layer 1 — Model governance
Evaluate the underlying AI model.
Layer 2 — Agent governance
Control what the agent is allowed to do.
Layer 3 — Identity and permissions
Determine which systems, accounts and data the agent can access.
Layer 4 — Incentive governance
Understand whose interests influence the agent.
Layer 5 — Transaction controls
Require approvals for high-impact actions.
Layer 6 — Monitoring
Continuously monitor agent behavior.
Layer 7 — Auditability
Maintain records of important decisions.
Layer 8 — Human escalation
Ensure humans can intervene when circumstances exceed predefined limits.
This is fundamentally different from governing a traditional chatbot.
The Biggest Question: Who Does the Agent Work For?
This may ultimately become the central question of the agentic AI era.
When you open a chatbot, the relationship is relatively straightforward.
You ask.
The AI answers.
But when an AI agent starts negotiating contracts, buying products, managing investments or interacting with companies on your behalf, the relationship becomes much more complicated.
The agent becomes an intermediary between you and the digital economy.
And intermediaries have power.
Whoever controls the intermediary can potentially influence the outcome.
This is why Stanford’s warning deserves attention.
The future AI battle may not only be about which company builds the most intelligent model.
It may also be about who controls the incentives behind autonomous systems.
What This Means for Consumers
Consumers should not necessarily avoid AI agents.
The technology could deliver enormous benefits.
An agent could save hours of research.
It could compare thousands of products.
It could identify better travel options.
It could manage schedules.
It could automate repetitive administrative tasks.
It could help people make more informed decisions.
But consumers should increasingly ask:
“Why is my AI recommending this?”
That question could become as important as asking:
“How accurate is this AI?”
The two issues are different.
An AI can be extremely accurate while still having incentives that are not aligned with your interests.
What This Means for Regulators
Governments now face a difficult challenge.
Regulation that is too weak could allow powerful AI intermediaries to develop opaque commercial incentives.
Regulation that is too restrictive could slow useful innovation.
The solution will likely require a risk-based approach.
An AI agent that recommends a restaurant does not require the same governance as one that manages a retirement portfolio.
An agent that drafts an email does not require the same controls as one that signs a multimillion-dollar contract.
The level of oversight should therefore increase with the potential impact of the agent’s actions.
Stanford’s broader AI policy work emphasizes the need to connect technological development with human-centered governance, and its 2026 AI Index has highlighted a widening gap between AI capabilities and society’s preparedness to manage them.
The Agentic AI Era Is Just Beginning
Perhaps the most important takeaway is that the AI-agent economy is still in its early stages.
Today’s agents are already becoming capable of browsing, reasoning, using software tools and completing multi-step tasks.
Tomorrow’s agents could become much more autonomous.
They could manage businesses.
They could negotiate with other agents.
They could continuously optimize spending.
They could represent individuals in digital marketplaces.
They could interact with banks, insurers, retailers, employers and governments.
At that point, the question of who the agent serves becomes fundamental.
If an AI agent is supposed to represent you, it cannot simultaneously be quietly optimized to benefit someone else without appropriate safeguards.
That is the conflict Stanford is highlighting.
Conclusion: Trust Will Become the Competitive Advantage
The AI industry has spent years competing on model intelligence.
Bigger models.
Better reasoning.
Faster inference.
More context.
More tools.
But the next stage may be different.
As AI systems become agents, trust could become one of the most valuable features.
Users will want to know that their AI assistant is genuinely working for them.
Businesses will want to know that their autonomous systems are operating within approved objectives.
Regulators will want to know that powerful AI intermediaries cannot secretly manipulate markets or consumers.
And developers will need to demonstrate that their agents are not merely capable—but appropriately aligned with the people they are supposed to serve.
Stanford’s latest policy brief therefore raises a question that could define the next decade of artificial intelligence:
When an AI agent acts on our behalf, whose interests does it actually represent?
The answer cannot simply be hidden inside an algorithm.
As AI moves from generating answers to taking actions, loyalty, transparency, accountability and incentive alignment may become just as important as intelligence itself.
The future of AI will not be determined only by how smart agents become.
It will also depend on who controls them, who benefits from their decisions—and whether users can trust that the agent is truly working for them.
Source: Stanford Institute for Human-Centered AI, Designing Loyalty: AI Agents and Conflicts of Interest, August 25, 2026.
