The AI Trust Gap: What Happens When an Agent Goes Rogue?

As companies rush to deploy AI agents, they're facing a critical trust gap: what happens when an agent goes rogue?

TL;DR
• Companies are racing to deploy AI agents, but trust is a major concern.
• Building trust in AI requires clear policies and procedures.
• Companies must balance the benefits of AI with the risks of agent overreach.

Summary
The deployment of autonomous AI agents in the workforce is accelerating, but a significant trust gap remains. Companies are struggling to balance the benefits of AI with the risks of agents going rogue. Dev Rishi, general manager for AI at Rubrik, has identified four phases of agentic AI adoption: experimentation, formalization, scaling, and autonomous AI. Currently, 50% of companies are in the experimentation phase, 25% are formalizing prototypes, 13% are scaling, and 12% have not started AI projects.

The main risk holding companies back is security and governance. Companies need to understand the risks when AI agents overstep guardrails and develop failsafes for such situations. Kathleen Peters, chief innovation officer at Experian, notes that executives want to feel safe and understand how to mitigate risks. Chandu Nair, senior vice president at Lowe's, compares integrating agents into the workforce to hiring people without an HR function.

To build trust in agents, companies need systems that provide confidence that agents are operating within policy guardrails and clear policies and procedures for when things go wrong. Key factors for building trust include identity and accountability, evaluating the quality of each agent's output, and reviewing the post-mortem trail of mistakes. Companies like Lowe's have seen strong adoption and tangible return on investment from AI, such as agent companions for store associates with extensive product knowledge.

As AI continues to mature, enormous potential upside is expected, such as agents trained on multiple doctors' expertise to catch tumors in dense tissue. However, human oversight is critical, especially in high-stakes industries like healthcare. The threat of overreach by rogue agents is ever-present, and companies must be prepared to address it.
• Companies need to develop clear policies and procedures for AI agent deployment.
• Human oversight is critical, especially in high-stakes industries.
• Building trust in agents requires systems that provide confidence in policy guardrails and clear procedures for when things go wrong.

As AI continues to transform the workforce, companies must prioritize building trust in their agents to avoid the risks of overreach.