Building AI Trust: Governance, Control and Responsibility

It is no longer enough to ask whether an AI system can be prevented from making a mistake. Companies also have to consider what happens when something goes wrong — and who is prepared to respond.

That was a central theme in a conversation with Shana Simmons, Zendesk’s Chief Legal Officer. Asked to explain what AI governance means from a customer’s point of view, Simmons described a framework built around three pillars: governance, control, and consequence management.

The aim, she suggested, is not to place all responsibility on either the customer or the technology provider. It is to make their respective roles clear, and to build the systems and working relationships needed to manage risk together.

Governance

“Do you have the right guardrails underlying the data?” Simmons asked, describing the first pillar. For customers, governance means more than a set of policies. It includes the privacy and data protections that underpin a system, as well as the ability to govern the AI tools they use. It also means asking whether the provider has put credible practices in place for its own systems, including independent certifications.

Control

The second pillar is control: who gets to decide how an AI agent operates, and when it should act? Simmons emphasized that customers should retain control over the AI they deploy. But customer control does not mean the provider has no responsibility. In some cases, she said, a vendor may need to intervene—for example, if monitoring reveals a problem with an integration.

The control is with the customer,” Simmons said. At the same time, she acknowledged that providers may need to be more prescriptive in situations where safety requires it. “We want it in some things, we want to be a bit more prescriptive to be able to shut it down when we need to.

—Shana Simmons

Consequence management

That tension reflects a broader shift in how organizations think about AI safety. The goal is not only to stop problems before they occur, but also to have the capacity to act quickly when prevention falls short. Simmons described consequence management as a shared concern, with the vendor prepared to support customers when an incident involves a system, partner, or outside actor.

Building a culture of trust

Technical controls, however, are only part of the picture. A culture of trust has to reach across teams and into the decisions made while products are being built. Processes, procedures and training do matter, but so does a consistent organizational commitment to trust from the beginning.

Simmons’ argument is that legal teams should not wait until a product is finished before offering guidance.

We can’t wait till the end,” she said. “It’s too late.” Instead, legal advisers need to be close enough to the technology and the teams building it to understand the choices being made — and to help shape them.

—Shana Simmons

That proximity requires lawyers to develop new skills. Simmons said she wants her department to become more fluent in AI through direct use of the tools.

I need my lawyers to become more AI literate,” she said. She described a planned shift: first, requiring staff to use AI; then, moving toward “mandated outcomes built using AI.

—Shana Simmons

The point, as she framed it, is not simply to adopt a new tool, but to make sure legal professionals understand the technology well enough to advise on its real-world use.

When agents go rogue

The challenge becomes especially tangible with AI agents — systems that can carry out tasks with increasing independence. How custom agents can be made safer and more consistent requires organizational leaders to define guardrails and decide when a person should step in. The right answer depends on the context. An agent handling a routine retail return presents different considerations from one operating in a more sensitive setting.

Simmons stressed that customers need to help make those decisions, because they understand their own businesses and the people they serve. The provider can supply tools and guidance, but the customer’s context matters. She also described a movement toward industry-specific agents, where providers and customers can learn together and make systems more appropriate to particular uses.

The same principle, she said, applies to where and how customers build. As AI development becomes more accessible, employees without formal engineering roles may create tools or agents of their own. Simmons described the opportunity — and the risk — in that democratization: organizations need ways to support experimentation while keeping development within environments where security, observability, and privacy practices are built in.

That does not eliminate the need for human judgment. A concern is that that people may become so accustomed to interacting with AI that biases creep in and oversight could fade into the background. Simmons agreed that people need to remain involved, not only in the moment an agent acts, but in the broader system of oversight and responsibility around it.

For her, that depends partly on organizational culture. Employees need the agency and creativity to build and improve systems, but they also need to understand their obligations. Simmons described the goal as fostering both agency and governance: empowering people to use the technology while maintaining care for customers and the wider public.

Accountability and responsibility

The discussion eventually widened from AI controls to an old organizational question: does responsibility come before accountability, or does accountability create responsibility? Simmons found the question compelling, and suggested that accountability may be what gives responsibility its practical force.

This exchange echoed the challenges of AI governance itself. Assigning responsibility is not enough if no one can see what a system is doing, measure its effects, or respond when it causes harm.

Simmons also likened the distinction between responsibility and accountability to the relationship between a company’s mission and vision. She described her department’s mission as facilitating growth while mitigating risk, while noting that a company’s broader vision may need to evolve as technology changes. The underlying task is to keep the organization’s direction connected to the needs of customers and the people affected by its systems.

In Simmons’s account, trustworthy AI is not a single certification, a single human checkpoint, or a promise that failures will never happen. It is a continuing practice: put guardrails in place, give customers meaningful control, build systems that can be monitored, and prepare to respond when outcomes are unexpected.

Just as important, bring legal and security expertise into development early — and ensure that the people using AI understand both its capabilities and their accountability for the results.