Building a Robust Security Infrastructure for AI Agents

In this exclusive Q&A with Vinay Patel, Chief Trust & Security Officer, Zendesk, the discussion revolved around the robust security infrastructure required for deploying agentic AI, particularly in customer service contexts like Zendesk.

Key takeaways include the importance of a risk management framework, offensive security testing to prevent attacks like prompt injections, and ensuring transparency in AI agent behavior through conversation logs.

T he significance of third-party data flow and API durability was also raised, emphasizing the need for clear permissions and actions for AI agents. Building customer trust was highlighted as a daily effort, requiring transparency, competence, and continuous improvement. The responsibility and accountability of AI actions were also discussed.

What should a robust security infrastructure for AI agents include?

Vinay Patel: It starts with grounding ourselves in a risk management framework. For us, it starts fundamentally with governance and culture.

Let’s say someone comes along with an idea for an agentic AI feature. What should we do? We do an impact assessment of that particular feature along different dimensions, the outcomes of that assessment are something that we capture and log, and the risk that we capture and document then forces us to say “What are we going to do about that particular risk?” and this is all before anything is shipped to a customer.

This is not dissimilar to a lot of other risk management frameworks that you might think of in software development, where everyone's got kind of a circular loop that feeds into itself.

—Vinay Patel, Chief Trust & Security Officer, Zendesk

How should organizations test AI agents against attacks such as prompt injections?

Vinay Patel: Organizations should ensure they are very resilient to these types of attacks. But t here are two things to consider, otherwise you have a false sense of security when dealing with AI .

1. These are non-deterministic systems, and just because an AI agent behaved a certain way one time does not mean it's going to behave that way each time. I t's going to vary based upon context, so even that technique of just testing these malicious problems repeatedly needs to be not done once, but repetitively .
2. At Zendesk, we have a dedicated offensive security team, where we try to break our own agents. We look at actually poisoning the interactions in our offensive testing – when it looks very normal and the context is building up, we try to poison it, and repeating that thousands of times over thousands of different types, and then measuring a quantitative score.

Why is end-to-end visibility important for AI agents and third-party integrations?

Vinay Patel: A robust security infrastructure also means making sure you're clearly defining the purpose of that agent, then thinking about the permissions with which it operates, the data that it has access to, the actions that it is allowed to take, and what are those third-party systems that it is allowed to connect to.

Without such visibility, with the evolving nature of agentic AI , it will be more difficult to build trust with your customers.

How can companies build and maintain customer trust in AI?

Vinay Patel: For me, trust is a one - day contract . You have to renew that contract every day by consistently executing on the promises that you make, demonstrating the outcomes that you showed that they would achieve, and then ultimately, how you show up not only when things go right, but also when things don't go so well.

Trust is a one-day contract. You have to renew that contract every day...

—Vinay Patel

Who is accountable when an AI agent does something it should not?

Vinay Patel: You can never outsource accountability. We personally have an obligation to make sure we're doing our utmost best to build products with AI in a responsible way, in a secure way.

While we want to give organizations the control and flexibility, they also have an obligation to understand the technology a little bit, about its own kind of limitation - what it can and cannot do.

What should businesses keep in mind when using AI for fraud detection, where human and non-human identities are interconnected?

Vinay Patel: When there’s a tendency to think that AI will solve all problems – that’s usually when people get it wrong. There are certainly great use cases for where it will support that outcome, and there are equally a lot of use cases where it’s the absolute wrong thing to use, as there are some other people or tools that are much better at it.

If I just take a whole bunch of stuff and throw it to AI and say find a problem in it, that’s not going to be a good solution.

—Vinay Patel

What’s more useful and important is how they can build trust with their own customers. I think it fundamentally comes down to first and foremost demonstrating w hat happens to their data once it enters the system: How do we use that data? Where AI is involved, how is AI using that data? What are the guardrails that we implement around the agentic features that we make available to our customers, and what are the controls that we give them to further tune and customize what they want to be able to do with that?