Raju Dandigam speaks with host Amey Ambade about building robust AI agents for production. Raju recommends treating an agent as a software subsystem rather than a standalone model call. That means designing decision contracts and tool boundaries, choosing between predictable workflows and multi-agent orchestration, and managing the runtime, state, and streaming that make agents durable. A recurring theme is that the model should not own the system: the right architecture is usually the simplest one that meets the product need while preserving reliability and control, with the language model placed inside a controlled runtime that settles authentication, policy, data freshness, and idempotency before it is ever asked to reason.
They focus largely on how teams can be sure that an agent works and how to debug it when it does not. The episode covers behavioral testing against contracts, golden scenarios, and observability that captures the full execution path rather than flat logs, which is the gap behind agent-inspect and its readable execution trees. Raju and Amey close on operating agents in production: measuring cost and latency per run, knowing when a simpler model or no AI path is the right call, and treating prompts, schemas, tools, and evaluation sets as owned, versioned artifacts.
Brought to you by IEEE Computer Society and IEEE Software magazine.
Show Notes
Related Episodes
- SE Radio 719: Birol Yildiz on Building an Agentic AI SRE
- SE Radio 689: Amey Desai on the Model Context Protocol
- SE Radio 633: Itamar Friedman on Automated Testing with Generative AI
- SE Radio 610: Phillip Carter on Observability for Large Language Models
- SE Radio 534: Andy Dang on AI/ML Observability
- SE Radio 733: Max Corbridge on Securing AI Agents
References
- Aagent-inspect (execution tracing for TypeScript AI agents): GitHub – rajudandigam/agent-inspect: Local evidence debugger and trajectory-test toolkit for TypeScript AI agents: inspect causal runs, catch wrong tool paths in CI, and share safe offline evidence.
- Raju Dandigam, articles on production agent engineering (DEV): — DEV Community Profile
- “How to Build Production-Ready Agentic AI Systems with TypeScript” (Raju Dandigam, HackerNoon): How to Build Production-Ready Agentic AI Systems with TypeScript
- Model Context Protocol: What is the Model Context Protocol (MCP)?
- OpenTelemetry
- Intro | Zod
- Pydantic Docs
Transcript
Transcript brought to you by IEEE Software magazine.
This transcript was automatically generated. To suggest improvements in the text, please contact [email protected] and include the episode number and URL.
Amey Ambade 00:00:18 Welcome listeners to Software Engineering Radio. I’m your host, Amey Ambade and today I am joined by Raju Dandigam. Raju is a staff software engineer at Navan with more than 15 years of experience building AI powered products, scalable web platforms, test automation and developer tooling. His current works center around production AI agents, TypeScript based agent systems, proactive AI workflows, structural outputs, and the observability and engineering practices that move AI features from demos into reliable business applications. His recent work includes agent Inspect, a local first debugging and execution tracing tool for TypeScript AI agents. Today we’ll be talking to Raju about building production grade AI agents, how to treat an agent as a software subsystem rather than a prompt. What has to do with surrounding the model to make it reliable and how you test, observe and debug rather than prompt the systems once they’re carrying real work. Welcome to the show, Raju.
Raju Dandigam 00:01:24 Thanks. It’s great to be here.
Amey Ambade 00:01:26 Thank you. So, let’s talk about agents and what an agent is as a software subsystem. Listeners have heard agent take AI frameworks in a lot of different ways. So, let’s anchor this. In a few sentences, can you talk about what a production AI agent is and how it is different from a single call to a model?
Raju Dandigam 00:01:49 So, a production AI agent is a software subsystem in which a model has a controlled freedom to choose its next step or use an approved tool while working toward a defined goal. So, the model here is only one component. The complete system includes authentication, authorization, policy checks, type tools or the timeouts and everything around the regular application, right? So, the regular application, what we build and the operational ownership. So, the surrounding components are what make the agents safe enough and predictable enough to serve real users. So, in case of production agents, it’s usually talking less about prompt writing and it’s more about the engineering around the model. So, what it can see, what it may decide, which tools it may call, how its outputs are validated and what happens if it fails and how the team can stop or maybe roll back safely. So, how do we basically control or manage its behavior? So, that’s all makes the production an agent.
Amey Ambade 00:02:48 So, let’s build it up in stages. In a real system, what does a deterministic application own and what does the model own? Where do you draw that line?
Raju Dandigam 00:02:59 So, the application should own anything that can be decided reliably from clear facts, rules and that includes basically authentication, authorization, the checking policies, and maybe it can be looking for the prices or depends on the application, getting the data and everything related to the deterministic logic. So, that goes into the application. Whereas the model is useful for understanding the flexible language, comparing options, ranking, understanding the human input, and then coming up with some suggestions the next steps or maybe the response towards a goal. So, which has all the tools and basically the model has all the necessary inputs to define that, but then it has the power to come up with the suggestions and the options. So, the deterministic is where basically clear facts and clear paths, whereas the model side is basically more of finding the reasonable options and then the suggestions and the probable inputs to the user.
Amey Ambade 00:03:54 So, you have spoken before that a model should not own the system. What does that mean concretely for somebody who is wiring up their first agent?
Raju Dandigam 00:04:04 So, in case of a model, like let’s say that we are building an agent and then there is a model in between the system. So, what does it mean is basically the model is going to take care of a particular part of the system. So, let’s say that we take an example at a hypothetical personalized food ordering assistant. Let’s say that we are building a food ordering assistant and then the app or it can be anything, right? It can be a web application or a mobile application. Any user interface where the user comes in, obviously a logged in user comes in. So, in this case the surrounding parts which is going to be controlled by the application, but let’s say that as a model, what it can control is basically as a user I type in saying that, hey, I want a food or dinner for four people by a certain time or within a budget.
Raju Dandigam 00:04:47 And then soa dn so requirements like maybe I want a vegetarian or maybe I have some allergies and all those details. And then I wanted to order and then give me the options. So, that’s the user input. So, now the model is basically going over the input and then it’s going to process and then it’s going to come up with some options, it’s going to decide what it’s going to call and then it’s going to get the data and then process it, rank them and then come back to the output. So, the model is doing more than a single step.
Amey Ambade 00:05:16 I see what you mean. So, when we place a model or an LLM inside a controlled runtime, can you walk me through what the runtime settles before the model is ever called? Can you talk about, for example, authentication, authorization data freshness and rate limits?
Raju Dandigam 00:05:33 Okay, So, even before the model kicks in, we have this surrounding application which we have been building across any software application. Today we have any general application or any application, we’ll have the authentication just to make sure the user is a valid user and he has all the credentials and he’s available user who already has the account, and then the authorization, the permissions and what he can do, what he cannot do, and all other the policy checks and the features, what he can access, and then the inputs and basically the outputs are what he can get. So, this is the surrounding application we have. And then, sorry, I think what was the last part of your question?
Amey Ambade 00:06:06 I wanted you to talk about the rate limits and policy checks and data freshness.
Raju Dandigam 00:06:11 So yes, these are all going to be premodal, right? These checks and these application logic is still going to be controlled by the application and once we decide this user is cross all the steps where he basically authorized and then he has the policy, he has the permission to view the feature and then he is a valid user who can actually request and then get the data, get the information. So, this all controlled by the application and once the application reaches that point this is a place where the model can enter and do the action on behalf of the user request. So, that’s when the model kicks in.
Amey Ambade 00:06:43 Great. The distinction I want listeners to leave with early is what a workflow is versus what an agent is. What is the difference here and why is a maximum autonomy usually the wrong default in production? Can you take us through an example of a real feature where you choose a predictable workflow over an agent or the reverse?
Raju Dandigam 00:07:04 Yeah, let me step back here. Basically, we have these workflows and agents, right? And then we also have the simple LLM feature we just talked about. So, if you look at high level, we have this LLM feature where it is just basically a model gets some input and then responds with an output or with a text or maybe a structured output. So, it’s just basically a one-step where it gets input and then gives an output. That’s one part of the diagram. Let’s say if we look at it as a flow diagram and then it’s on the one part, but then we have these workflows, basically the agentic system. So, the agentic system has two levels. One is on the workflow side, and the other one is the agent side. When we come to workflow is basically workflow is where you have the model and then you know the defined path.
Raju Dandigam 00:07:48 So, the code here decides what to do next step. Meaning, I started with the food delivery system, and I just want to reorder my previous order, basically I want to just reorder. In this case I know all the path. So, basically, I need to go to the orders and get the order item whatever the last order and then get all the details of that order and then reorder it. So, here the steps are predefined. In this case we don’t need an agent. The workflow can easily do that part. So, that comes under workflow. The workflow can have various types, but it can be a prompt training where you feed one prompt or output to another next prompt and so on until you reach a goal. That’s prompt training. And then you can also do a routing based or you can do an orchestration based.
Raju Dandigam 00:08:29 But the common thing here in the workflow is that we follow the code, decide what to do next. Whereas in agent, the model decides what to do next. The control is basically the difference is who decides the next step. So, let’s say, now let’s take an example of an agent, right? So, the agents can be a single agent or multi-agent. So, keeping the single agent, multi-agent aside, but we’ll just talk about the agent first. In case of an agent, what happens is we still have this surrounding application, but the agent which has a model, what it does is basically has access to a set of predefined tools, skills and maybe it functions and the number of series of instructions. What it can do, is it has a set of goals and then it can basically figure out what the next step is.
Raju Dandigam 00:09:13 I mean get the user input and then process it and understand the user intent and then figure out what it needs to do the next step. Should it call a tool or should it call a skill and then understand what the user needs. And then it goes for a next step and then again comes back to the model and then again, it figures out the next step until it reaches the goal. In this case, I can go back to the same food ordering example and then can explain a scenario, right? Let’s say that going back to the same food order example where a user requested with a certain requirement where he wanted to have dinner for four. And then there are some constraints, a hard constraints, preferences, and then the input. So, let me put down the user input, right?
Raju Dandigam 00:09:52 For example let’s say that he wanted a dinner for four people by 7:15 PM under the budget of $70. And then he has a constraint of that one vegetarian meal he needs and then he has some allergy of peanut and then he also wants something mild and then maybe not pizza and then also, a restaurant which he liked. So, here I’m sharing a couple of constraints so we can understand how the agent is going to think. In this example, we have some of the hard constraints like a delivery time, which is basically the user wanted to deliver by 7:15 PM and a budget constraint, which is also a hard constraint which agent cannot decide. And also, a vegetarian option, which is basically there’s no option, right? So, it’s already decided and there is also allergy and then soft preferences, allergy information. Now we divide the constraints into hard and then soft preferences, right? So, we can now think through how the agent is going to do this. So, the agent first reads the input and then figures out what are the soft constraints and hard constraints then picks the next tool and then it basically goes on processing it. So, in this case, the workflow cannot do this level of decision making because it doesn’t have that freedom to pick and then go on to the next step.
Amey Ambade 00:11:08 So, seems like what you’re trying to tell me here is a workflow is very deterministic, whereas an agent doesn’t necessarily have to be. So, once we’ve decided that the model gets to reason about something here, the next question is how it hands its conclusion back to the rest of the system safely. So, let’s talk about contracts and let’s talk about the tool boundaries here. First, can you tell me why you would say freeform text is not necessarily the best way to show an interface between an agent and distributed system?
Raju Dandigam 00:11:45 So, the free from text is basically for the end users, but for the applications it needs structured data. The reason is that we give a free text to the application, then it has to do process again to understand what it is basically requesting like as a model started with something, maybe requested a tool and then got some output. Let’s say if the output is also, a free from text and the next model or the next step would need to again understand what it is talking about. So, the general recommendation or the system is basically application always needs a structured input and output, whereas humans can obviously need a natural language or maybe a UI to process it.
Amey Ambade 00:12:23 So, let’s make structured output here, very concrete and agent make a decision or picks a tool. What should that object actually contain? Can you talk about the outcome, the reason code, the confidence, all of those things here.
Raju Dandigam 00:12:37 So, the output actually should contain a minimum information. The application needs to understand and safely continue the workflow. So, let’s say again we are going back to the food aid system example, right? So, the result may include an outcome such as clarification required meaning that does it has enough information to proceed or not, or a recommendation ready, maybe it already has information and then is ready to proceed to the next step. Or maybe a card proposed or maybe a confirmation required or maybe it cannot complete or a fail code or a recent code. So, it definitely needs some kind of standard things like a reason code WhatsApp and the status and then some feature specific or the requirement specific properties and then the structured fields, which basically can the next step can understand.
Amey Ambade 00:13:19 So, can you talk about schema validation layers, just Pydantic or some type of JSON schema, right? Is there a rule of thumb what a model can vary and what it should not vary at all.
Raju Dandigam 00:13:30 The schema validation checks that the model output as the expected runtime shape. So, that means that for any tool or any agent specification, there will be a tied input and then the structured output. So, we are going to say that hey, you’re going to get these parameters, and I wanted this in the this structure. So, I wanted the output to be in this structure. So, the Pydantic is for the python and then the similar option in the, for the TypeScript we have the Zod. So, the JSON schema is a language neutral format. So, it can verify that and outcome is allowed, required fields exist. Then the types, basically the ID are strings, and the prices are in the number format and each edition type contains the correct fields. But the schema validation is only the first layer. It cannot prove that the price is current. Meaning like let’s say that it can only tell that the price is a number, but does it match the user requirement and does that price is the current live price that cannot be validated by the schema. So, that has to be the next layer, or the application code still need to perform the semantic checks and then verify the live prices or whatever the live data.
Amey Ambade 00:14:32 Can you give me an example where an invalid model output would cause real damage downstream if the schema does not necessarily catch it at first. What does that type of failure look like?
Raju Dandigam 00:14:44 Yeah, So, first let me give the variety of failures we can think through. So, there is basically a validation failure. It could be a policy rejection, a tool failure, or a delivery failure. So, the validation failure means the model output does not match the expected structure or the business in variance, whereas a policy rejection means the system correctly refused the operation, meaning the user did not authorize the payment or maybe the budget was exceeded. So, that is an expected outcome. That’s not a technical failure; it’s an intentional policy failure. Whereas a tool failure means the correct capability was selected, but it’s APA or dependency timed out or return and error. Whereas a delivery failure means the decision was correct, but the response or the notification or the downstream action did not reach its destination. So, these categories matter because they need different retries, time alerts or metrics or owners. So, in case of a tool failure or rather schema failure, so, let’s say that a schema we expecting a certain format and then if it doesn’t, the model does not written in that format, obviously it can either do a safe one or two retry to expect again, but it can obviously should have a safe fallback or it should actually let the user know or let the system know that it couldn’t get the expected format.
Amey Ambade 00:16:00 So, you are drawing a very sharp line between these failure types and invalid model output is a different thing from a tool failure or a policy rejection or as you said, a delivery failure. Why does keeping this separate matter for reliability and for debugging later?
Raju Dandigam 00:16:17 Yeah, keeping them separate gives a clear picture when we trace back to a failure or to a user session. So, let’s say that we figured out or maybe we found an error either in the user behavior or while the user using the application or maybe in the logs or maybe in the observability. Now the important point is where it is failed, like where the system failed? Is it failed because of a tool or is it failed because the policy is not allowed or is it failed because services are down? So, to understand them clearly, we need to first categorize them into different failures, the ones which I mentioned earlier. And then it’s easy to understand that I understand and identify where the issue happened and then basically figure out the fix.
Amey Ambade 00:16:58 I see. So, let’s talk about tool boundaries. Now I’m curious, agents obviously should not get unrestricted access to every type of tool. Setting the security side aside here from a reliability and correctness standpoint, what belongs in a tool contract? Can you talk about the input shape and the output shape and the failure modes here?
Raju Dandigam 00:17:43 Yeah, So, the tool contracts is basically the tool basically it starts with a name and then it has a description and then its schema which is basically, typed input and then a structured output. So, then also, tool at the same time, it doesn’t have all the access to all the tools, right? So, there are read-only tools and then the approval gated tools, basically what it can read and what it can write. So, the read only tools can usually run earlier in the workflow with a narrow risk. Basically, there’s not major risk that you’re just reading the data. They may derive menu, in this case an example case or maybe a restaurant availability or a delivery estimate. Whereas the approved data tools like create or change something. They require the current validation maybe as timeouts are an impotency and the explicit confirmation.
Raju Dandigam 00:18:26 So, we don’t want to update a payment method, or we don’t want to just process the order without the user confirmation. So, the tool cannot just do that. So, a readily timeout may be retried, we can try to get the data again, but a timeout after an order placement request is different because the order may already exist. So, at that point we might need the impotency ID. So, the system must check the result before trying again. And then the tool contact is basically we define the input and the structured output and then we can also define what is allowed to read and what needs a gated.
Amey Ambade 00:18:56 So, you’re saying that these read-only tools need to be designed differently than the tools that only unlock after these deterministic checks are running. How are you actually designing these different types of tools?
Raju Dandigam 00:19:08 So, as we discussed, the tools have basically a certain type. So, each tool has its own purpose. So, in a flow, like let’s say that the same example, we would need a tool to get the restaurant and then obviously we would, I mean, which eventually calls an underlying API and then get the restaurants and then filters out. And then we might need another tool to get the prices for each menu or maybe get the menus for each restaurant and then maybe check the availability. So, each tool has its own scope and purpose and then each tool has its own capability, what it can do or what it cannot do, or what the limitations. So, separating them and then having that clear description. And then clear boundaries are what helps the model to do what it can do. I mean, helps the model to figure out and then the process while handling with the tools.
Amey Ambade 00:19:56 How much of this is just good API design and type system discipline and how much is genuinely new territory that agents are introducing here?
Raju Dandigam 00:20:06 So, the underlying APIs and the contracts are still the existing application building logic we have been doing so far. The new territory of what the new system, what we brought here with the agents is basically how these agents are going to create the tools and then how these tools are going to communicate with the API. So, the new territory part is basically the communication between the agent and the tool. So, how the agent is going to construct the tool and then define the structure and then how those tools can be talk each other like in the next step, let’s say that we picked up such, we picked the restaurant and then how are we going to pass those restaurants into the next action? Or next step is to filter out based on the user preferences. So, that’s going to be the communication between these two steps and then how the model of the agent is going to use the first tool and then get the output and then pass it to the next tool and then get the set an output and then again pass and so on so on. So, this part is the new territory versus the existing API contract and the structure typed is all always there in the software applications.
Amey Ambade 00:21:04 So, let’s say you’ve got these clean contracts and very well scoped out tools. The interesting engineering here is how you compose them. So, let’s talk a little about orchestration. Production agent systems mix a lot of different patterns from my understanding routing, there’s prompt chaining, there’s loops and orchestrator worker coordination and human checkpoints introduced. How do you decide which pattern actually fits a task instead of reaching for agents everywhere?
Raju Dandigam 00:21:36 Yeah, So, I think I was briefly going in the earlier example, I’ll come back to that. So, there are two levels. So one, we have this simple alert feature and then we have these agentic systems. Within the agentic system, we have these workflows. So, workflows as I mentioned, that where the code decides what to be done next. So, within that, we can have different patterns, like we have a prompt chaining, we have routing based, and then we have the predictable workflows, and then we have orchestration patterns. And also in the agent side we have the single agent, we also, have the multi-agent. We have these coordinators or supervisor patterns. So, now how the orchestration or how we are going to decide when to use what is basically, we are going to use prompt chain when one step feeds into the another one, whereas we use routing when the request belongs to a clear category.
Raju Dandigam 00:22:24 So, I can give a simple example. Let’s say that as a user I may come to the same food order example, I may just come to create an order, or I may just come to talk about some order which I add issue with. So, in that case, it is basically a support agent. So, I clearly the intention is just the support requirement. So, in that case, the first step is the routing agent can clearly say that, hey, this is for a supporting agent. We can clearly separate and then send it to a separate support agent So, that way the routing will be taken care of and then the next step will be taken care of by these support agents. So, we also have this parallel execution for independent reads, and we’ll also have these evaluator loops where a result needs a bounded review and repair step. So, going back we’ll have workflows which decides the next path where we have defined steps but goes through certain model calls, certain tool calls, and then come to a goal. But whereas we have agents where the model decides what to do in the next step.
Amey Ambade 00:23:20 So, you briefly just mentioned coordinator patterns. I want to talk a little more about that. Walk us through this coordinator specialist type of pattern without selling it as maybe the only answer. What does a coordinator actually do and why should specialist return specific results rather than giving out prompts?
Raju Dandigam 00:23:40 So, let’s go back to the same food example. So, one coordinator can own the user’s overall dinner request. A coordinator, meaning that’s an overall agent. The overall goal of that agent is to basically fulfill the user dinner request based on the requirement given, the conversation or the chat message given. In that case it may ask a restaurant specialist for the current option, the coordinator, and then a dietary specialist for evidence about vegetarian and allergy needs and a delivery specialist for the timing and the fee information. These specialists may run as services, fixed sub flows or maybe a structured model calls or bounded agents, but they do not all need to be autonomous agents and the coordinator agent as a goal to get all the data and then come back to the user with a response whether it can be a missing data or maybe it is a proposed menu item or a proposed J item added into the cart. So, each return set type data, the coordinator combines the results and prepares a proposal like what food item they can order. So, the application still controls the payment and order placement, or maybe a human in the loop confirmation would be needed. Then the coordinator can take the action.
Amey Ambade 00:24:49 So, when you are having a set of specialists that are working, they’re working parallelly, right? And there is, I suppose, a lot of cost associated with running parallel execution. Can you talk a little more about that?
Raju Dandigam 00:25:03 Yeah, So, the parallel execution can reduce the wall clock latency, but it increases the peak model and tool usage. So, each branch may repeat shared context, create more traces, maybe fail separately and require cancellation or timeout handling. The slowest required branch still controls the final response. So, as a coordinator, it still needs to wait for the last branch to make a decision. So, it can also increase the cost quickly if several specialists call the large models at the same time. But parallelism is still valuable only when the tasks are truly independent and the latency benefits the overall user request. So, the point there is basically the parallel execution helps when they’re all independent and they use the relevant model and then basically come back without adding so much latency into the process.
Amey Ambade 00:25:54 What I want to talk about now is multi-agent design. By design, when you talk about multiple agents, why shouldn’t a team just go for a multi-agent setup before they can see what is happening inside a run? Why would me as somebody who’s working with agents always want to not spin up multiple agents at the same time?
Raju Dandigam 00:26:17 So, the multi-agent system makes sense when we have the clear division of what each agent can do. So, the multi-agent may not work in a situation where there is like, I mean there’s like two levels, right? So, basically if the request is clear, I mean meaning as a user, I just want to basically, same example, right? So, get a dinner request done. So, I don’t think we would need to send it to multiple agents and that would not help the user to get the response. So, the multi-agent system will help that saying that there is a separate dedicated agent is going to do that, we are going to hand over it to that agent and then the agent is going to take off from that point. So, the same example where a user coming for a support request will have a separate dedicated agent and that agent can take care from the point that he is coming for an issue and then he’ll handle the entire request.
Raju Dandigam 00:27:05 But whereas in a dinner request, it has to come back to the coordinator agent or the supervisor agent who takes care of that and then get all the information and then respond back to the user. So, in this case, we could still have a multi-agent within that system, but the individual agents cannot actually fulfill the request. So, what I mean to say, when we have a clear division of the agent and that can take the request from the starting and then come up with a response to fulfill the user request, then yes, the multi-agent system will work and then we basically route them accordingly and then the multi agents will take over from there. But if there is a request from the user and the user wanted response after collecting so many things and then accumulating them and then coming back, then we would need still a single agent which is responsible to respond to the user.
Amey Ambade 00:27:53 Very Interesting. Have you seen examples where multi-agent design actually made things worse, and you had to collapse back into something much simpler?
Raju Dandigam 00:28:03 Yeah. So, in one of the production workflow, we had a model-controlled routing step. So, that selected a specialist for a critical path and the traces showed that the application context already determined which specialist was required in almost every case. Then the router added another model call and the more latency and a new failure path without giving us meaningful flexibility. So, we removed that routing decision and invoked the required specialist deterministically. The specialist still used the model reasoning where it added value, but the application controls the known first step. So, the lesson was that not every delegation decision needs to be agentic. So, the determining still helps and then we can still have the agent take the realistic step and then continue with the remaining model listening.
Amey Ambade 00:28:48 Interesting. Yeah, so that’s the last point you just made is seeing what is happening depending on the agent running as a durable system, not just as a single request, right? Let’s talk about this more. I want to explore runtime a little. So, an agent becomes a production system when it runs beyond just one single request and response cycle, right? What does that force you to build that a stateless endpoint doesn’t necessarily need?
Raju Dandigam 00:29:14 So, we would need a durable execution state. Checkpoints run ID impotency, cancellation, timeout handling and resume mechanisms, and a way to report progress after the original connection is gone. So, a stateless endpoint can fail and ask the caller to retry, but a long running agent may already have called several tools and created external state. So, the recovery must understand what has been completed and what it is still safe to repeat. So, having the state or the memory will have that clear picture of what is being done, what is being retread and what is missing and what is pending. Whereas the stateless endpoint, we may not have that visibility or may may not have that clear picture on what is failed and what needs to be retried.
Amey Ambade 00:29:59 So, I can think of a very clear distinction between the active state versus your long-term memory for the agent versus the context that is shared. Define each of these and why mixing them can cause trouble.
Raju Dandigam 00:30:14 So, the active state is the data needed to finish the current run. So, like you started a thread and then you started a chat and then you are talking to the agent and then you are working on some request. So, that’s the active current state, active state. So, the request, the constraint and the tool results, the proposed card maybe in this case and the retry count and the pending information, this all comes under active state, whereas a long-term memory is the information intentionally kept across interactions. So, it may be a new interaction, user might come back again. So, in this case it could be such as like an approved preference as a user preference or maybe a user choice, which he would’ve mentioned while he was chatting. So, in case of shared context, which is basically a trusted information, which is made available to several steps, and the specialist is basically across the application, which could be a user profile, the account configuration or a payment method or maybe a user details which are being shared. So, mixing these basically, mixing these memories will not help, but we have to use accordingly. So, we would need an active state when we are running the application, but when the user comes back, we would need a long-term memory or the shared context to get back where the user left and what the context was. So, those need to be used accordingly.
Amey Ambade 00:31:29 So do all three of these have different privacy implications and retention policies?
Raju Dandigam 00:31:35 Yeah, So, while obviously storing the memory in the long-term memory, so, obviously we need to make what we can be saved and then what cannot be stored as based on the user privacy constraints. And then what are the privacy settings. So, we would definitely not save the PII information or maybe the privacy information, which is specific to a user and which is not allowed or which is not confirmed by the user that we can store. So, each has its own limitations on what can be stored and what can be masked or what maybe what can be added as a shorter memory versus long-term memory.
Amey Ambade 00:32:09 And how do things change when you are doing long running work? Would you argue that the request path should not block or the reasoning and the tool calls when work is long term? And can you walk us through that pattern of working with long running work streams?
Raju Dandigam 00:32:25 So, the long running work streams is basically the agent takes the input and then figure out what it needs from the user if there’s nothing from the user and it has to figure out the tools and the function calling maybe a database operation or whatever it needed to get the details and then get the prices or maybe get the live data from the various vendors or maybe various downstream services. And then it has to repeat the iteration until it gets the information it needs to proceed to the next step. So, in case of failures, or maybe in case of timeouts or the delayed from the downstream or maybe the external services, it has to decide should it con keep on running? Should it continue to do the iteration or retries or should it basically understand or maybe have a safe fallback. So, the conclusion or maybe the decision depends on what and how many retries or what, how many timeouts are allowed and depends on what kind of tool it is.
Raju Dandigam 00:33:21 Let’s say that it’s a retool, we can say that, hey, you can do a one or two retries or maybe you can actually try until you get it, but I then infinite loop or iterating continuously will not help to the user. So, having the safer retries for a safer tool and then obviously for the right operations, it’s not recommended to have the retries, rather it is safe to say to the back to the user that’s saying something is not working or something is failing. And then the alternate options. So, in case of long running, So, there are defined retries and timeouts and then there are defined not to retry and then safely fall back and then let the user know. So, depending on how do we configure and what we led the model to do, what are the boundaries? Then it goes onto that long running cycle.
Amey Ambade 00:34:05 So, when you have an agent and you need to make a decision on whether you would have a synchronous call or a queued and an asynchronous call, is there a latency threshold that we need to keep in mind when we make that decision in practice?
Raju Dandigam 00:34:20 So, the synchronous and then asynchronous, the decision depends on the how much time or how much latency each call or each agent processing takes. So, I would keep a workflow synchronous when it’s P95 which is basically the percentile 95 can stay under roughly J210 seconds and the path is predictable and it does not need a durable waiting or a human approval. If it is likely to exceed 10 to 15 second call. Several variable external provides survival, disconnect or wait for an approval is better to return a run idea and use a progress model. Let’s say that in an example of food assistant, maybe the practical starting goals would be a request acknowledgement under 500 milliseconds would be good starting point or maybe a first useful progress within one or two seconds. And the first usable results within three to five seconds and the P95 completion around 10 to 12 seconds would be the good starting points may not be the standards, but the product experience end of the day determines the actual target. So, as a user, I asked for a dinner request and what is reasonable for the user to wait and what are we presenting? So, that defines what needs to be synchronous versus what needs to be asynchronous. And then there are scenarios where we just wanted to run some asynchronous job and get back to the user while the user is using the application.
Amey Ambade 00:35:38 Do you have any opinions on streaming? Users may want to see the progress of the agent before the actual answer is ready. Do you insist on a specific setup for this?
Raju Dandigam 00:35:50 So, it going to be a combination of things, right? So, the UI should receive an immediate acknowledgement that the agent or the application is processing. So, that’s the first thing as a user I would like to see as I type in or as I interacted with an application. And then we would definitely need to show the products, lifecycle events. What I mean by that is basically, we can definitely show the searching currently open restart or maybe searching for the menu, but that has to be defined by the product or approved by the product. And then it is basically a, a user facing text, which we are showing it as a progress to the user and we can have a structured cards as a result or maybe a rendering of the loading cards or something as they become available. And the text has to be definitely validated by the product or the respective teams.
Raju Dandigam 00:36:34 So, in case of a restaurant, the food ordering example, we can say that searching currently open restaurants or maybe checking vegetarian allergy information when the agent is actually going for a tool which actually gets the user allergy information and maybe we can show that we are searching for a restaurant menu cards or maybe rechecking the current price and so on and so, forth. This relevant and valid information help while the agent is working behind the scenes to get the data and respond to the user. So, the token streaming can make explanation feel responsive, but it should not expose a raw reasoning. It cannot say what agent is doing behind the scenes, or it cannot show the internal specialist messages or maybe subagents or the raw text or the sensitive tool payloads or maybe the unvalidated input. So, it has to basically have safe and valid text.
Amey Ambade 00:37:22 So, what if you have an agent that is streaming responses and it fails mid-run? Do you do some form of checkpointing there?
Raju Dandigam 00:37:30 Yeah. The checkpointing, we can safely fall back to the previous safe checkpoint and then tell the user that there is, I mean basically show the user the fallback error or maybe the exact error what happened, but in a user readable manner. And then propose alternative options like try again or maybe a look for another option or a safe alternate options basically.
Amey Ambade 00:37:52 So, you’ve got a new system now that runs, it holds its state, it is a streaming based system, it has checkpointing set up. The question I really want to dig into now is how you know any of it works, how you debug it when it doesn’t work. So, let’s start with talking about evaluation and testing. Which parts of an agent should be tested like ordinary software, and which parts need a completely different approach and what would that approach be?
Raju Dandigam 00:38:19 So, we have now there is like this application surrounding application logic and then we have the model or the AI agent logic. So, anything deterministic should use the ordinary software test. So, that includes the regular unit test and then the integration test and the usual testing CICD runs and everything. So, those things still goes under the same usual regular testing that includes the permissions check policy checks or maybe a jaw or Pydantic schemas or semantic in variance, budget calculation, tool adapters, everything still goes under the regular software testing and these tests should be fast and then should not require a live model. They can just, they are the one vastly covered and they, in the pyramid, they actually take the majority of the testing and which parts need the be behavioral evolution is basically the ones needed where the model judgment matters.
Raju Dandigam 00:39:10 So, meaning if there is something we want to test what the model is generated or what the model is giving back. So, that needs behavioral evaluation, which is specific to the agents or the workflows. So, which includes, there can be examples like whether the agent asked for a car clarification question. So, let’s say that the same example, the user gave some requests and asked for a dinner request. The first thing the agent has to know is whether the address has the valid address to create the order. So, whether the agent asked for the clarification as soon as the user requested something. So, that can be evaluated or the agent selected the right tool when the process started meaning like when it started to fit the restaurant or does it follow the dietary constraints, which user has mentioned, does it follow the allergy information? So, these are all the evaluations the testing needs to be done over the model judgments. So, these evaluations are done at the tracing level, like each and every step level and as well as the overall outcome level.
Amey Ambade 00:40:11 So, your test is actually checking whether the response is satisfying the contract is calling the right class of the tools, is it refusing forbidden actions or not? And is it asking for clarifications when there is context that is missing, right?
Raju Dandigam 00:40:25 Correct? Yeah.
Amey Ambade 00:40:26 Cool. So, not every test should be called a real model. How do you mix mocked responses and cache responses and do live smoke tests? What does each one of these kinds of cache?
Raju Dandigam 00:40:40 So, most determining success should not call the real model. So, these are all regular tests, but the live models are used when the model behavior is the thing we are being evaluated. So, the otherwise the test become slower and more expensive and less reputable.
Amey Ambade 00:40:55 Expensive in the sense of?
Raju Dandigam 00:40:57 In terms of the cost, let’s say that we are using a real model just to test the deterministic part. That doesn’t help much where we can have the regular unit test or maybe regular integration test to come up with the mocks and then come up with the assertions and then we can validate that easily. So, there are not effectively using the model, but there are scenarios where we wanted to use the model as the testing, which is basically LLM as the judge. So, what we could do is basically, I mean there are two ways to do the behavioral evaluations. One, we are going to have the expected input or the input and then the expected output in it evolves. So, what it does is basically you say that, hey, this is a user input and then this is the output I would need and I’m going to mock all the tools, the tools, which agent is going to call in real time, but I’m going to mock all the tool responses and everything, but I’m going to keep the model live.
Raju Dandigam 00:41:46 So, the model is real, but it’s going to, it is going to have all the mock data, which is needed, and then I’m going to see what it going to output. So, let’s say that for the same example, the tools is, I’m going to go have a search the restaurant as a mock and I’m going to have a prices as a mock and everything, the checkout, the cart, everything information as a mock, but then the, I’m going to pass everything to the model and then see what the model is going to output when I give this particular input. So, is it going to actually come back without the menu or the items or is it going to fail somewhere or is it going to pick the right tool, right action the next step. So, this is what is called mock testing for the models, which is basically mocking the evolve. And whereas the other part is basically we can again use the LLM to judge the model behavior. So, now the same scenario, and once we get the output, we can pass that output back to the LLM saying that, hey, I gave this input and then I got this output. Does it make sense? Does it actually, does the output follow what user is requested and does it follow the standards? And how does, what do you score this one? So, those are the two ways we would do the evaluations.
Amey Ambade 00:42:53 And testing obviously for any software engineer goes hand in hand with debugging. Does debugging change with agents instead of becoming event debugging? Does it become workflow debugging? And how do you actually debug a workflow?
Raju Dandigam 00:43:06 Debugging workflow is different. I mean it’s not different, but it has its own steps. So, debugging can basically happen at various levels. I mean the debugging will reveal various issues at various levels. So, let’s say that a workflow has a pretty defined path, which is basically defined by the code that hey, you get this data and then go to the next step and then do this one. So, that’s a workflow and then that is little easier relative to the agent. So, in the workflow, we kind of know what is the next step and where it stopped by using the trace. So, we can be clearly see the logs that, hey, it entered into a searching restaurants and then it picked the restaurant and it picked the menu, but it couldn’t proceed to the card because it didn’t have the live price.
Raju Dandigam 00:43:48 So, that’s like a predefined step, but whereas an agent, so the next step is decided by the model. The debugging also becomes trickier because we wanted to see what step it started with, which tool it picked up, and then does it have the expected output and then does it follow the next step according to the plan? Or is it like pick up the wrong tool or pick up the wrong next step? So, the agent debugging basically depends on the path, and then it may not actually have the same path for each and every request. Sometimes user may provide all the information needed to execute the request, meaning like same request could have everything needed, like the address and everything, all the exact details. But some of the requests may not have everything. It may just say that I want to have dinner, but then user has to go onto the loop getting all the information and then that debug is completely different from the other one where we had everything in place. So, the agent debugging actually has to have a full trace and then figure out each parent step and what are the nested steps the child steps and then wear it again, go back the complete trace.
Amey Ambade 00:44:50 You mentioned logs. Tell me more about them. What does production grade agent observability actually need to capture? Does it only need to capture the model version and the retrieved context, or does it also have to log reasoning and sensitive data?
Raju Dandigam 00:45:05 So, the log is going to contain to start with their, each log is going to have a minimum information and then depends on the feature, depends on the scenario. It may have more, but at least to start with each log would going to have a run ID or recession ID, which is going to be common across entire user trace, meaning like I started by conversation and from the starting point to the end the conversation what have been happened, each has to be tracked by the singular ID so, that it can be used across the services. Like let’s say that the agent started with a user request and it has a trace id and then agent went to make an API call by using a tool. And then the API also, should have the same run ID, which can be used to deep dive into that run and what happened exactly with it’s API.
Raju Dandigam 00:45:49 So, the run ID at the station ID is the one used across the trace, which gives the entire trace. And then it also, need to have the agent, which agent it picked up and what’s the workflow and what is the prompt version, model version and the schema versions. And also, it should have the retrieved context references and the important routing and state transition, what tool called it made and what result it received, what are the validations, does it have, has it done any retries or any fallback decision it has made? What are the latency tokens, how much it used, and then what was the estimated cost? Does it ask for a human or what is the outcome? What is the input? So, these are the few things as each and every log would need, but the idea here is to basically explain the user path without adding the unnecessary sensitive content. So, the other question is basically, do we need to put everything into the logs? Obviously not. So, we have to have a PII safe, logs, meaning there should be a wrapper or some other way to not put everything into the logs. So, the information which is safe needs to be sent to the logs.
Amey Ambade 00:46:51 Can you quickly clarify what PII stands for?
Raju Dandigam 00:46:55 Yeah, So, the PII is basically the personal information, which we want not to send to the logs or to the models, right? Those information is basically need to be either stripped out or maybe removed from the logs or maybe it can be masked or used ID, which is basically to represent them that user ID has something to represent them. And then obviously we would definitely not want to send the name, email or any personal information to the logs or even to the model.
Amey Ambade 00:47:23 Have there been bugs that you could only find by looking at the logs and at the execution tree and you know where there’s been multiple similar runs, but they took different paths and changed the outcome completely?
Raju Dandigam 00:47:35 So, in one of the multi-step production flows, passing step failed instead of stopping. So, it produced an empty result and allowed later steps to continue. But in another run, the passing succeeded and the workflow used the correct context. On another similar run passing failed. Iteration resumed from a slightly different checkpoint, and the workflow followed the default path with the incomplete state. So, the final output looked like a model quality issue, but the execution tree showed that the first real problem was the silent passing failure, and that changed the retry path. So, when we changed the pass to fail clause and made the state transition explicit, improved the trace and added the scenario to the regression. So, what I’m trying to say is basically that, any one of the steps which is basically open and then the passing was required, it picked up a path, but when we say that this is no required and it’s like closed and then it picked the right path. So, yeah, the debugging or the log trace would help you to find the bugs and then it picks a different path based on the state.
Amey Ambade 00:48:38 Very interesting example. So, the payoff is the failure attribution that you get through these logs. But when a run goes wrong, how do you tell whether the cause was the context or the prompt or the model behavior or a tool, what are you thinking about when you’re making that decision?
Raju Dandigam 00:48:57 So, to figure out that where and what caused the issue? So, I would first find the first point where the actual run difference from the expected run. So, let’s say that we wanted to, in a given example, we wanted to say that hey, the agent has to pick this, pick the details and then call so and so tools, and then move on to the next steps. So, in that path, if we figure out where it started into the unexpected path. So did the system, in that point, we first need to figure out whether does it has the right context, did it use the expected model and the prompt version to start with? And did the model choose the correct tool? And were the arguments valid? Like the arguments what we passed to the tool are the valid and did the tool return the right data and the data and the schema and the business validation path and the, did the graph choose the right next step and at the end does the delivery succeed? So, in this case, we will go each step by step. So, we figure out what started with and where it actually followed the path and where it actually missed the path, and then what all it picked up after that. So, that would actually give the root cause where it started, and then we figure out who is responsible for that failure. It could be a tool, it could be agent or it could be a model, or it could be an application logic, or it could be a downstream API service failure.
Amey Ambade 00:50:16 So even with all of this, an agent that works completely fine can still be very expensive or become nobody’s job to maintain, right? Let’s talk about operating in production really quick. Agents can get expensive very quietly with long context or with retries. And all these parallel calls that we just discussed about and model routing, do you insist on measuring cost and latency at the run level or at the service level?
Raju Dandigam 00:50:42 Each model call may recent part of the conversation or the system instruction. So, the parallel branches multiply those calls, but evaluator loops add more model work and then retry repeats the same or similar work. So, the model routing may introduce an extra decision or a call before the real task actually starts. So, the cost of funds goes through many individual reasonable choices rather than one. Obviously, the expensive operation. And the same is true for the latency also. As we introduce tools, models or agents, the latency also increases. So, way to measure the cost depends on the user’s experience. So, as a user, the user experience is one complete task. Though we have multiple layers, multiple steps behind the scenes and the agents and the workflows running. But as a user, we are going to have as one complete experience that the user entered and requested for something and what he received.
Raju Dandigam 00:51:35 So, I would say as a user when entered and the run may include a routing, a specialist, a tools and a repair attempt or maybe evaluator, and then looking only at the individual calls or maybe individual tools that hides the real cost and the response time. So, the run level measurement lets us compare the resources consumed with the final outcome, but the useful order, did we get the useful order out of that? Did the user complete the task? Did the agent require multiple data? We don’t know all these things; we just look at the runtime. So, but if we look at the successful task or the overall service level, then we will understand what happened through the entire task. Meaning the user asked for something and then he got the response. If the response is valid response, then we can compare how much the latency was, how much the cost was. So, in my opinion, it is at the service level is what helps to measure the cost and latency So, that way we can say that for one successful response and an average, how much we spend and how much was the latency versus we go runtime picking a tool or maybe picking an agent or picking a model may not help because it varies, depends on each request.
Amey Ambade 00:52:41 And you know, sometimes the right engineering decision is to use a simpler model or a deterministic rule or no AI path at all. How do you make that call?
Raju Dandigam 00:52:52 So, if there is something deterministic and there is something a code can decide, obviously we don’t need a model, but if there is something we don’t know what need to be done or maybe something changes as the system learns. So, that mean, what I mean here is basically a user give a request and if you know that, if it is just a reorder or if it is just a support request for a refund, that’s a straightforward code path logic, which may not even need a model or maybe I need a little of model. And whereas we user gives him open-ended questions and then wanted to do something which needs lot of processing. And then decision making as it goes through, as it learns the current system, meaning like gets the prompt, gets the input, and then figure out that hey, there’s like an open or maybe open menu at the live prices and then goes back to the user or maybe figure out that there’s only few available in the address or maybe in the radius. So, that kind of decision making and the process definitely needs a model and agent. And then we would need to use the models accordingly, whereas a simple, straightforward might not need a model or maybe a deterministic code path would solve that problem. So, it depends on the requirement on this scenario.
Amey Ambade 00:53:57 And who actually owns an agent’s behavior after it is launched. What happens when something goes wrong and you are supposed to find out who the owner of the agent is? What would you say about that?
Raju Dandigam 00:54:12 As a user? So, we wanted to have a best experience for the user and then the expected output. So, the end for the user, it’s basically the experience. So, going back from the application side as a product owns the entire user experience. So, meaning, what user is coming to for the application, what is trying to do and what is getting out of it? Like what is getting at the end? So, that’s the user experience, which product will be owned. But when it comes to the agent’s behavior, it has multiple parts associated with it. It has the application logic, which engineers own the data, administrative logic and all the infrastructure tooling. It also has this behavioral part or the prompt part, which also decides what’s going to happen to the user. Like what’s going to outcome, right? So, the prompt or maybe the model versioning or the model two model version and the prompt version is owned by the not owned, but it is collaboratively owned by the product.
Raju Dandigam 00:55:06 The reason it is because the prompt and the model is going to decide the user response and the user response is what the user experience which should owned by the product. So, to get the expected user response, the product should define what it has to give as a prompt and what it has to expect and to maintain that or to, handle that, basically we would have like a different versions or the safe prompt versions and then save release so, that it’s not having any impact, but at the same time have the control to the product so that they can change. So, to summarize, what I’m trying to say is that basically product owns the prompt and anything which impacts the user directly, whereas the application and the engineering logic, the application logic will be the engineering team and then the overall model support and then the cost and the budget, everything owned by the infrastructure or the platform team. If they’re responsible for the overall models.
Amey Ambade 00:56:00 That’s great. If you could give one piece of advice to a team that’s trying to put their first agent into production, what would that be?
Raju Dandigam 00:56:06 So, my advice would be start with boundaries for an agent, not the autonomous. Meaning give the agent only the freedom it actually needs to solve the problem. So, before asking the agent what it can do, define what it is allowed to see, which decision must be remain deterministic, which tools it may call, and which actions still require human confirmation. So, then design the failure path as carefully as the successful path. Validate every important output, boundary, right and cost and provide a safe fallback and make each capability easy to disable if there is a fail or error against.
Amey Ambade 00:56:42 Awesome. Well Raju, thank you for your time today. Before we wrap up, would you like to direct our listeners to where they can find out more about you and your work?
Raju Dandigam 00:56:50 You can find me on the LinkedIn as Raju Dandigam and also, I’ll be active as a maintainer and the GitHub with the repositories like agent inspect and then all other NPM libraries. And then reach me on the email as well on my LinkedIn.
Amey Ambade 00:57:04 Well Raju, thanks so much for speaking with Software Engineering Radio. Thank you for listening. I’m your host Amey Ambade. See you next time.
[End of Audio]
