Sowjanya Pandruju, Cloud Application Architect with Amazon Web Services, Vice Chair of the IEEE Computer Society Madison Section, and Chair of the IEEE Women in Engineering (WIE) Madison Section (Region 4), speaks with SE Radio host Gavin Henry about managing agent state. Sowjanya discusses what agents are and are not, what agent state is, the role an LLM actually plays, why agents need state, the different types of state, where to store agent state, how multi-agents change things, and the solutions to the various challenges created by managing agent state.
Brought to you by IEEE Computer Society and IEEE Software magazine.
Show Notes
- SE Radio 733: Max Corbridge on Securing AI Agents
- SE Radio 730: Birgitta Boeckeler on Harness Engineering for AI Agents
- SE Radio 728: Clare Liguori on the AWS Strands SDK for AI Agents
Other References
- Securing Agentic AI: A Comprehensive Threat Model and Mitigation Framework for Generative AI Agents
- Framework for Enhancing Fairness and Transparency in Multi-Agent Creative AI Systems: A Position Paper
- Sowjanya’s LinkedIn
- LangGraph, particularly for stateful agent workflows, checkpointing, and persistence
- Semantic Kernel, for agent orchestration and integrating agents with tools and application state
- Temporal, for durable workflow execution, retries, and recovery
- Model Context Protocol (MCP), for connecting AI applications with tools and external systems
- Vector Databases Aren’t Enough: Why AI Needs Multi-Modal Memory Architectures | HackerNoon
- COSTAR: Cloud-Observed Safety and Trust-Aware Agentic Reasoning for Enterprise Workflows
Transcript
Transcript brought to you by IEEE Software magazine.
This transcript was automatically generated. To suggest improvements in the text, please contact [email protected] and include the episode number and URL.
Gavin Henry 00:00:18 Welcome to Software Engineering Radio. I’m your host Gavin Henry. And today my guest is Sowjanya Pandruju. I think I’ve probably pronounced that wrong. Would you like to correct me on that please?
Sowjanya Pandruju 00:00:30 Hi Gavin. I’m Sowjanya Pandruju.
Gavin Henry 00:00:33 Perfect, thank you. You’re a cloud application architect with Amazon Web Services, and you’re also an IEEE Senior member, and you serve as the Vice Chair of the IEEE Computer Society and Chair of the Women in Engineering R4 Madison Section. You also have published peer-reviewed technical work on AI and Cloud Architecture in IEEE Conferences and Computer Magazines, and you’ve spoken at industry and technical conferences. Welcome to Software Engineering Radio. Is there anything I missed in your very impressive bio?
Sowjanya Pandruju 00:01:05 No, thank you so much for the wonderful intro.
Gavin Henry 00:01:08 Excellent. Well, today we’re going to be talking about managing agent state, so I’d like to lay some foundations because it would be a good way to start and I’m not too up to speed on these things easier. That’s why I really look forward to speaking to you. So, let’s start with an overview of AI agents and then we’re going to chat after that about agent state, why agents need state, where agent state is stored and then some of the challenges and how they’re solved. So, what is an AI agent and what is not an AI agent?
Sowjanya Pandruju 00:01:44 So I think the simplest way to define AI agent is a system that can take a goal, it can reason about what needs to happen, it can interact with tools, it can observe results and then it can decide what to do next. For example, if I have to plan a trip, let’s say I want to say I want to go to New York and this is my budget, a typical traditional app, it might have a predefined workflow like you know, it could look at searching for flights or searching hotels and then it can present me with those results. With AI agents, it can have more flexibility, it can think about, it’ll ask me, hey, when are your travel dates? Do you have any flexibility? I can also search for flights but also compare options and if I give a travel time constraint, it can search again and it can decide what’s the next appropriate action I would say. So, there’s this loop that I want to highlight with AI agent is that it can observe, it can reason, it can act and then observe again until it finally reaches the goal or you know, until it determines that it needs more input or things like that.
Gavin Henry 00:03:14 So in your example, it’s much more than just a search tool.
Sowjanya Pandruju 00:03:18 Yes, right.
Gavin Henry 00:03:19 In your example, could you break down what that AI agent could possibly be? Is it like a binary that runs on a server somewhere or is it one component of a bigger thing or how would you define an AI agent in general terms? Because I know there’s different ways to do one.
Sowjanya Pandruju 00:03:36 Right. It can be any of those things. I think it helps to separate the logical concept of an agent from how we actually deploy it. So, at implementation level, an agent is really software. It could be a Python application that could be running on a server or a container or some serverless application or it could be inside a very big, larger enterprise system. So, there isn’t like a necessarily a special executable that would say this makes an agent. What makes an agent is the behavior or it’s the architecture around it. So, you have some kind of orchestration code or agent runtime, that code has a goal and constructs for an LLM. So, this model helps determine what should happen next. Like the runtime may invoke a tool, then maybe an API or maybe a database, a search service and things like that. So, you can picture this loop, right? Like there’s applications and context to the model. The model would propose an action, the application would execute that action, get an observation back and that cycle continues.
Gavin Henry 00:05:00 You mentioned LLM, just for anyone that’s not familiar, what does that stand for?
Sowjanya Pandruju 00:05:04 It’s Large Language Model.
Gavin Henry 00:05:06 Thank you. And an AI, something that isn’t an agent would just be something that doesn’t have that adaptability and that feedback loop. Is that correct?
Sowjanya Pandruju 00:05:17 Yes, exactly. So, a traditional application we would have to say first to A then B and if B returns this, then do C. So, the developer has essentially encoded those possible paths into the application. But with an agent we are giving the system more flexibility to determine the next step based on what it observes. It can take an action, look at the result and then adapt its next action rather than simply following a completely predetermined sequence.
Gavin Henry 00:05:55 The first way you describe would be like a rules-based program or system and then the way you’ve described an agent there, what it would do next would be based on the large language model has learned, as for what the next probable outcome would be or is it access that agent has to various things to answer the next step. I’m trying to not say put words in your mouth of things I’ve heard like skills or when you’re using an agent in an IDE or something like that, I’ve seen it pop up and say, you know, once you’ve asked something it’s like go how would I architect this? You can see it saying, oh, using skill, file, read or something like that. Where do those types of things fit in with the agent? Are they things that are configured for what the agent’s allowed to do or should be doing to answer your question or how does that work?
Sowjanya Pandruju 00:06:44 I think there are two different things happening that are useful to separate. So, the large language model provides reasoning capability. So based on the goal, the context it has and what it learned during training, it can determine that a particular next step is appropriate. But that doesn’t mean the model itself can perform that step. So, the agent runtime gives the model capabilities, it can act through. So, the ID example you mentioned, right? Maybe the agent has access to a tool for reading a file, another for searching like the code base, another for running tests. Those tools are exposed to the model along with information describing what they can do and how they can be used. Let’s say if I ask why this test is failing the model might reason for, I need to inspect this file, it’s selects the file reading tool, the surrounding application actually executes that tool and returns the result of the model and things like that.
Sowjanya Pandruju 00:07:55 And also yes, as you mentioned, the capabilities can and should be constrained. The application determines which tools the agent has access to, what permissions those tools have and potentially whether certain actions require approval. And then things like skills, they can add another layer. I tend to think of skill as instructions or specialized knowledge. So, a skill might tell the agent the preferred process for reviewing code or a file reading tool actually lets it read the file. So, I wouldn’t say the large language model simply predicts the next outcome. I’d say it uses the information available to select or propose an appropriate next action. Well, the framework, the Asian framework determines what actions are actually available and the application and security boundaries determine what it’s allowed to execute and things like that.
Gavin Henry 00:08:56 Yeah, that’s helped me really understand the LLM side of things because I wasn’t sure what part that played because everyone talks about, oh the next version of this out is really powerful but the way you explain it makes much more sense as to how it can supply reasoning. But there’s still things that need to be actually called in to act upon those things. Thank you. Going back to permissions, in my example of the IDU, it would then go, oh, can I read your files, and you have to click yes or no or click something that says just do everything for this session or you’ve got full permission. So, I do understand that bit as well. So, I’m going to move on. I think that was fairly high-level coverage of what an AI agent is and what it is not. But not going too deep at this stage. I’m going to move us onto agent state. So, let’s define state what is agent state?
Sowjanya Pandruju 00:09:45 Okay, so the simplest way I would define agent state is the information the agent needs to carry forward so it can continue working correctly. Let’s use the ID example we were just discussing. Suppose I ask an agent, find out why this test is failing and fix it. The agent could read a file, it could discover something, it could search another file, run a test, get an error and decide what to do next. So, at each step it needs some awareness of what has already happened, what was my original goal, which files did I looked at, what did my test return, what changes did I make? Where am I in this task? So, all of that is part of the state of that agent’s work. And one thing or one distinction I would say is that state is broader than just conversation history or memory.
Sowjanya Pandruju 00:10:46 We tend to hear age in memory a lot, but that’s really only one type of state. It could have conversation state, which is what we’ve been discussing. You could have a task or workflow state which steps have completed. You could have tool state such as whether an action has already been performed. You might have longer term memory like something an agent learns from an earlier session and, you would have security context who’s making the request and whether they are authorized to do so. So, when I think about agent state, I don’t think of it as one big piece of memory that is attached to an LLM. So, I think of it as I would say collection of information of the overall agent system needs to maintain so that it can maintain continually, make the next decision correctly and know what has already happened.
Gavin Henry 00:11:49 What sort of resources would, just to go back to my example, because that’s what I’ve experienced, I’ve seen when it’s going through using tools and responding to my question, it can be quite a delay. I’m specifically talking about the copilot local agent. I can see it saying, oh, patching session log. So, to me that looks as if it’s updating a text file or sometimes, I’ve seen it’s doing an SQL update or something. So, it’s either my understanding is writing to a text file log or updating a database to keep that state updated. Is that correct?
Sowjanya Pandruju 00:12:24 Yes. Conceptually, that’s correct. I am not looking at that specific copilot implementation. So, I wouldn’t want to say exactly what patching session log means, but what we are observing is a good example of, I would say externalizing state. So, the agent can’t rely on LLM itself to permanently remember everything that’s happened. So, the surrounding application needs some mechanism to persist information outside the model. So that could be a file, a database, a session store, and some other durable storage mechanism, right? So, I would say yeah, a text file or SQL database could absolutely be mechanisms for storing state, but I wouldn’t equate a log specifically with the complete state of the agent. A log may simply record what happened for recovery or observability while another store holds the current state.
Gavin Henry 00:13:28 So, it could just be stored in memory or written to desk. It just depends on the actual implementation?
Sowjanya Pandruju 00:13:33 Yes, it depends on the implementation and importantly how durable that state needs to be. For something very temporary, say information that I only need for current execution, it could simply be held in memory that’s fast, simple. And if that process terminates, the data is gone. But if the, let’s say the agent it needs to survive a restart or it has to continue a long running task later, that will generally externalize that state somewhere more durable like a file, a database or some system. There’s so many possibilities there.
Gavin Henry 00:14:10 Yeah, and you have got me thinking about when it goes back to the LLM, so I’m just being a bit naive here. So, if the agents state, and I think we’ll get onto this in the next couple of sections, the agents saving at a state of different things it has access to and then it goes back to the LLM saying, oh right, we’ve done this, I’ve got this new information, can you give me the next recommended type step? Is there a danger that it’s summarized or recorded a state incorrectly and it’s giving the next step of process the wrong initial information? Because obviously you said the LLM can’t remember anything or if I misunderstood this?
Sowjanya Pandruju 00:14:47 It’s a very good question and it actually one of the important challenges in managing agent state as well. So, I would make this distinction between the authoritative state and the state that we give back to LLM. So, let’s use a coding agent example. You have the agent that runs tests and three of them fail. The actual test results are information produced by the testing tool. But perhaps to avoid sending thousands and thousands of lines back to the LLM, the application itself could summarize those results. So, if that summary incorrectly says only two tests failed, then the next model invocation is reasoning from incomplete or incorrect information. So, the model might make a perfectly reasonable decision based on the context it received, but the point here is that the context itself was wrong and you are correct about the LLM not remembering in a traditional application sense too on the next invocation, the application essentially has to provide the relevant context. Again, here’s the goal, here’s where we are, here’s what happened and here’s the information you need to decide what comes next. So that’s why I wouldn’t want to generate a summary to become the only source of truth for important state. We can use summarization to make context manageable, but ideally, we preserve the underlying authoritative information separately. Then when something is important like saying whether a deployment actually succeeded or whether a payment actually occurred, we can go back to that authoritative system rather than trusting a potentially lossy summary.
Gavin Henry 00:16:37 So agent state is absolutely critical really, isn’t it? Yes. Perfect. So just to finish off this section, can we decide what things an agent remembers? For example, are we configuring a framework here and if so, how do teams decide that? How do you sit down and decide what an agent needs to decide if you’re not using an off-the-shelf tool?
Sowjanya Pandruju 00:16:59 Yes, and I think this is where we move from the concept of state into actually designing the state model for an agent. I would say if I were building an agent from scratch, I wouldn’t start by saying, let’s remember everything. I would start with the job the agent is expected to perform. And I would work backwards from that for each piece of information I’d ask, does the agent need this information for a future decision? How long does it need it? What happens if this information is lost or wrong? Let’s say, for example, a customer support agent. So, during the current interaction it probably needs the customer’s question what troubleshooting steps have already been attempted and the results of those steps. And that’s a short-term state. If the customer decides to come back tomorrow, maybe we want to remember that there’s an open support case and what was already tied and that’s durable state. But we probably don’t need to preserve every intermediate thought or every piece of information the model has encountered. And yes, frameworks can provide mechanisms for things like checkpoints, conversation, history and all that, but framework shouldn’t really make the business decision about what deserves to remember. That’s an app design decision I would say.
Gavin Henry 00:18:33 Perfect, thank you. That brings us nicely onto next action to close that one off. So, I want to talk about why agents need states kind of gone over it a little bit, but we need to talk about the different things an AI agent needs to remember. And you very nicely touched on the scenario there of an AI chat bot. There’s nothing worse than getting through to another one the next day and you have to repeat yourself. So, I understand that in my notes here. I’ve got a few bullet points of the usual types of state. If I could just read those out to you and then we could think of an example that might contain all those different things, that would be great and then we could go through it as a soft small case study as it were. So, I’ve got the types of state would be the user goals, potentially conversation history, task progress, the tools, results like we were talking about in the ID example, if any permissions are needed, whether they need to ask for them, you know, or they’ve already been given any intermediate decisions and prior actions.
Gavin Henry 00:19:36 So maybe the AI chat bot is a good example to use and break down what these different types of state would be. Or if there’s one that you usually like to discuss, have a go, that’d be great. Or if I’ve missed any, let me know.
Sowjanya Pandruju 00:19:49 I think the customer support agent is actually a great example because we can fit all of those types of state into one scenario without making it, you know, artificial. Let’s be more specific like I want to say I contact an eight-line support agent and I say my flight tomorrow has been canceled, I need to get to New York before 2:00 PM and I’d like you to rebook me.
Gavin Henry 00:20:15 So that would be, you’re the user and that’s your goal.
Sowjanya Pandruju 00:20:18 Yeah. That’s exactly right. The first piece of my state is goal. The agent needs to carry forward that I’m not simply asking why the flight was canceled, my goal is to get to New York before 2:00 PM.
Gavin Henry 00:20:32 So that would be the LLM’s job to pull apart that sentence and recognize the key parts?
Sowjanya Pandruju 00:20:37 Right. And then there’s the conversation history you mentioned. Maybe the agent asks, ìAre you willing to fly from a nearby airport?î And I say yes, but I don’t want a flight with more than one connection. So those constraints now become relevant context for subsequent decisions.
Gavin Henry 00:20:54 Yeah. And the next two are task progress and tool results.
Sowjanya Pandruju 00:20:58 Exactly. So, we have the task progress, maybe the agent has identified my canceled reservation, it verified which flights I’m eligible to switch to and is now searching for alternatives. If something interrupts the process, we don’t necessarily want to start all over again. We also have tool results that you brought up. So, the agent might call the airlines to the reservation system and discover let’s say two or three available flights. So those results become information it used to determine what to do next. And this is when the permissions become interesting. Searching flights might not require any additional approval from me, it already has that, but changing my reservation is an action with a side effect, right? The system might say I found this flight; do you want me to rebook it for you? My approval becomes part of the state. Importantly the application should still be enforcing what the agent is actually authorized to do, right? This is where there are these intermediate decisions. Maybe there were three flights, but the agent eliminated one because it arrived after 2:00 PM and another because it had two connections. So, we may want to retain enough information to understand why it selected that remaining option. And let me think
Gavin Henry 00:22:33 That one was prior actions.
Sowjanya Pandruju 00:22:35 Yeah, they’re extremely important too. Let’s say the reservation has actually been changed; the system needs authoritative record that says this action already happened. We don’t want the agent to lose context and retry the workflow and accidentally, you know, attempt to rebook me again. So, all of these pieces of state work together but they are not necessarily stored or in the same place or treated the same way.
Gavin Henry 00:23:04 So I’ve got two questions that have popped into my head as you were explaining those, you mentioned side effects. Is that a type of state or is that a prior action? How would that work? The side effect would be, in this example you might have to pay a little bit more because you’ve changed your flight. Is that correct? Would that be a side effect or is that just an outcome?
Sowjanya Pandruju 00:23:24 I would say it would be like side effect isn’t really a type of state, it’s a change the agent causes in the outside world. Okay. And then we often need to record that change happens. So, in our airline example, right, if the agent searches for available flights, that’s mostly a read operation, it hasn’t changed anything. But when it actually rebook my flight, that’s a side effect because it has changed the reservation in the airline system and if that rebooking also charges me an additional $50, the charge would be another side effect the result the state might now be reservation changed, new flight number payment of $50, it completed confirmation issued. So, this distinction becomes really important for reliability,
Gavin Henry 00:24:10 Especially with payment involved in it.
Sowjanya Pandruju 00:24:12 Yeah.
Gavin Henry 00:24:13 Well I was thinking there when you’re configuring this airline support agent chat bot, can it figure out what state it needs to save? If you haven’t told it, what state to save? Are there facilities there to say hold on, this is something that not being told I should keep a record of but in order to do the next thing I need to save this. Is that a thing or is that not?
Sowjanya Pandruju 00:24:35 Yeah, it can be a thing, but I would also separate working state. The agent discovers it needs during execution from I would say durable state, that we are deliberately designing the system to persist. So, in our airline agent it searches for flights and discovers three options that we talked about. Nobody necessarily had to configure a field saying remember flight option number two, the agent can recognize that those search results are relevant to completing the current task and keep them in its working context. It might use those results as the next step to compare arrival times or arrival or the prices. So, some agent frameworks also allow us the model to interact with memory or state management tools so the agent could effectively decide if this information will be useful later and write something to a memory or create a checkpoint that’s possible. Where I’d be more cautious is allowing the model to independently decide everything that becomes durable or authoritative state.
Sowjanya Pandruju 00:25:53 So if I have to go back to the airline example, I wouldn’t want the agent to decide on its own. I think I’ll remember that this customer paid $50 so the airline’s payment system should be the authoritative source for whether that payment occurred. Similarly, the application should explicitly define important things like the customer authorization reservation status or the complete transactions. So, I think it’s really two layers we can give the agent flexibility to manage its working state, like things that discover and need while solving the problem. But for business-critical state, the engineering team should define the ownership source of truth. I would say the persistence rules.
Gavin Henry 00:26:44 Yeah, that’s a good point with deciding and configuring like the authoritative place to get that information. So, I’ve experienced that in the ID agents or when you’re speaking to some chat systems and it totally forgets completely where it is because it’s run out of working state or it’s not gone back to the source of truth to validate something. Maybe that’s not correct in this example, but that’s just how I feel when I use them sometimes, you know. So, would that be working state or is that the context or when things forget things.
Sowjanya Pandruju 00:27:16 I think what you’re experiencing could be different layers. Why can it feel like agent forgot. I can think of two possibilities. One is the context window. The application may have a lot of history stored, but it can’t send all of it to the LLM on every call. So as the interaction is getting longer, it might drop older information. You might see prompts where it says, you know, it has gotten a log so I’m going to summarize and save it or only retrieve what it thinks is relevant. If something important gets left out or badly summarized the model effectively doesn’t know it anymore even though the application may still have it stored somewhere. And the other one is the working state. Imagine our coding agent discovered 10 steps ago that a particular configuration while file was what was causing the problem. If that information was temporary, like maintain only temporary and wasn’t carried forward or checkpoint leader, then the agent may start investigating something it has already solved. That’s a state management problem rather than the LLM literally forgetting. So, when an agent appears and says we get that it appears to forget something, I usually wouldn’t immediately say the model forgot, I’d ask was this included or retrieved into the current context, was it summarized correctly? And that’s one of the interesting things about agent state too. Having the information somewhere and having the right information available to the model at the right moment are two different problems.
Gavin Henry 00:28:58 Yeah, that’s helped me understand it a lot actually, because the way you’ve described it goes back and forth, it’s still got the original question, the user, you know, the user goal, but it’s constantly getting more information and more context and different steps it’s done and they all need to be re included, don’t they? And the state needs to be valid and up to date it feels like it could be really brittle at any one of those stages. Like you said, 10 steps back. If it makes an assumption, then what you get out the end is nowhere near what to get achieved in the first place.
Sowjanya Pandruju 00:29:30 Yeah, that’s one of the reasons agent systems can become surprisingly brittle as tasks get longer, we all experienced that the agent, you know, it starts with original goal, but every step is adding new information. There are tool results, there’s decisions, maybe even the user might start giving new instructions. So, the application then has to decide what from all of that should be carried forward into the next call to the model. So, and you know, the important thing is also it usually doesn’t just resend everything forever. That would become way too expensive. It could make it too slow and potentially exceed the model’s context limits as well. So, the system may summarize older information, retrieve only selected pieces of state or reconstruct the current situation. But that’s where errors can compound. So yeah, I think of this like, and your intuition is right, the longer and more autonomous the workflow becomes, the more important stage management becomes. So, the model can reason very well, but if we give it the wrong or incomplete representation of where the system is, it can confidently reason from a bad starting point.
Gavin Henry 00:30:48 Yeah, I’ve experienced probably what most people have experienced, like you’re saying where it gets lost or I think it’s done something or it’s lost a bit of context, but I personally feel like I have to give it a break and come back the next day to see if it’s refreshed itself. Sometimes. Yeah, that might be just me. Like I had one the other day, I think I had two different IDs open from JetBrains. I had IntelliJ and Rust Rover and I was working on a Kotlin project and one using Copilot and then in the other one I was using Copilot2 to make some changes, but only one of them was actually interacting with the system and doing things. I asked the other one said, oh yeah, I’ve made that change for you. And I’m like, hold on a minute, you haven’t touched any of the code so oh no, yeah, I’ve done it now. Thanks for pointing that out. I’m like, you still haven’t done anything. It’s like, oh yeah, yeah, yeah, I see that I haven’t done anything ever. I’ll do that now for you. But I actually had to restart the system because it obviously either lost the connection or some context or I’m not sure how that happened but it’s happened a couple of times.
Sowjanya Pandruju 00:31:52 Yeah, what you are telling me is an experience that we’re going through and it’s a really good real-world example just because we are all experiencing that and there are two different realities to that situation too. There’s the model representation of what it believes has happened and then there’s the actual state of your development environment. So, we could think like the model may have reasoned the next action is to modify this file and perhaps even generate such modification. But somewhere between that decision and the tool actually executing it, something failed. Maybe the tool connection wasn’t available or the result wasn’t correctly returned to the agent or without, I mean knowing the implementation, we can’t say where, but the interesting failure is when the agent telling you. I’ve made the change and at that point it’s treating intended action I would say as though it were a completed action. And that’s exactly why I think prior actions and tool results are such important pieces of state; a robust agent shouldn’t update its state from I intend to modify the file to the file was modified simply because the model generated some tool call. So ideally, I would say the tool executes the operation, returns evidence that it succeeded and only then does the system record the transition has completed.
Gavin Henry 00:33:22 Yeah, I think it plays nicely with the example of taking payment that you said. The agent would keep insisting that it’s taken 50 pounds or $50 but because it’s not going back to the payment authoritative system to confirm that it’s in the same situation that I just explained where I’ve made the change on the file but was like we haven’t gone and checked the file so you’re just making things up, you know?
Sowjanya Pandruju 00:33:45 Yeah, exactly. And I think that connection between the two examples is really useful because it shows that it’s not specific to payments or coding agents or it’s a general reliability problem, right? Payment example, the model believed I submitted $50 payment, therefore the payment happened and it didn’t check in your ID, it believed I generated the file changed; therefore the file was changed. So those are two separate things, the intention and the resulting state of the external system. So yeah.
Gavin Henry 00:34:43 Well that brings us on nicely to my next section, which is about where agent state is stored. So where is this stored and what sort of volume of data are we talking about when we’re thinking of state data?
Sowjanya Pandruju 00:34:58 It can be stored in few different places like we touched upon previously and the right choice really depends on what kind of state it is and how durable or I would say query able it needs to be. So, some states may just live in memory while the agent is running. That could be the current goal or some temporary scratch information that’s fast. But if the process dies, the state disappears, right? For anything that needs to survive beyond that process, we usually externalize it. This might be a relational database or a NoSQL database or a key value store or it can be an object store, a file or sometimes you need semantic memory, you have semantic memories vector database. So, in a production system it’s common to use more than one of those because different states have different access patterns, I would say. So, let’s take an airlines case.
Sowjanya Pandruju 00:35:57 So the current task status might live in a structured database. The customer’s long-term preferences might be memory store, the actual reservation is in the airline’s authoritative booking system, right? And the execution history is all recorded in logs or events so that it has that information for auditing and debugging. And you brought up a very interesting point, the volume. On the volume side, it can vary enormously too. The core structured state for one task is often small like gold status identifiers that might be in KBS or kilobytes. But the thing is the volume grows quickly when you start storing conversation transcripts or code or documents embeddings or long histories across thousands of users. So, at that point the issue is not just raw storage size, it’s also about how much of that state can you effectively retrieve and put back into the model’s context at the right time. So, I think the distinction would be storage capacity might not be the hardest problem. Selecting the right state to retrieve and present to the model is also a big one.
Gavin Henry 00:37:23 So you got me thinking there about the data storage and retrieving it, but the context that the AI agent has access to or is configured to use, I guess it’s configurable, that’d be like the buffer it has to store the intermediate data, it needs to go fetched to sort of refresh the context for the next step. Would that be a correct understanding of that?
Sowjanya Pandruju 00:37:45 Yeah, that’s I would say the right mental model. I would just upgrade the storage from context because they’re related but not quite the same thing. So, your idea of a buffer is very useful, but I wouldn’t necessarily call it literal buffer because implementation’s very, I would describe an agent’s working context or a working state, it’s similar to how we work as humans, right? I might have thousands of documents sitting on my computer, but I don’t need all of them open in front of me to answer one question. I retrieve the few that are relevant or to what I’m doing. And that’s essentially, I think what good context management is.
Gavin Henry 00:38:26 Yeah, I wasn’t sure if the context was the size of information that could act on at that point. Okay, cool. So, I’ve got a few lists of existing approaches to handle the state data and you’ve mentioned a few ones, so I’m just going to read out a few others. If you could confirm whether or how they would be used in the AI airline agent for the type of data that’s designed to use. So, we’ve got in-memory state, which is the working state, I think you mentioned we’ve got vector stores, databases, the traditional ones, something I’m not too familiar on at all workflow engines and we’ve got event logs which I think we can understand what that is. And then external state store. So, in our example of the airline part, does that fit nicely with these six types of storage approaches or is there anything I’ve missed that would be sort of critical?
Sowjanya Pandruju 00:39:19 I think the airline example is still really good, so let’s use that. So, and we want to talk about the different places or mechanisms for different kinds of stage, right? So, I would start with the in-memory state, that would be the temporary working state. So, in our airline’s example, that might be the fact that we are currently comparing three replacement flights or the most recent tool response we just got back. It’s useful because it’s fast, but if the process disappears, that state may disappear too. And then let’s bring up the vector stores. Are they more appropriate for semantic memory? Maybe the customer has a long-term preference for aisle seats, or I prefer nonstop flights or things like that. If we want to retrieve relevant information based on meaning rather than an exact key, a vector store can help. So, I wouldn’t use that to determine whether the customer’s reservation was actually changed.
Sowjanya Pandruju 00:40:21 That’s where traditional database comes in. Very important. We might store structure task state there like the current case ID, the current workflow status, which replacement flight was finally selected or whether user approval was received and to store such structured information, traditional database fits very well. And let’s also bring up that workflow engine. It’s a little different. I think of that as the component that helps manage a very long running sequence in a reliable fashion. So maybe agent needs to identify the cancel flight, find alternatives or they might have to rebook or you know, send confirmation, things like that. So, a workflow engine can persist where we are in that sequence. It can handle retries, let’s say if we are having or seeing some sort of connectivity issues with that. It can wait for events, and you can resume after failures. So that process is not just storing data, it is mostly helping manage the life cycle of the process, I would say.
Gavin Henry 00:41:30 Is that engine a completely different system that might have other backends and you speak to over an API or something or how would you define the makeup of that workflow engine?
Sowjanya Pandruju 00:41:41 So it could be a scheduler or an executor, right, that knows when to run the next step and what not. So, it could be a compute that you would leverage.
Gavin Henry 00:41:53 Okay. And event logs, that’s just logging to files, I guess?
Sowjanya Pandruju 00:41:58 Yes. So, let’s get back to that. Even logs are extremely important, especially I work in enterprise systems and ability to debug, and audits are extremely important because they tell you what happened over time. Like search requested, three flights returned, customer approved it rebooking attempted. And that’s valuable for observability. For debugging or sometimes if you want to reconstruct what happens. So, event logs are extremely important. And you also mentioned I think about external state stores. I would interpret that as systems outside the agent that already have important information. As in the case that we are working with the reservation system is the authoritative source of actual booking and the payment system is the authoritative source of whether the fair difference was charged. So, the agent shouldn’t try to recreate those truths in its own memory.
Gavin Henry 00:43:03 I see that I was a bit confused as to what that might be. The external state. Is there anything that you deal with or I’ve come across that we’ve missed in those six different things? And just to summarize again is in memory state vector stores, which is what you described to ask more question-based information rather than just concrete information, traditional databases, workflow engines, the event logs in the external state.
Sowjanya Pandruju 00:43:30 Yeah, I think those six cover the major concepts quite well. And external state, it could be very confusing terminology. Like I wouldn’t necessarily treat external state as another storage technology like a database or a vector store. I think it more of a state that is owned by another system completely outside the agent.
Gavin Henry 00:43:55 Yeah. I wasn’t sure if it was a service that you’d be subscribing to. Yeah, didn’t realize it was a concrete existing thing.
Sowjanya Pandruju 00:44:04 Yeah. Have to summarize this as you know we were talking about, so I would say in memory for temporary working state, vector stores for semantic retrieval or you know, similar searches, databases for any structured durable state, what did we talk about? Workflow engines. Workflow engines for durable process execution, event logs for auditing the whole history and then the external systems for authoritative business state. That’s how I would summarize the whole landscape.
Gavin Henry 00:44:37 Yeah, so it’s clicked now. So, workflow engines are what stage in the process of maybe running a tool and how to store that? Yeah. Okay. I was a bit unsure of that. Okay, so that brings us nicely into the last section, which I think we might have covered quite a bit already, here and there. So, what are the challenges and how are they solved with agent state? I’ve got a few questions here. So how do we avoid stale context? I think that was something we touched upon initially, but if you could have a go at that one please.
Sowjanya Pandruju 00:45:10 Yeah, I think the key thing here is stale context can happen anytime. Even when the agent remembers something perfectly, the information itself may simply no longer be true. So going back to the airline example we’ve been using, let’s say the agent searched 10 minutes ago and found three seats available on a specific flight, it stored that results in its state and 10 minutes later the customer could come back and say, yeah, book that one. The agent still remembers perfectly that there were three seats but there’s been 10 minutes between this and somebody else may have booked them in the meantime. So, for the agent still remember great, but things are not the same anymore, right? So, information that can change, we need to I think distinguish between remembered state and current truth. So, there are a few ways to handle that. One is to attach expiration time so that the state knows we have to refresh it.
Sowjanya Pandruju 00:46:20 Another is to define certain information as something that must be refreshed before a consequential action. So, before you actually rebook that flight, the agent goes back to the airline system and checks the availability again. And, there’s stale context about the task itself. So, let’s say through repeated summarization, the agent has lost the fact that I must arrive before 2:00 PM that’s a slightly different problem for important constraints like that, I prefer keeping them explicitly in structured state rather than relying on them to survive several rounds of generated summaries. So, I think there are two complimentary strategies I could think of like preserving important task variance explicitly and also refresh dynamic information from its authoritative source when freshness matters.
Gavin Henry 00:47:20 Thank you. And I’m not sure if this is the same question, I’ll leave this to you to decide, but so how do we avoid inconsistent memory that’s different from stale context?
Sowjanya Pandruju 00:47:33 Yeah, so let’s say stale context means the information was correct when we captured it, but the reality has changed, right? So, we said that in the airlines example, we correctly recorded, three seats were available, but then 10 minutes later those seats were gone. And inconsistent memory means that, agent has two or more pieces of state that disagree with each other and now it has to know which one to trust. That’s how I would think of it. So, let’s say in our example, right, in our airline conversation, I said I need to arrive before 2:00 PM later maybe another part of the state says customer prefers cheapest flight regardless of arrival done. Or perhaps the agents’ memory says my reservation has been changed while the booking system airline actual booking system still showing the original flight. Now you just don’t have old information, you have conflicting information.
Sowjanya Pandruju 00:48:36 So that’s the problem. I believe the way to handle that would be establishing ownership and maybe precedents as well. Yeah, so not every piece of state should have equal authority. The reservation system owns the booking status; the payment system owns that payment status. So explicit user instructions should generally take precedence over an older inferred preference. So, an important structured constraints shouldn’t be overwritten casually by a generated summary. So, versioning can help as well. But if two agents’ processes are working on the same task, we can attach a version to the state. So, if they both read version five and one is updating to version six, the other shouldn’t blindly override, with its own. So, it needs to detect that change and either reload or resolve the conflict.
Gavin Henry 00:49:34 Are you kind of looking for some type of convergence algorithm or bullying to say right, if this then that the reservation system says that, so that’s got priority or we’ve got four reservation systems and three of them are correct, four of them, one of them isn’t and you are not so they’re correct. So, it’s kind of like ah, forget the term, like the raft protocol or something where they’re trying to come to a consistent decision.
Sowjanya Pandruju 00:50:00 Yeah, I completely agree with that.
Gavin Henry 00:50:02 So I think we’ve covered still context and inconsistent memory to make the decision, but how do we avoid, they made a decision and they go off and act and then they do it again and again, how do we avoid repeated actions?
Sowjanya Pandruju 00:50:16 So I think this is where I think, I’m not sure if this is the right term, but in my mind ID impotency becomes really important and you know, let’s go back to the airline example. I think we have got a really good example with the airlines. So, I need to charge the customer $50 for this rebooking. Okay, it sends the payment request but before it gets the response there’s a timeout. So now the agent doesn’t know whether the charge succeeded. If it decides to retry the action, there is a danger that the customer could be charged twice. So, a common way that I have seen implemented is to assign a logical action or a unique ID, which is in some of the terms we called an item potency key. So instead of saying charge $50 each time as though it’s a new action, the system says something like execute payment operation 1, 2, 3, 4, 5. So if the request is retried with that same identifier, the payment system recognizes that it already processed that operation and returns the existing result instead of charging again. So, we’d also want to keep a durable state around the action itself. Something like the stream proposed then executing, then succeeded or failed, then the agent can check what actually happened before deciding to execute again. And for important the product principle would be like, don’t depend on the LLM, remembering not to repeat an action, make a repeated execution safe at the system level.
Gavin Henry 00:52:02 Yeah, that’s a good point about the simple solution of using IDs, but I guess it would depend on another step of who assigns the ID. You don’t want the agent to make up the ID, it needs to be given by somewhere that’s going to save the original one, so they know it’s been retried or reused.
Sowjanya Pandruju 00:52:20 That’s a really important and good question that you brought up. I was going through my experience on how, because these are some of the things that we have been handling these days. So, the important thing is that ID, this it or you know, it needs to represent the same logical business operation across retrial, right? So ideally the surrounding application or workflow layer creates and persist that identifier when action is first initiated. Then if the network times out or the agent re-try later, it re-uses the same key. So, in the airline example, the workflow might create something like a rebooking operation ID when the customer approves the change. That ID gets stored with the task. So, the payment call, the reservation update and any retries that we are talking about it can reference that same operation. So, I think that’s how I would put it.
Gavin Henry 00:53:21 Thank you. So, I’ve got a few points, few questions to ask left before we have to start wrapping up. So, we’ve spoken about avoiding repeat actions and I think we’ve touched upon hidden side effects already. For example, a summary gets mis-summarized if that’s a word I summarized. And then, would that be a hidden side effect? That’s something that’s happened or how do you define hidden side effects and how do we deal with it?
Sowjanya Pandruju 00:53:49 Really interesting question. So, I define hidden side effects a little different from a bad summary. If we think about right, a bad summary is a state quality problem. It can certainly lead to an incorrect side effect. But the side effect itself is some change the agent causes in the outside world that may not be obvious from its internal state. So, in our airline example, it is calling a rebooking API that API changes the reservation, it charges a fee, it releases the old seat, maybe it is sending a confirmation email. So, from agent’s perspective it may look like one tool call a very simple one at that. But several things in the real world have changed and those are the side effects. So, the problem becomes especially serious with failures. Imagine the rebooking succeeds, but you know, the agent response was a timeout. The agent state might still say rebooking in progress while in the real world the passenger has already been rebooked. So, if the agent is retrying blindly, we create additional intended effects. So, I think the first solution is to make side effects explicit in the design. That’s how I would put it. So, we should know which tools are read only, which ones mutate, external systems for consequential actions. We want durable operation IDs like we talked about or the ID potency where it’s possible and verification against the authoritative system like we talked about.
Gavin Henry 00:55:28 Yeah, it seems like if you take the term agent state out of the conversation, these are all common problems that you have to deal with when you’re building web systems, online systems, you know, things that have a decisions made, it has an impact and to make sure it’s actually followed through. It’s just a new frontier that we’re trying to solve existing problems through. Okay, so what is session affinity? How do we deal with that?
Sowjanya Pandruju 00:55:55 So session affinity, I sometimes call it a sticky session. It’s when a particular user’s request needs to keep going back to the same server or worker because that server is holding some state locally. And even with AI, without AI session affinity has been a longstanding thing going on. So, imagine you have three instances running our airline agent. Now I’m talking about the application of the compute behind it, right? So, the first request goes to instance A. Here I’m talking about the actual servers that are running my airline agent. I’m just using three as an example. It could have many more and whatnot. So, let’s imagine we have three instances running an airline agent. The first request is going to instance A and instance A keeps my conversation and the task state in its local memory. If my next request gets, because requests are often balanced through a load balancer to the backend server.
Sowjanya Pandruju 00:56:58 So that’s where the whole session affinity is coming into the picture as well. So, if my next request is getting being load balance to instance B and instance B may have no idea what we’ve already discussed. So, this is where situations like this is where session affinity becomes important. This way you configure session affinity. So, the requests may continue going to instance A. So that can work, but the problem is that they create a dependency on that instance. So, if any requests are going to one system, like one instance and with session affinity until you have the timeout, it keeps going there so it could get overloaded. The load balancing wouldn’t work the same way. So, it would have those problems on the instances it could crash. We’ll have a scaling issue with the application, whatnot. So for production systems, particularly for important long-running tasks, I prefer to externalize the state that needs to survive because session affinity really helps so that you don’t have to repeat yourself and you know the context is not getting lost every time you are speaking to it. But for long-running tasks, I always prefer to externalize the state that needs to survive. Then instance A can handle one request, instance B can handle the next time both can retrieve the necessary state from a durable storage. So, but that doesn’t mean we can’t keep anything locally. Cache is a temporary working state can absolutely sustain memory. It’s just I was also addressing what happens if that particular process disappears or you know, crashes and whatnot.
Gavin Henry 00:58:41 Perfect. I’ve got two things left before we wrap up. I’ll try and excuse them. I think we might have covered this one. So, what about decisions that are difficult to trace or reverse? For example, with the airline, we’ve booked something and the seat’s being taken, so that’ll be authorization mistakes potentially the things that have poor observability. How do we deal with those? And that’s the second last question before we wrap up this, just so you know.
Sowjanya Pandruju 00:59:10 And I think we have touched a little bit on pieces of this.
Gavin Henry 00:59:13 Yeah, you talked about the IDs. And the timeouts and identifying what state is authoritative.
Sowjanya Pandruju 00:59:20 Right. So yeah, for decisions that are difficult to trace or reverse, I think about three things coming into my mind. Prevention, observability and recovery. That’s how I would put it. So first is prevention. So, before anything an agent performs, like booking a flight, issuing a refund, whatnot, I distinguish authorization from approval. So, authorization answers things like, is this user or agent permitted to perform this action? Approval answers has the user actually agreed to this particular action. So, the model shouldn’t be the component that ultimately enforces either of those guarantees. The second is observability. So, we want to record the meaningful parts of the execution, like what goal or request led to the action, what tool was called, what operation ID was used, what did the tool return, what state transition happened, was an approval in any of this. So, this gives us a trace we can follow later without needing to capture every internal reasoning step of the model. So that’s why observability is tremendously important. And third I would say is recovery. Some actions can always be reversed, others can’t. So, if we book the wrong flight, maybe we can cancel or rebook it in a distributed system, I will say, I call this a compensating action rather than a true rollback. So, if the agent sends an email or submits something externally, we may not be able to undo it at all. In those cases, prevention becomes much more important and we might deliver it.
Gavin Henry 01:01:00 So, it really depends on the decision that’s being made. If it’s a permanent decision like payment, it’s not too scary because you can issue a refund.
Sowjanya Pandruju 01:01:12 Yeah. The principle would be like, you know, harder an action needs to reverse the stronger the control should be before it happens.
Gavin Henry 01:01:19 Okay, so finally before we wrap up in your example there of things to reverse, you know, I could use if it’s an IG, we’d have Git for that type of thing, but I’ve only got one agent running. So, if we’re using multi-agent, how big a difference or change does that make to everything we’ve spoken about so far? First of all, what is multi-agent and does that make state more complex when we’re dealing with more than one thing doing stuff?
Sowjanya Pandruju 01:01:48 Very interesting question and this is something I have thought about a lot and you know, that’s what makes this thought process much more complex too. Yeah, definitely it makes state more complex because now we are moving from one decision making component, maintaining continuity to several components that may be working independently and concurrently as well. First by a multi-agent system, I mean a system where multiple agents have different responsibilities, but they all collaborate towards some larger goal. Let’s say an airline example. Rather than having one agent do everything, we might have one agent that is responsible for finding an alternative flight. Another could be dealing with payments. One could be dealing with customer communication. So, there might also be a coordinating agent that delegates work to these specialized agents. So, the advantage is specialization and separation of responsibilities. But from a state perspective, we’ve now introduced so many questions.
Sowjanya Pandruju 01:02:56 What state does each agent own? What state do they share? How do they communicate changes? What happens if agents act on same information at the same time? So, I wouldn’t solve communication between agents becoming really important. And for example, in our red line case, so flight search agent could find a seat and pass that information to the booking agent. By the time booking agent acts, that seat may be gone. Perhaps two agents both believe they’re responsible for let’s say a refund and they both try to execute it. So, these are very familiar distributed systems problems, but now they’re occurring between autonomous components also. So, I generally wouldn’t solve that by giving statements like every agent access to every piece of state. I’d say clear ownership and minimum necessary sharing. So, each agent gets the context for what it’s responsible for, while also sharing important facts like task status, the ID that we talked about, like a transaction ID, completed actions, approvals and things like that.
Gavin Henry 01:04:12 Perfect. Thank you. So that’s us. I think we’ve covered everything I set out to learn and hopefully the listeners have learned too. And you’ve enjoyed yourself very happy with the things that you shared with us. So, I’ve really enjoyed chatting with you about AI state, but it’s a question I always like to ask my guests. If there was one thing you would like a software engineer to remember from our show, what would you like that to be?
Sowjanya Pandruju 01:04:35 Oh, there’s so much I would be excited about. I love how conversational these chats are. So, I don’t know if this is what you’re looking for, but lately I’ve enjoyed listening to the show. There are so many perspectives and unique ways of looking at things. So, I started going through at least three podcasts each month. And I mean, it’s not a lot, but it has improved the amount of exposure I have on topics that I wouldn’t usually touch. So, you know, switching from listening to adjusted music to at least one podcast a week has made me much happier and much smarter. So that is something I would encourage everybody to do.
Gavin Henry 01:05:29 Excellent. Thank you. I was kind of mean, something you’d discussed in the show that you’d like them to remember. Oh, but no, that’s a good bit of advice in general. I like it. So yeah, let’s rephrase it. If there’s something, if someone said right, what was the number one thing to remember from this episode? What would you personally like to be? That’s my question.
Sowjanya Pandruju 01:05:48 So, one thing I’d like saw to take away is that Agent State is really application state because you know, we’ve been talking about managing state. So, when we talk about agents as the model itself, remember, decides next, but in the production system, the model is really one component within a much larger software architecture. So, application still has to decide what information is authoritative, where it lives, how long it lives, who can access it, and things like that. So, I would say given the agent flexibility in reasoning, but keep important guarantees deterministic, let agent decide that we talked about the flight being rebooked, things like that. So, if we approach agents that way, not as magical autonomous systems, but as components within a well-engineered system, I think we can build the systems much more reliably. I think that’s the point I was going for.
Gavin Henry 01:06:51 Yeah, that’s what I’ve taken away. I should have used the word takeaway rather than remember. That seems to be more common. But yeah, for me finally understanding that the LMM is just a component of an agent system really, hit home for me. So, so that was good. So, is there anything we missed that you thought might come up that you’d like to mention before we finish there? Or were you happy?
Sowjanya Pandruju 01:07:13 No, I’m extremely happy with the way you conducted this, the level of, it’s a well-rounded session, I would say. We’ve covered almost everything that I could even think of.
Gavin Henry 01:07:23 Thank you. So, if people want to get in touch with you, to discuss anything that’s come up, how can they reach out?
Sowjanya Pandruju 01:07:30 They can reach out to me on LinkedIn. I don’t know if I have shared that.
Gavin Henry 01:07:35 Yeah, I’ll put that. I’ve got that in the show notes for when it comes out.
Sowjanya Pandruju 01:07:39 Yeah. And my email address is public so anybody can reach out to me through email as well.
Gavin Henry 01:07:46 Cool. So, Sowjanya, thank you for coming on the show. It’s been a real pleasure. This is Gavin Henry for our Software Engineering Radio. And thank you for listening.
[End of Audio]
