Clare Liguori, a Senior Principal Engineer who works on developer tooling and agentic AI at Amazon Web Services, speaks with host Sri Panyam about the Amazon Strands Agents SDK. This episode explores the philosophy, design decisions, and emerging patterns behind building production-grade AI agents.
Clare frames any agent as three core components: a model, a set of tools, and a prompt. During this interview, she describes the origin story of Strands, the model-driven approach vs. workflows and custom orchestration, steering hooks, tools and MCP, sub-agents and multi-agents, memory layers, production readiness, testing and evaluation starting with use cases where trajectories can be evaluated deterministically, and anti-patterns for newcomers. She describes what’s next for Strands, and offers some closing advice for getting results from working with agents
Brought to you by IEEE Computer Society and IEEE Software magazine.
Show Notes
Related Links
- Strands Agents SDK GitHub
- Strands documentation
- Blog post: Introducing Strands Agents, an Open Source AI Agents SDK | Amazon Web Services
- Model Context Protocol: What is the Model Context Protocol (MCP)? – Model Context Protocol
- Clare Liguori’s website
- X.com: @clare_liguori
- Amazon Builders Library
Transcript
Transcript brought to you by IEEE Software magazine.
This transcript was automatically generated. To suggest improvements in the text, please contact [email protected] and include the episode number and URL.
Sri Panyam 00:00:19 Hello and welcome to another episode of Software Engineering Radio. Today we’ll talk about the Amazon Strands agents SDK with Clare Liguori. Clare is a Senior Principal Engineer at Amazon Web Services where she works on development tooling and agentic AI. She focuses on Kiro and the Strands agents SDK. She has spent 10 plus years bringing amazing advancements to AWS and to all developers to our great delight. Clare is also a contributor to the Amazon Builders Library. Welcome, Clare, to Software Engineering Radio.
Clare Liguori 00:00:53 Thanks.
Sri Panyam 00:00:54 Before we dive in, just a bit of a primer on AI agents: How would you define an agent, what is an agent? I know it’s been going on for a while, so where are we today? And as developers, how should we think about it?
Clare Liguori 00:01:06 I like to define it as three different things. One, a model; two, tools; and three, a prompt. And I think pretty much any agent system that you look at, agentic framework, those are the three core things that you need to get started building an agent.
Sri Panyam 00:01:25 And again, we all start off with ChatGPT or Claude AI — the web, I guess, entry. What makes an agent different from these typical chat use cases?
Clare Liguori 00:01:36 There’s all kinds of use cases you can do with agents beyond chat. I actually think for a while there, we got very stuck on chat being the only agent use case. But I see a lot of customers automating things. Many, many teams at Amazon are automating operations, doing ticket troubleshooting, things like that. We see customers automating business processes as well as of course, chat. Very common today obviously coding agents are very common today, but generally, I think compared to something like an out of the box agent like ChatGPT, building your own agent gives you a lot more control over what is the prompt, what are the models and what are the tools. You can build tools yourself, you can pull tools off the shelf with MCP servers, but generally you have a lot more control over what it is you want that agent to do and what success looks like.
Sri Panyam 00:02:33 Fair enough. Now, I guess that’s a really good segue into what we’re doing here, Strands SDK. I’d love to hear the origin story. How did it come about? When, where, or how?
Clare Liguori 00:02:42 I was working on a product called Amazon Q Developer in, let’s see, two years ago now. And that was the time around, I’m trying to think back maybe Claude Sonnet two, maybe three. But at that time, the models were not very reliable. They were great for chatting back and forth, but when you wanted to actually automate something like tool selection, tool execution, coding, they weren’t very reliable. I mean, we started back on Claude two, where they hadn’t actually yet enabled you to put tools into a tool section that was structured. And so, you just had to describe your tools in markdown, in your system prompt, and then you would try to get the model to give you the response, the tool it wanted to select in a very specific format. So, you could parse it out of this natural language string, but of course it wouldn’t do it.
Clare Liguori 00:03:50 So, you would try to give it partial the JSON that it was giving you, but it would put a sentence at the beginning, like, here’s my tool selection that would screw up your parsing and you would have to give it feedback. And so, we started building simple loops to improve reliability of these agents, which looked a lot like trying to parse the response and then telling the model what the parsing error was so it can fix itself. And we saw these wild productivity gains just with those simple techniques of just giving the model feedback for reliability. But then we started to, models started to get pretty good. They started to train on model selection. They started to train on actually modeling tools and tool selection into a separate part of the inference request. We didn’t need all of that scaffolding anymore.
Clare Liguori 00:04:47 And we also started to see that models started to get pretty good at tool selection. This was somewhere around, I would say Sonnet 3.5 was where we saw it start to get really good at model selection. And by that point, we had built up all of the scaffolding. We had built up prompt pipelines where we tried to put the perfect RAG content into the prompt ahead of giving it to the model. We had done all of this work around parsing the model’s response, and we found that as the models were getting better and better, we were actually making reliability and power of what the agent could do worse because we were kind of artificially constraining the model. So that was really the key aha moment behind what turned into Strands was what if we just throw away all of the scaffolding and go towards what we call the model driven approach, where you give a simple prompt, you give the tools and you have the model, and you let the model choose the tools, choose the context that it needs.
Clare Liguori 00:06:00 So as an example, we stopped doing these mega prompt pipelines where we would try to query from RAG the perfect content, that context that the model would need, and instead provided RAG as a tool. And so, it would come up with the query for RAG pretty well. It knew what it needed to get. One really specific example was we had an agent at the time that was for querying the AWS documentation, answering questions about AWS and the RAG database was about, was just the AWS documentation. But we found out that the agent already knew so much about AWS that the RAG content, the kind of proactive RAG context was actually confusing the model. There would be an entry that wasn’t really all that related, and it would just kind of zero in on that. And it was almost like it was kind of doubting its own, its own context that it had in its training data. And we found that once we got rid of that, sometimes it would query the RAG database and sometimes it would just use its own training data. And so we found that the quality and the accuracy actually went way up once we stopped trying to proactively give all this.
Sri Panyam 00:07:20 I would love to hear about the origin story of Strands SDK. How did it come about? What was it trying to solve? What was the pain points?
Clare Liguori 00:07:27 Well, I started building agents as part of a product called Amazon Queue Developer three years ago now, I think. And that was the time of Claude 2, before models had really been trained on this idea of tools. You could provide a set of tools and you could have the model selected, a set of tools. So, we had all of this scaffolding around trying to get models to understand tools and to tell us what tools it wanted to run. We had all of the scaffolding around describing the tools in JSON inside of the prompt, and then we would try to get it to respond with JSON about what tool it wanted us to choose. And we would have to try to parse that. But the models at the time were very obsessed with natural language. And so, they would always say, here is your JSON, or here is the tool I’d like you to run.
Clare Liguori 00:08:21 And that would break all of our parsing. But we found that we started just adding these kind of retry loops around that. We started parsing the JSON and then giving that JSON parse error back to the model. And we found that the models were actually really adapted to fixing themselves on the next time they would be able to provide the correct JSON. So I think that was one learning was the models dealt with feedback really well. The second was around the time that Sonnet 3.5 came out, this was when models now had structured ways for you to produce, to give them the tools and structured ways to tell you what tools it wanted you to run. And by that time we had built up so much scaffolding. We had these massive prompt pipelines that tried to pull from a RAG database, the exact right context.
Clare Liguori 00:09:17 And we had all of this logic about what context went into the prompt and where, and we found that with later models, we were actually making the accuracy and the power of these agents works. We weren’t taking full advantage of the evolution and the power of the latest models. And so, the big aha moment was, let’s throw all of this away. Let’s see what the model can do itself. Let’s just give it a simple prompt. Let’s give it tools including access to the RAG database as a tool it can figure out what information it wants to go get. One example was we had all of the AWS documentation in a RAG database, and it could choose to go get new information or it could choose to rely on its training data for the model. And we found that all of us sudden we were, we had these much more powerful agents and they were simpler and they were quicker to build because we didn’t have to build all of the scaffolding.
Clare Liguori 00:10:18 We could just provide prompt, tools and choose the model. And that was the kernel that led to Strands agents. That’s that we started building our own agent framework to do that, to take on this model driven approach and simplicity approach to building agents. Probably about a year later, we looked at the success that we’d had internally. We were shipping agents in a matter of weeks. The QCLI, which is now the QO CLI, we shipped it in three weeks using this internal framework. And so, we decided to release this in open source so that many more people can take advantage of this approach that enables you to take full power advantage of the full power of the model, advantage of the full capabilities of the model. That was last May, I think that we went, that we open-sourced it and since then we’ve had over 25 million downloads. So, it’s been great to see the community pop up around it.
Sri Panyam 00:11:18 Nice. I feel like I have a lot of amnesia because now that I mentioned it, I can recall the pain of having to describe what JSON looked like in plain English, right? So, the model would spit out a closing brace, matching the open brace and it would get subtle things wrong, like adding tactics all over the place. And now we’re at a place where you just give it a context free grammar, and the elements can generate amazingly, syntactically correct outputs. What a crazy 18 months. It’s interesting that you mentioned model driven approach as opposed to like step driven manual. If you were to kind of call this something different, what is it like?
Clare Liguori 00:11:58 Yeah, I see two approaches that people take to building agents that seem way more complex than they need to be. One is building a ton of custom orchestration around the model. And that’s what I described around all of the scaffolding that we had built around models that over time the models made all of it unnecessary. With the model driven approach, you don’t really need to change anything about your agent to start taking advantage of a new model and all of its capabilities. You just have to change the model ID equals line to start taking advantage because you don’t have all of this or custom orchestration that’s now been made unnecessary. The other thing that I see people building is workflows. A lot of folks are building things that they want to add determinism to. They want to take advantage of these models, but they also want to do determinism, right?
Clare Liguori 00:12:59 And there’s a lot of use cases where you do need a lot of determinism, but it would be great to automate with agents. So, things like booking an airline, wanting to automate flight crew assignments to flights very hard to do, needs a lot of determinism. There’s a lot of rules around it, right? There’s unions and all kinds of things. Financial services companies I see trying to automate things like mortgage processing, loan approvals, transactions, and all of these things have, are pretty risky in terms of, if you don’t do the right thing, if you don’t verify someone’s income prior to approving a loan, big risk for the company, right? And so, a lot of folks kind of fall back on what we know as software engineers. We know workflows, we know procedural code effectively. So, we want to build for loops, we want to build if statements, we want to build a while loops.
Clare Liguori 00:13:59 And I actually find that with agents it’s important to think a little bit differently. This comes back to some of the additional features that we’ve been building into Strands since our launch last May. It’s hard to believe that it’s almost a year now, but I would say Strands is transitioning much more into an agent harness, meaning we’ve been adding the lifecycle hooks, the context management that you need in order to build a more reliable agent maybe. Recently we released a feature called steering, and this is my personal answer, I guess to workflows. I think that workflows maybe appear to give you determinism, but in practice they’re actually really difficult and there are a lot of scaffolding. Again, I’ve seen a lot of workflows developed inside of Amazon that over time again, they just make it less able to take advantage of model, improve model capabilities and almost make it worse in some cases.
Clare Liguori 00:15:03 So we steering takes a different approach where, let’s take the example of loan processing, for example. You want to verify income before actually approving a loan. The model might often do those things in the right order, but sometimes is not going to do those things in the right order. So as engineers, we fall back on our procedural knowledge, right? We say, okay, let’s create an if statement in there and let’s ensure that the income verification came back with this value. If so, continue to loan approval. But that ends up being very brittle is what we find. So you might’ve asked the agent to do loan verification or loan approval, or you might have asked it just a question about what’s the risk of this loan? And so, you end up with these very brittle single use case agents. With steering instead, you get to define rules about certain restrictions on the agent.
Clare Liguori 00:16:05 Things like it must call loan. If it’s going to approve a loan, it must first check the value, the user’s income, and it must be above this limit or else it cannot call this this loan approval tool. And that it kind of flips how we think about software execution on its head, where it’s just in time verification of what has already happened. As opposed to when I’m building the agent, I’m going to say exactly the set of steps that it needs to take like a workflow. We found that it can actually produce better results, better accuracy than workflows can, which is super exciting, but it’s a very different mindset than what we’re used to with workflows.
Sri Panyam 00:16:54 Interesting. Well, itís funny you mentioned this sharing hooks because looking at the efficacy of it, this being is been, it’s pretty, pretty impressive right now if you take the traditional agent harness, but something that’s not rule-based, but still a harness that calls tools invokes the model as lifecycle looks where like, just to come to the baseline, where does the agent harness end? Where did you find Strands was kind of leveling out? Leveling out where you go with your latest invocation, latest incarnation of the agent harness?
Clare Liguori 00:17:26 One of the biggest changes, the differences that we see and I’ll focus on steering because I think overall we’re really working backwards from customers who are trying to automate business processes. We see that as one of the biggest use case right now for agents. We have the coding agents, like the coding agents are out there. Those are the biggest harnesses we know of, right? That we see every day on social media and in our lives, right? As software engineers, those are what we’re using. But we find that most valuable agents out there are probably automating business process often get asked what is the difference between hooks, which pretty much every agent harness has and steering very naively they look the same, right? I’m going to inject myself into the agent right before a tool call and do something. One of the biggest changes are hooks will only give you context about what is the thing it’s about to do.
Clare Liguori 00:18:30 And you can say, this is dangerous or this is not dangerous, or I want to allow this or I don’t want to allow this. We see this often with coding assistance, right? You get a hook and you can say, I don’t want to allow rmrf Stark and I can block that with a hook, right? Steering gives you the full trajectory of everything the agent has already done. And as it turns out for automating a process that’s super important because you’re able to express things like if you’re about to go approve that loan, call the loan approval tool, then in that steering hook I need to see if it’s already verified income, if it’s already called that tool and what the value is that it came back with. Some of the other things that we see is models are sometimes apt to hallucinate inputs there’s a lot of use cases right around taking an output from one tool and providing it as an input to another tool.
Clare Liguori 00:19:29 Let’s say, a user ID maybe you retrieve the user ID from one tool, you provide it in another. Bad things happen when you hallucinate a user ID in the other tool, right, as input. And so, with steering you can see in that hook when it’s about to call that tool and provide that user ID, does this user ID actually match what was outputted in another tool? So, it’s super important for reducing hallucinations as well. Not every agent use case needs that level of control, but I find that there are so many use cases around automating business processes, ensuring there’s low hallucination rate handling data appropriately. That steering helps so much more than workflows.
Sri Panyam 00:20:16 How about hooks? You mentioned that today hooks will tell you or at least will signal what is happening next. What is the challenge in getting that context, I guess context in a lose way about how it got there and where, and why it got there and passing that as well for the hook implemented to act on, as in why isn’t that possible today?
Clare Liguori 00:20:38 The key that we’ve added is what we call the ledger. So, a recording of everything that the agent has done and in a general agent harness, you can probably build that up yourself with hooks. Like you can inject yourself into every single thing that the agent is doing and build that up yourself. But Strands provides it out of the box and also provides out of the box a few steering providers that you can also use out of the box. Things like an LLM as a judge are super valuable tool and Strands makes it easy to just provide a prompt as a steering provider and evaluate what is, what has the agent done and what is it about to do and is that okay?
Sri Panyam 00:21:23 And how would this relate or how common is it to various kinds of memory that an agent might maintain or a developer might provide as a sidecar?
Clare Liguori 00:21:31 So are you asking how it is different from memory?
Sri Panyam 00:21:34 You would’ve maintain your own ledger, which you probably wouldn’t, but I mean if you were to maintain your own ledger or keeping track of that history of what has happened so far and what caught you here, what would you say would be the parallels or similarities and differences between let’s say a memory system?
Clare Liguori 00:21:48 I do see a lot of interesting work around turning agent trajectories into memory. I think of memory as somewhat, maybe a naive way to think about it as a summarization or pulling out nuggets of things in natural language that can be retrieved layer and analyzing an agent trajectory is one way to do that, right? Looking at all of the tool calls that are done, we’ve actually seen interesting experiments internally around saving agent trajectories as memory. So, this gets really interesting where often when an agent figures out a way to do something, you kind of want to remember that so it doesn’t have to figure it out again. So, there is I have an example where we had a use case for this was again, a set of APIs or AWS APIs more of an AWS use case where particular API within AWS requires you to go and gather a bunch of information from other APIs. In order to call it. You have to call I’m not kidding, 18 APIs in order to gather all of the information that it needs.
Clare Liguori 00:22:59 So that’s very challenging to get and especially at the time with earlier models, was very challenging to reliably get an agent to call 18 tools, all of the right tools before calling that API gathering all of the right information. And so, we started to save trajectories that in a semantic, in a vector database keyed by what was the prompt that led to that trajectory. And as it turns out, models are very good at transferring knowledge. So even if a prompt wasn’t exactly the same as what was in this semantic search, it could retrieve something similar and then kind of adapt the agent trajectory to make the right tool calls for the particular use case it was working on. So, I think that’s super interesting where you can also add a little bit of determinism for common use cases. Let’s say your agent has the same use case 80% of the time coming in, saving that trajectory as a type of memory but more procedural memory than what we typically think of memory as like these natural language nuggets getting better determinism and even better efficiency, right? Because it has kind of the answer the procedure it needs to take right away.
Sri Panyam 00:24:17 Interesting. So, it sounds like if you know the prompt, then you can look at the most similar closing of prompts, kind of braggish (?) or vector such and you would use that to see what other similar things is a scenario where these similar looking prompts, they might key to the same kind of trajectory but over time the similarity would just drift. Would that ever happen?
Clare Liguori 00:24:38 And you can even think about maybe later models find a more optimal way to solve that problem too, right? So, you kind of have to think about are you going to expire some of those entries over time? Are you going to reevaluate some of those entries over time or not provide that RAG effectively RAG tool to the agent and see if it comes up with something more, more optimal, more efficient over time. Because again, you don’t want to get into this place where you’ve built all this scaffolding, you have all of this data that’s effectively generated by an older model and it’s just making your system worse. The challenge in the agent building space is that the models have these massive leaps what seems like every three to six months now and so, potentially what you built or what you generated three to six months ago is now potentially making your agent worse than it could be with the latest models.
Sri Panyam 00:26:03 I think we’ve talked about tools so many times in this episode already, but before I jump to it, there was one thing I would I want to ask about one of the hooks, one of the problems that’s tiering hooks or good at solving the problem of handoff. You mentioned hallucinations, you don’t want to have the wrong user ID being passed across tools, right? How does Strands kind of keep me even with having the trajectory and the memory and the history, what goes on with Strands and steering hooks to actually ensure that verification is accurate?
Clare Liguori 00:26:30 One of the things that I appreciate about steering hooks is that it is entirely, or it can be entirely deterministic. I talked about LLM as judge earlier, but it can just be Python code. And so, you can deterministically create a steering hook unit does the bejesus out of it, right? To make sure that the validation is correct. You can use things like, something like Rego or Cedar to actually do something that’s verifiably correct if you want. So, you can get a lot of value out of kind of injecting this deterministic check in the middle. I recently published some findings from a project that I did where I wrote most of the steering hooks using just plain Python and I was able to get a hundred percent accuracy across 600 evaluation runs and I was showing, I think workflows only achieved about 80% accuracy and then a simple prompt was nowhere near that course.
Clare Liguori 00:27:32 But I found that that a 100% accuracy effectively comes from that determinism that you are injecting in there. Being able to deterministically look at what were the inputs and the outputs of the tools, which tool outputs are significant to you, right? Which ones are critical and that’s only something that the agent developer really knows by looking at the tools and what’s the data being passed between. So that’s where it really becomes specific to the agent. Like I can’t give you a general checker of how the tool outputs link to the tool inputs, but you can provide your own knowledge of the data that is transferring between these tools and that the agent is handling about which ones need to be the same absolutely. And which ones are kind of natural language that, like my example agent that I was evaluating with steering hooks did have some natural language things like it would send an email to the user and I’m happy for the agent to come up with natural language to send to the user. And in that case, I used an LLM as judge just to make sure that the tone was correct, it wasn’t rude or anything. And then I was able to do very deterministic checks for the actual data handling in between the tools.
Sri Panyam 00:28:50 In this evaluation run — legacy evaluation set — how did you choose the members of that evaluation set?
Clare Liguori 00:28:55 So I took a sample project, or I took a sample project where it had different rules that it needed to follow. I’m trying to remember, I think there were four types of rules, and I think these generally mapped to a lot of what I see people trying to do in automating with agents, automating business processes. One is tool ordering; this tool must be called before that tool. The other is input hallucination. This input must come from the output of this tool. You don’t want like a user telling the agent to be rude to me, you don’t want it to be rude. And then the last one was an overall workflow adherence. So one of the things I noticed actually I was trying with different, I was trying out different models and I noticed that some of the models, it was interesting, I had a confirmation step at the end, it would do an action and then I told it to send a confirmation to the user.
Clare Liguori 00:29:54 There were models where reliably it would not send the confirmation to the user because it would consider having done the action as done. I’ve asked, I’ve done what the user asked me to do, and it would totally ignore the part of the system instructions where I said, okay, after you do this you have to send a confirmation message. And so that was one thing that steering guided the model to say before returning a model response back to the user, hey you’re not done yet, you haven’t sent the confirmation. And so, it was actually super token efficient because I didn’t have to have this mega system instructions prompt defining all of these edge cases and things like that. I could give it general rules, bullet points, 1, 2, 3, 4 and then if it didn’t follow that, if it didn’t adhere to those rules, I have the steering hooks to guide it back with a little bit of a message back to it. And so, I was able to get that 100% accuracy even with very small models that we would not expect to do well with. Being reliable, I was using GPT OSS 120B0. So really, really cheap, it was a lot cheaper to use, a small model plus steering than to use some of the frontier models that would probably be more reliable but are also much more costly in terms of expense.
Sri Panyam 00:31:22 You mentioned token efficiency, did the actual, I guess conversation size or I guess past size matter, but today, even though a lot of models promise the 1 million context window, you would very hardly, rarely go to the full extent in or have a session that exploits the entire session, right? Because you end up having long context problems and all that. How did this compare with longer context scenarios?
Clare Liguori 00:31:46 The only thing that I compared was what we call SOPs, standard operating procedures for agents. So, a lot of what I see people trying to do to make agents reliable is just longer system instructions. Very explicit system instructions that define step-by-step effectively a workflow of steps that they wanted to take without implementing it as a workflow that is not usually a million tokens, right? But it is longer can be very verbose, right? And you are adding to it over time, I call it the prompting treadmill where, you start with this pretty small prompt and then you run into all these edge cases and you say, okay, don’t do that. Always use these very clear words like always, never, must, but you add overtime to the system instructions and then you’ve got this gargantuous system instruction prompt with all of these caveats in it and it does end up requiring a lot more tokens, right?
Clare Liguori 00:32:55 You get better accuracy, but you don’t get a hundred. I actually found that we have a tool for Strands that does generate these SOPs. It can be useful for simple use cases or use cases where you don’t really mind it not being token efficient. We use it a lot internally for operations like on-call troubleshooting, which are super easily described as a set of steps in a system instruction and the accuracy doesn’t need to be super high. But I found for this use case that I was, that I built and was evaluating with these different techniques for steering, I found that SOPs achieved 99% accuracy for the things that I was testing for. But it was three times the input tokens, which tend to be the expensive ones, right? And so, they don’t cost as much as output tokens, but they can be expensive on big models and so more expensive from that perspective. But you can actually get pretty far just with a well-structured system instruction set.
Sri Panyam 00:33:57 Interesting. And what about long running tasks long coding, long running tasks, how does it fare there?
Clare Liguori 00:34:03 I also didn’t evaluate that in particular, but thinking about it, the context window management is what happens when you get into long running tasks, right? And what’s interesting about the steering approach is that if you get the full ledger, the full ledger doesn’t need like the full conversation history effectively doesn’t need to be shown to the agent. In fact, can’t be shown to the agent when it’s very long. But you can still analyze it, right? You can still write a Python function that looks at a huge conversation history going back forever to determine if your rules are still applying, if it’s still adhering to a workflow. And so that makes it a little bit different from let’s say an LLM is judged where the context window is going to be limited as well or even workflows. I think what we find with agent workflows is that you lose a lot of contexts in between the steps, right? You have one each node in the workflow, you lose a little bit of context going from node to node. And so, what’s interesting is you probably are not going to build up that much context, but you also are probably missing context from previous decisions in the workflow.
Sri Panyam 00:35:19 Yeah, fair enough. How does one get started writing all of these Python functions to as steering hooks? Like what are the gotchas, what are the constraints?
Clare Liguori 00:35:27 We have some pretty good resources. I published the example, the complete example that I evaluated when people get a hundred percent accuracy on. And so that’s a good place to start. And then you can also go to strandsagents.com. We have a full user guide on using steering hooks.
Sri Panyam 00:35:45 Switching to tools I guess, tools are really integral fundamental parts of an agent harness. I think every agent understands today integrates with tools. So where does Strands take it on from? How are the tool support and integration in Strands same and different from your typical harnesses?
Clare Liguori 00:36:04 I think that, like you said, every agent framework has a tools concept, has a way that you can write tools, Strands supports Python, so you can provide like a pedantic style function and it will turn it into a tool. Similarly, we are GA the TypeScript SCK for Strands. We’re coming to a second language, very exciting. And that also supports providing Zod types for functions to turn that into a tool. We try to make it really easy and developer friendly to just take a regular function and turn it into a tool. But largely we also see MCP becoming critical to kind of the tool ecosystem, right? It’s super powerful to be able to pull a MCP server off the shelf from one of your SaaS providers or just a utility MCP server and connect it to Strands. And then what I’ve been seeing a lot is companies thinking about how do I connect my internal APIs to agents now and MCP becomes that kind of lingua franca.
Clare Liguori 00:37:19 MCP becomes this lingua franca for agents where you have an MCP gateway that turns, let’s say an open API schema for one of your internal APIs into an MCP server with tools and all of a sudden any agent developer in your company can start to connect APIs together in really powerful ways. So, we’re seeing a lot of investment in thinking about how to bring internal APIs to MCP just in order to bring them to agents. And that’s probably not trans specific, right? I think every framework supports MCP, but it’s certainly becoming an important part of agent development in general.
Sri Panyam 00:38:01 In terms of Strands own kind of native tooling mechanism, right? How far do you see taking that versus switching more into MCP or where do you kind of dial that knob? How would you help developers choose one or the other?
Clare Liguori 00:38:15 The way I like to think about it is methods versus libraries in regular software development. When you find a piece of code that can be used in a lot of different places, you turn it into a library and I think of MCP servers and sort of native framework specific tools in the same way. MCP came about because all of these frameworks had different ways of defining tools and there wasn’t really a way to reuse them outside of any particular agent and outside of any particular framework. So when you start to see this tool is really valuable, I may have built it in the Strands framework or anything else, this tool is really valuable, I want to share that across a lot of agents turn that into an MCP server or I already have this API I want a lot of different agents to be able to connect to it, turn that into an MCP server.
Clare Liguori 00:39:12 We still find a lot of folks writing Python and TypeScript based tools and Strands using our native support for tools. It’s really easy. It’s like the easiest way to get started when you’re building an agent and you want to add tools; you just start to write them as Python functions or TypeScript functions. It’s so easy, right? And then over time you find the things that are reusable across agents. One of the things that we released recently is something called AI functions, which is interesting. We created a new GitHub organization called Strands Labs to be a place where we could be a bit more experimental, right? Internally and among users we have folks that are relying on Strands for production use cases. So, we don’t make a lot of breaking changes, right? But Strands labs is a place for us to think boldly about where is agent development going.
Clare Liguori 00:40:05 One of the things that we released is called AI functions where today when you write a function as a tool, provide the inputs and the outputs and then separately you write steering hooks to validate what the model is doing. Or even regular hooks validate the inputs and the outputs to that tool. AI functions kind of combines them together in a crazy way. You write a regular Python function just like you would in Strands to create a tool, but instead of actually writing a deterministic function, you write a prompt for the function itself and at runtime it turns that into code and executes it. And then you can also provide what we call preconditions and post-conditions. Basically, what must be true going into this function, like the inputs must be like this where we must have some system has to be in this state. And then what are the post conditions?
Clare Liguori 00:41:04 What should the output look like? What steps should have been taken? So, the example that I usually give is a method that’s going to do OCR on a receipt image, and you have an image coming in and you need structured output coming out. You need what was the total cost on the receipt? What was the vendor, what were the itemized things? So, there’s some structure to it. Instead of trying to write all of that OCR code yourself, you actually have it do it at runtime, you have it generate code at runtime, execute code at runtime, and then you have these preconditions like I must have an image and then post conditions, like I need the structured output and it will continue to prompt the model. Again, going back to that feedback loop for the model and it being so adept at self-correcting, come make sure it comes back with that structured output from the receipt.
Sri Panyam 00:42:00 That’s interesting. So, did you find that there are certain runtime or dynamic kind of conditions in the space of problems that the agent is solving, which I guess prevented you from left shifting it to ahead of time generation?
Clare Liguori 00:42:15 Because a lot of this could be generated by our coding assistance, right? We no longer have to actually write this by hand, right? One of the things that’s interesting about kind of the theory behind AI functions is when you run an agent many, many times you’re probably going to get this kind of, if you generate code many, many times against the same prompt, you’re probably going to get this Gaussian cloud of code solutions for it. And some of those are optimal and some of them are not. And when we do coding, it’s largely one shot to the model accepting like, running it through tests and things like that and getting the coding assistant to self-correct. But what happens when you run that a thousand times in production, you’re going to get a thousand samples of what code is optimal. You’re going to get real runtime feedback about how did it performed against real inputs, which is hard to do in a pre-production environment with just test data. You’re going to get information about the latency. How often did the model have to self-correct itself until it achieved the post-condition? And so, the theory here is over time, can we arrive at the ideal deterministic code and shift that left at that point, bring that back into the code base over time once we find those very optimal solutions to some of these problems.
Sri Panyam 00:43:48 In interesting, in fact, the way you’ve phrased it, the interesting takeaway from for me there was you can both capture the runtime conditions across a 1,000, 10,000, 100,000 runs and use that as both new inputs or new conditions for your function, right? Hey, now I’m running on a dark background, now I’m running on a, I don’t know, a different language, whatever, right? But if that’s the case, do you ever need to have a final deterministic version at all? Can you just go, what our system is own mini classifier that keeps generating this classifying function or whatnot and we’ll keep using that as long as we have the way to capture details about what is working, what’s not working, some way of getting feedback.
Clare Liguori 00:44:31 Well that comes back to your point about memory, right? If we can build a memory system for this AI function and that’s what we’re working on now, if you can build a memory function of here’s the code that I produced here was the prompt and here is how it performed, then the agent can start to choose what are the things that are bubbling up with some amount of kind of probabilistic noise to make sure that it’s improving over time, but arriving at maybe a few optimal solutions or trying different things over time, but mostly choosing what is the optimal solution. So no, I don’t think you ever necessarily need to shift left, especially as the models get better, right? Think about all the code we wrote with Sonnet four checked into our code bases and now we have Opus 4.7 now. And so think about how potentially how much better that code could be if it was regenerated with latest models.
Sri Panyam 00:45:31 You mentioned the AI deciding what’s better, what’s worse. So, is AI in this case identifying and assigning the rewards or there was somewhere else?
Clare Liguori 00:45:39 That’s an interesting question. I think we haven’t really thought through the rewards functions yet and potentially the rewards functions are maybe use case specific, right? Today the reward function is the post-condition, which is very binary, right? Of this code works or it doesn’t work. But I think we haven’t gone into defining the reward function around the gradient of performance or the gradient of how many turns did it take with the model to get it to actually produce the right result
Sri Panyam 00:46:11 Or even the quality of the runtime environment at that point in time.
Clare Liguori 00:46:14 That’s right.
Sri Panyam 00:46:15 No, nice. Looking at some of the practical aspects with tools and the various harnesses as trans obviously, what are some of the deployment patterns? You identified, you recommend, you would learn from, you would teach us?
Clare Liguori 00:46:29 Well I think the thing that we’ve learned over time using Strands is start as simple as possible. The vast majority of agents that I see can follow the model driven approach, maybe add in hooks and steering for runtime checking, but generally speaking can be a single agent. I think engineers want to build and it’s easier than ever to build. And so, I see a lot of engineers building these very complex orchestration systems that for the vast majority of use cases are, are just not necessary. You’re building more than you need. And so, what I counsel customers on is build the simplest thing first and evaluate and see how it does. And often you’ll be surprised, you’ll be surprised at the power of these models that we have at our fingertips today. The one other pattern that I do like a lot, I don’t like workflows as I think we’ve talked about are graphs, but subagents for context management I think is still something that is a very common and useful pattern because even with million token context windows, you might not want to fill it up all the time with context windows that are much lower, especially subagent as tools are very powerful.
Clare Liguori 00:47:51 So effectively letting your main agent choose when and which subagents to invoke as tools and that’s really the pattern is subagents as tools and that allows you to effectively encapsulate any contact and thinking and tool calls that the subagent is doing while it’s executing on a very particular task. One that I have used recently was I built an agent that needed to go pull a bunch of websites and get very specific information from each website extract like an event for example, from a web, from a bunch of different websites. That’s a ton of context to carry around for your entire agent. And so, the main agent was able to invoke a subagent whose sole job was to extract an event pull one website extract event return event. So instead of carrying around the full HTML content from all of these pages, it was able to return to the MA agent just that event and throw away the rest. That’s very much a pattern that I love and use.
Sri Panyam 00:49:03 Hey, it’s almost like we are going to the roots of software engineering and running functions and not one big function.
Clare Liguori 00:49:08 And you can get very sophisticated with it of like having an input schema and an output schema for your subagent literally just like functions, right? We are learning encapsulation all over again.
Sri Panyam 00:49:24 No, no, it’s interesting how we, I mean in a way look, if it’s improving the awareness and appreciation for software engineering, I think that’s amazing, right? Subagent is one thing, what are multiagent? Multiagent capabilities? I guess without generalizing subagent as multi-agent, where would you draw the parallel? Where would you draw the line? How would you kind of look at what Strand offers today?
Clare Liguori 00:49:46 Yeah, so Strands offers multi-agent patterns and we offer a really great guide on our website, it’s transagents.com that walks you through kind of generally with this model driven approach. When should you think about using some of these patterns versus, again, the vast majority of use cases that just need a simple, simple agent, multi-agent I find most use cases don’t need. What I will say is I find that they are often necessary for organizational reasons rather than technical reasons. What I mean by that is, let’s think about microservices. Why did we move to microservices? Because different teams can own different logic and they can move independently of other teams and we can have APIs as the contract between us. What I am seeing in companies that do have these multi-agent use cases is often it’s kind of like microservices. We’ve got micro agents where one team is owning an agent and another team is trying to integrate with that agent.
Clare Liguori 00:50:50 That agent becomes an API that is called and you start to get into these, multi-agent use cases as opposed to a simple tool that’s calling a simple API. These agents are going off and doing work and they’re long running. And so the interaction pattern looks a little bit different from typical tools. I see that a lot and that again coming back to software engineering basics, it feels very much like that the separation of concerns from more organizational reasons, which is also valid, right? But a little bit less than this would also probably be just as accurate and powerful if it was all in one system.
Sri Panyam 00:51:30 Yeah. And what are some of the communication pat patterns that differ from subagent versus multi-agent how would you kind of characterize those?
Clare Liguori 00:51:38 What I tend to see is folks trying to think about how much of the previous context from the main agent needs to go into the subagent or the other agent exactly? Sometimes for very well-scoped tasks, very little, right? In my example of pull down a website extract event, all that sub agent, or even if that was another agent owned by a different team, all it needs is a URL to go and pull. It doesn’t need any of the previous context, but if you think about, maybe a very complex process, like refund processing, is a big one. Everybody wants to automate refund processing, but that touches so many different systems and some of those teams have built different agents. And so, you start to build up a lot of contexts for what those different agents need. It needs to know what the item is, what is the risk profile for this customer? What interactions have you had with the customer already about returning this item and what did they say? So, I think it depends on just kind of what the use case is and what the subagents maybe need. And that’s going to be dependent on the use case of what those do, right? What is the information they need to be as accurate as possible.
Sri Panyam 00:52:56 I mean, if you remove the LLM for a second from all this, then you’re describing or are we talking about giant, giant orchestration workflow, various kinds of saga patterns all over the place, right? So, it’s almost like 90% of this is plumbing that has to go no matter what is building it or who’s building it.
Clare Liguori 00:53:15 You can imagine in those orchestration systems we’ve had shared documents, right, that describe what’s happened so far, what is the information. So, you can think about that shared document for all of these subagents, but then again, you have to worry about the context window, right? Yeah.
Sri Panyam 00:53:32 Which reminds me, so context window is one thing. What a history, I mean history across agents. Is it more for renewability model? Is it a transfer model? When I say model out in the LLM, just the pattern like how is history I guessed managed in this scenario or in this setup?
Clare Liguori 00:53:48 I think one of the changes that I’ve seen with harnesses in particular is a different way of thinking about history. Often with a harness, you’re going to call the same agent over and over again until done, right? We have the Ralph Wiggum loop effect, right? But that means that agent needs to save important information somewhere, even if it’s thinks it’s done, it needs to save what is the state of things that you have left it in. In coding this is what is the state of tasks, coding tasks that you’ve done. But in other agents it’s going to be something else basically like save my state to disc and often it’s just disc, which is so like easy and nice, but then when the agent wakes up again because it’s not actually done, it can resume from that natural language description that it saved for itself and its past life, right?
Clare Liguori 00:54:44 So it’s almost like I think of it like, sometimes at the end of the day I’ll download my brain into Obsidian or something like that so that the next day, one so I don’t have to think about it at night and two, so the next day I wake up and I say, oh, what was I thinking about yesterday? What did I want to get done in the morning? There it is. It’s kind of just like that with agents and they’re very good at prompted to kind of save the state of their work and then resume from that. I think that’s the biggest thing that I’ve seen. Biggest pattern I’ve seen emerge just in the last like four months as agents harnesses started to get popular is that kind of save your work pattern, which is completely outside of conversation history, right? It’s about the agent itself. Almost summarizing what are the important things to resume from the context window that I have.
Sri Panyam 00:55:40 What if history itself was a tool, let’s say if history was nothing special and if it just happened to be a tool call that the agent knows how to effectively call, do you see anything happening in that sense? Or is that even useful? Is that valuable? Is that just?
Clare Liguori 00:55:51 I’ve seen that used for a while. I mean in a lot of agent memory systems you have three different levels of agent memory. You have that quote unquote short-term memory, which is not necessarily short term, right? It can go for a long time, but it’s that conversation history, that raw conversation history that the agent has the tail end of it basically. But you can search back in time so that somebody can say, what were we talking about yesterday, right? And it can go back and look at that raw conversation history and then kind of the, what some people kind of call medium term or long term where it’s, you just have a kind of a rolling summary of the conversation history, but you’re going to lose a lot of important details in that. And then you get to this like semantic memory where you pull out these, again, these nuggets, right? This person likes Java over TypeScript or this person has a very, somebody told me was looking at what their agent had saved in its memory and said this person uses a lot of British schism, uses a lot of British slang. And for whatever reason the model thought that was very important for its future interactions, right? So, letting the model decide what’s important.
Sri Panyam 00:57:08 Okay. Fair enough. I think history and memory, we keep going, well it’s only been 18 months, but we do keep going kind of back and forth on that dial. Right? So, literacy kind of where they land. I want to touch-base on some of the production aspects of your deployments. I mean AWS, Strands is very heavily used in AWS from what I’m hearing. I’d love to learn more about the lessons from scale on which products, how does the product specific thing come in?
Clare Liguori 00:57:34 Well, we worked really hard to make sure that Strands was production ready. What I mean by that is that you don’t have to add in all of the things you need to put it in production. It supports open telemetry out of the box so that you get observability over traces and metrics, things like that. We added an out of the box session persistence so that you can save it to S3 or a file or whatever it may be, because that’s very important once you get into production to actually persist these things. And we did that first for ourselves, right? We were building production systems with this that led us to build and get agents to production really quickly. Within three to six weeks is what we were seeing for time to deploy these new agents to production. And we’ve seen that over and over again inside of Amazon as well as outside.
Clare Liguori 00:58:31 We just recently added a managed, kind of a managed Strands service called Agent Core Harness that runs Strands under the hood, but it also combines in all of the other things that you need, like a memory system, an identity system. What does the identity of the agent, how does it get credentials? All of the observability information going into CloudWatch connectivity to your MCP gateway all of these things that you do still have to set up when you are creating an agent. You have to deploy it somewhere to some runtime agent Core harness takes care of deploying it to a new VM, connectivity to the models. All of these things that are, are still steps that you need to take to get it into production. So Agent Core Harness takes out a lot of that work and just lets you provide what is the model, what is the prompt, what are the tools so that you can focus on what is it that you want the agent to do instead of kind of all of this other stuff that you, we’ve seen customers build over and over and over again to get into production.
Sri Panyam 00:59:38 Oh nice. How can customers today use open telemetry, for example, or traces, for example, to I guess walk through a CH agent — a core harness, right? — to understand, I guess, which path the agent took at various parts of the run?
Clare Liguori 00:59:53 So locally you can just see them get dumped out of the open telemetry exporter. It’s interesting, I was looking for a really good desktop application for open telemetry traces. I feel like there’s not a great one. So, if somebody wants to build that, I would love to use it. In production and in in your testing environments, you can export it to your favorite OTel-compliant provider, be that CloudWatch or any other tracing provider, and use their visualization to kind of walk through what it is that the agent does because it’s going to show you all of the things that you just don’t see when you get the final output out of the agent and you’re wondering what just happened? How did it arrive at this conclusion? So you can see the tool calls that it made, the tool inputs, the outputs, the thinking that it did as it was choosing the tool like, ah, I need to do this first, now I need to do this.
Clare Liguori 01:00:52 Seeing that is super useful in working back from what the result was to what actually happened there. And then, you can also use those for offline evaluation. Those are super valuable for once you’ve run this agent in production a thousand times, being able to look across and say, how many of these trajectories do I think are good and how do I, maybe tune the system prompt or the, the steering hooks or something to make these trajectories a little bit better. And then, looking at, again, using that same offline evaluation with whatever system prompt tweaks or thing that you’re doing. And then often we see customers investing in AB testing because it’s so hard to test agents. It really is. And sometimes there’s just no replacement for testing it with real customers, right? When with these agents. And so before even at Amazon in AWS, we’d never really invested in AB testing because we would always do just testing.
Clare Liguori 01:02:01 Like we would just have deterministic tests. You give this input to the API, you get this output back, it works, right? But with agents, it’s obviously not true. You’re not going to get the same answer twice. And so, we’ve started investing in AB testing for agents quite a bit internally for our own agents because it can be very difficult to predict what people are going to ask this thing? Or what inputs are they going to give? And it’s very challenging to confidently roll out changes to your tools, to your tool descriptions, to your system prompt, to the model even just going a 100%. And so, yeah, we’ve been doing a lot more AB testing where we roll out to 10% of users for a couple of weeks and see how it goes.
Sri Panyam 01:02:48 It’s like changing every part of your car while driving it, right? I mean in terms of adoption and best practices, I think you mentioned start off small, start off simple, right? What are some of the common anti pattern and mistakes you see in developers kind of make when they start off with agents beyond?
Clare Liguori 01:03:07 They build too much? Playing with agents is so fun, and so I can’t blame people for wanting to just build a bunch of stuff around agents. And I think also people build agents when they probably could have built an MCP server for example. That one thing that we are seeing a lot internally is, we went through a year, like two years ago where it was like, okay, everybody’s going to build an agent and then we ended up with a million agents, right? And there’s a lot of overlap between them, of course, and it’s very hard to understand when to use what agent. And so, I think now we’re on a path of, well, maybe everybody should just provide context to a few central agents. Everybody’s got an agent harness in their pocket now with coding assistance. Everybody’s got access to easily build an agent harness with Strands.
Clare Liguori 01:04:07 But often I think the question to ask yourself is, should you be building an agent for this? Or should you provide context to a centralized agent? That’s really the process that we’re going through now where we’re trying to figure out what is that data that we want to make available to agents. So, recently at AWS we put the AWS MCP server in preview. What that does is provide tools for agents to call AWS APIs and retrieve skills about how to do things with multiple AWS APIs. And the learning there was, instead of trying to do this kind of provide an agent, for example, and try to expose it over HOA and all of this, probably our users just want an MCP server and have their agent do it themselves. That’s a lot simpler. That’s a current pattern for me is pushing teams to actually not build an agent sometimes. And instead to think about what other ways to make your data and your context available to agents.
Sri Panyam 01:05:13 And as they do this, what would be like a minimum set of things that they monitor and kind of observe early on, at least to know that your agent isn’t kind of going off the rail soon than later.
Clare Liguori 01:05:24 Evaluation is super important for both creating agents and even MCP servers because if you think about the tools that are available through your MCP server, that’s going to impact people’s agents. If your tool descriptions aren’t good, the agents using that MCP server are not going to be very powerful. So, evaluation is super important and figuring out what it is that you want to evaluate for. I always tell people, if you want to build an agent, start with a use case where you can deterministically evaluate trajectories. So, things like, were the right tools called for a particular input? Were they called in the right order? Did they have the right input? The harder things to evaluate are things like chatbots are honestly difficult to evaluate because you’ve got these multi-term conversations and you end up trying to do these LLM judges around like, how good is this conversation? I don’t know. It’s kind of up to the user how they felt about it, right? But a lot of these automations use cases are actually really great, low hanging fruit because they can be evaluated against some kind of existing process.
Sri Panyam 01:06:37 As we wrap up, what’s next for Strands? What’s in your kind of horizon?
Clare Liguori 01:06:42 We are going farther into supporting folks who are building agents on top of an agent harness. I think that is really with the power of models today, there’s so much power around that’s been enabled for agent harnesses. And so I think a lot more around how do you keep agents running for a long time? What are the things that you need out of your agent harness to do that? And what are the new patterns for agents? Things like event driven agents, very, very long running agents that are interruptible are all in my mind right now.
Sri Panyam 01:07:20 Where can our business cell learn more about Strands and get started?
Clare Liguori 01:07:23 Strands agents.com.
Sri Panyam 01:07:25 Are you presenting any other communities? Are you talking elsewhere? Do you have any conference coming up that they can listen to?
Clare Liguori 01:07:32 No.
Sri Panyam 01:07:33 All good. We will definitely be adding all the resources that you mentioned here in the show notes and on the page itself. Any last words of wisdom for our listeners? Have
Clare Liguori 01:07:43 Have fun with agents. Start with model, tool, prompt.
Sri Panyam 01:07:48 Awesome. Thank you. Thank you. Thank you very much, Clare for joining us on Software Engineering Radio.
Sri Panyam 01:07:48 Awesome. Thank [End of Audio]
