Birgitta Boeckeler, a Distinguished Engineer and consultant focused on AI-assisted software delivery at Thoughtworks, joins host Priyanka Raghavan for a deep dive into harnesses for AI agents. The episode begins by unpacking the concept of harnesses and harness engineering before exploring the core building blocks — guides and sensors — that help AI agents operate more reliably in engineering environments. Priyanka and Birgitta discuss practical implementations of harnesses in real-world workflows, including the use of guides with .MD files and sensors with tools such as SonarQube and Semgrep, which steer agent behavior. The episode also explores how harnesses integrate with existing CI/CD pipelines and pull-request processes. Birgitta describes how stronger harnesses can improve trust in AI-generated code, while emphasizing that harnesses themselves require continuous maintenance as underlying foundation models evolve. The episode concludes with a thoughtful discussion on accountability between humans and agents, along with future directions for harness engineering and AI-assisted software development.
Brought to you by IEEE Computer Society and IEEE Software magazine.
Show Notes
Related Episodes
- SE Radio 711: Scott Hanselman on AI-Assisted Development Tools
- SE Radio 603: Rishi Singh on Using GenAI for Test Code Generation
- SE Radio 589: Zac Hatfield-Dodds on Property-Based Testing in Python
- SE Radio 693: Mark Williamson on AI-Assisted Debugging
- SE Radio 710: Marc Brooker on Spec-Driven AI Dev
Other References
- Birgitta Boeckeler
- Thoughtworks
- Harness engineering for coding agent users
- Cyclomatic complexity
- Harness engineering beyond skills: Using sensors to keep your coding agent in check
Transcript
Transcript brought to you by IEEE Software magazine.
This transcript was automatically generated. To suggest improvements in the text, please contact [email protected] and include the episode number and URL.
Priyanka Raghavan 00:00:19 Hi everyone, this is Priyanka Raghavan for Software Engineering Radio. And my guest today is Birgitta Boeckeler and the topic is Harness for Coding Agents. Birgitta is a principal consultant and a software developer with Thoughtworks and is passionate about helping teams and organizations break down complexity and find new perspectives to look at their systems. She’s a frequent conference speaker and podcast guest on many platforms. You can just Google her or YouTube her and welcome to the show, Birgitta.
Birgitta Boeckeler 00:00:49 Yeah, Hi Priyanka. Thanks for inviting me.
Priyanka Raghavan 00:00:51 Is there anything in your bio that you would like the audience to know other than what I’ve mentioned here?
Birgitta Boeckeler 00:00:56 Yeah, maybe the thing that’s interesting is what my current role is or what my role has been for the past two and a half years at Thoughtworks. So, in 2023, Thoughtworks decided to introduce a full-time role for somebody to just look into the topic of using generative AI or language models for software delivery and how it changes that we’re a consultancy, so we have to always keep our advice up to speed. And speed is definitely something that’s happening a lot in this space. So yeah, I’m a distinguished engineer there and I’ve basically been fully immersed in full time in this space for the last two and a half years and so kind of seen the history.
Priyanka Raghavan 00:01:33 Okay, great. So that will bring us right into our show at Software Engineering Radio. We’ve done a few shows on AI and software development, whether it’s Episode 711 with Scott Hanselman on AI-assisted Tools. We did Episode 603 with Rishi Singh on Using GenAI for Test Code Generation and Episode 693 on AI-Assisted Debugging. So, before we start the show, I thought we’d spend some time on the first part of the show where we’ll go through some definitions and I’m heavily quoting here from the article you wrote on harness engineering. So, to start with, what exactly is harness and harness engineering and how is it different from say, prompt engineering or context engineering?
Birgitta Boeckeler 00:02:17 I mean, as I just mentioned and as everybody’s feeling right now, this space is really fast evolving and one of the challenges there is what language we’re all using, right? I mean, we can come up even faster with stuff as well, right? So, we’re all throwing lots of terms out there and I think that’s been a challenge for me, to find the right words to describe what is happening because that also helps me think about it better, right? And this word harness, it has gone through a little bit of an evolution, but recently it’s increasingly being used for describing something that sits on top of a language model and orchestrates everything that we want to do with the model, right? So, it has a system prompt, it can ask the language model for tool calls and stuff like that, right? So basically, what we’ve also been calling an agent, and then when we talk about a coding harness that as examples, that would be Cloud code or cursor or the PI coding harness.
Birgitta Boeckeler 00:03:12 So increasingly people are calling that harness, right? And then in combination with the model, when you run it, it becomes an agent, right? So, I’m almost starting to think of like an agent as something that is an instance of something running and maybe harness is the tool, the skeleton of what runs it, right? So that’s where my head is now, right? And then harness engineering is how you make those harnesses better. So, for example, in a coding harness, you can make it better by thinking about providing it with different tools or there’s other schools of thought where you make it better by providing it with less tools and with less complexity, right? So, there’s these different approaches to how to make the coding harness better. But then the part that I wrote about in my article is about the users of the harness.
Birgitta Boeckeler 00:03:55 So us or me as an application developer, I’m using Cloud Code as a coding harness and I’m trying to extend the harness for my specific code base, right? So, I’m trying to give the harness the agent, whatever we want to call it, even more information, even more tools specifically to my application that I’m working on to make the results better. So that’s, it’s almost an onion, multiple layers type of thing. So first I choose the base harness that I’m going to use, for example, Cloud Code, and then I give it even more stuff like skills and tools and stuff like that to get the results that I want for my situation.
Priyanka Raghavan 00:04:29 I think the timing seems apt because most people are now using AI coding agents and AI agents for doing all of the coding work. And I guess the question I wanted to ask you is when do you feel there’s a need to have a harness? Is it immediately, as soon as you start coding with the coding agent?
Birgitta Boeckeler 00:04:47 I mean the coding harness itself you need, right? That’s the coding agent basically. That’s what I was saying. The terms that people use are changing right now a little bit, right? But when I think about that expanded harness, you can totally start without anything. And I see this also as a barrier that some people now who are coming to agentic coding a bit later maybe, and there’s all of this tooling and all of these terms and it’s almost some people before they even want to start, they ask, oh, what skills do I need? What should I put into the AGENTS.md? It’s almost they’re afraid to get started because it feels there’s this whole discipline of how to use these things, right? But you can absolutely just get started plain vanilla without putting anything in there. And I would actually recommend it when you first do agentic coding just to feel what it actually does without it, right?
Birgitta Boeckeler 00:05:31 So it’s like a big skill that we now need as developers, I think, to understand what’s actually going on, what tools are already there, how does this tool work out of the box? Like I said before, it might be quite different using Cloud code or Cursor versus a much more lightweight harness the Pi coding agent, right? So, getting a feeling for what already works without putting anything in there. The models have gotten a lot stronger. I feel some people who have already built-up skills and lots of instructions might want to revisit that as well with newer models because some things are maybe not necessary anymore, right? So, it’s worth understanding what it feels to work with them without your custom stuff and then build your own stuff up step by step on top of that.
Priyanka Raghavan 00:06:12 So you describe as the harness is something that goes around the model, right? So can you break it down to us, for example, in the article you talk about these things called this guides and sensors. Can you break that down for us?
Birgitta Boeckeler 00:06:24 Yeah. So this, again is all about I was trying to come up with vocabulary for us to talk about this, right? So, this is not necessarily new stuff in particular the part that I call guides. So, by guides I mean everything that we feed forward into the agent. So, we try to anticipate both what we wanted to do, but also what we do not want it to do and we write it down, right? So, this is what lots of people are already doing and have been doing for quite a while. It usually manifests in a bunch of markdown files. So, the AGENTS.md or the CLAUDE.md file is the most prominent one that has been around for quite a while. And so, we write into that file, I want you to do this, I do not want you to do that, never ever do that.
Birgitta Boeckeler 00:07:03 It’s this important, this type of stuff. And it can actually be all kinds of different guides, right? So, it can be normative ones, right? That sets up the norms and the conventions that we want to use in this repository. Or it can be informative, right? So, it can describe the context of the application, what we’re trying to achieve, describing what the architecture is or stuff like that, right? So, it can be lots of different things. And then on the flip side of that, so this is feed forward, we anticipate and try to increase the probability that it will do very well on the first go, but it’s often not perfect on the first go, right? So, then we want to give it feedback as well. And that’s what in the article I call the sensors. So, there we think about what sensors I want to make available so that they can self-correct before I even have to look at them and find all of these maybe even small hygiene type of things, right?
Birgitta Boeckeler 00:07:57 And a sensor can be both another large language model and the large language model itself, right? So, a sensor can be a skill that has code review instructions, right? That again tells it we want this, we don’t want that, this is bad, this is good. But a sensor can also be something computational, right? Something that actually runs on the CPU, not on the GPU. So that would be something like static code analysis or our test suite or coverage data or any of those types of things. And on the feed forward side, by the way, we can also have computational guides. So the most powerful example of that I would say is that of code more tools. So those are tools OpenRewrite are very prominent in that area. There are a bunch of those in the JavaScript space as well. And so code more tools basically can do mass refactoring, mass changing of your code base, right?
Birgitta Boeckeler 00:08:50 And there are lots of case studies and use cases out there where you combine language models and these Codemods as extra guides to do migrations and upgrades and stuff like that. Version upgrades that require a lot of stuff, right? Where this computational guide can maybe cover 80, 90% of the work and then the rest of that is a bit more semantic and the large language model fills in there, right? So, the guides are all about increasing the probability that it does a good job in the first place. And then the sensors are about giving it direct feedback and maybe a little bit of additional guidance to self-correct.
Priyanka Raghavan 00:09:26 So do you feel that the role of the feedback loops makes the code more reliable?
Birgitta Boeckeler 00:09:32 Yeah, definitely. I mean, there’s a school of thought that the language models will just get better and better and better until they’re just perfect at coding and the code will always be perfect, right? But yeah, I don’t think that’s realistic or if it ever gets to that point that we do have setups that can write good code almost all of the time. I think it will have to involve these types of sensors to help the model correct. Also, in the existing big experimental and less experimental stuff where people orchestrate lots of agents that just collaborate with each other and they orchestrate and they go off and do a task, there’s also a lot of sensoring going on, right? There’s censoring with the sea almost each other, right? So yeah, I think they play a huge role in my experimentation that I’ve done with, for example, static code analysis. We can go into that a bit later if you want. Or checking for coupling, lots of stuff about the maintainability of software. I’ve definitely seen them kick in a lot even when I was using powerful models to do the coding. So yeah, there’s always little flaws the same as when we as humans write code and it’s just a great feedback mechanism.
Priyanka Raghavan 00:10:40 Are architectural constraints good guardrails for AI agents?
Birgitta Boeckeler 00:10:44 Yeah. I actually write in the article about how, to me, there’s different dimensions that we can think through for the guides and sensors, right? Like, what are the things we want to regulate for, right? So, I was thinking about some parallels to cybernetics that the code base that we’re generating is actually something that we’re regulating, right? And so, these different things that we’re regulating can be maintainability, for example. So, trying to make the risk of a change in the future as low as possible. Or it can be architecture fitness, right? So, thinking about sensors and also guides that describe what we want in the architecture and then sensors that check if it’s there, right? Can be around the typical -ilities and architecture — fitness, functions, performance, and those types of things. Or in a way you could say architecture is also, people often call code-based design architecture as well, right? So that could be about coupling and stuff like that, which maybe goes in the direction of maintainability again. And then of course another big thing that we want to regulate is that the code actually does what we want it to do, right? So functional correctness. But so yeah, your question about architecture, depending on what one means by architecture, which is a big topic as well. But yeah, I would say absolutely.
Priyanka Raghavan 00:11:52 I noticed something on Twitter a few days back where someone said that they tried to make a prompt, very abstract and then the AI agent didn’t perform very well. So, I guess it had to be at a level where it could understand abstractness it doesn’t do very well with, right? So I think in that sense it does make sense about what you’re saying about defining the architectural constraints properly so that the AI agents can pick it up.
Birgitta Boeckeler 00:12:15 Yeah, I mean it depends on what is meant by abstraction, right? When you create an abstract instruction, that abstract means with little detail, right? That could be a problem. But of course, language models are actually sometimes good at doing abstractions, right? It’s sometimes all that they do because they translate stuff, right? They right translate from one thing to another. So, I think it depends on what is meant by the abstraction in that context. But you were also initially asking me what’s the difference between harness engineering and prompt engineering and context engineering? I would say it’s all different forms of a similar thing, right? So, I mean we started two and a half years ago with saying prompt engineering, super important, right? Exactly how you craft your prompt and where you put stuff and you give the agent or the LLM a role and which order do you put it?
Birgitta Boeckeler 00:13:02 And that has become a lot less important with the big models, right? If we want to go back to smaller models, there are lots of good reasons for it might be more important. Again, then we came to context engineering, which is all about integrating ideally the information that the model. So, you tune the information that you give to the model in a way that you get better results. And I think harness engineering is a specific form of that. So, with guides you tune information and instructions, right? And then with the sensors, again, you give it information. So, I think it’s a specific form of context engineering, but yeah, crafting the prompts exactly is not as important anymore with the latest models.
Priyanka Raghavan 00:13:41 So harness engineering, it can be also coupled with all these traditional practices unit testing, integration testing, CICD, and static analysis, right? Can you talk a little bit about that?
Birgitta Boeckeler 00:13:54 Yeah, so that’s what I’ve been experimenting a lot with on the sensors side of things, right? So, we have a lot of existing tools actually that we can use and now make available to these agents. And interestingly, some of those tools that we have, we have not used a lot in the past as humans, right? So static code analysis, for example, I’m a consultant, right? So, I’ve seen lots of different organizations build software and I often see them having a setup of code analysis, a sonar server somewhere in the corner, but then it’s not really being monitored or used. And I think one of, there’s maybe two main reasons, one of them is that I think especially experienced developers are aware that it can only do so much, right? It’s not a linter can actually guarantee the quality and changeability of a code base in the future.
Birgitta Boeckeler 00:14:44 But the other thing is also that it often becomes noise overload, right? or the signal to noise ratio was often not very good, right? So, you get even when you use them from the beginning in a code base, you often get overwhelmed by all the messages and you don’t want to fine tune every single one where you want to make an exception or not. So, you just get overwhelmed by it, right? And I think there’s a new opportunity now with large language models and with coding agents to maybe get through that and actually have a good baseline that is always clean with the analysis methods. Because what you can do is you can write custom messages for these lint messages, right? So, let’s take an example, a very common pitfall or a very common failure mode of AI coding agents as well.
Birgitta Boeckeler 00:15:31 Functions that have too many lines that are too long, right? Like max-lines per function or whatever. I was doing this with ES lint in a TypeScript code base, so it was max lines per function. So now I overrode the message that comes back, it doesn’t just say this function is 120 lines long, we only allow X lines long. But it also says this might be a smell. Please consider if we should refactor this, if we should split this up, if this function is doing too many things, make a judgment call. But if you decide that it’s okay if this function is five lines longer or we just can’t do anything about it, or it’s just a test data function or whatever, then you are allowed to create a rule that increases that threshold for this function. So, you give the large language model guidance and ask it to make a judgment call, right? So, this makes it explicit for me. I can actually review and go through the places where it increased the threshold and say I’m okay with that. I’m not okay with that, but this automatically does the tuning for me, right? That I would usually never suppress it directly in the code because it’s just too tedious when I was doing it myself. But with the help of the agent, I can actually try and get a clean slate all the time and then it becomes a much better sensor.
Priyanka Raghavan 00:16:41 Well that’s very interesting. You talked a little bit about the lines of code, and I was thinking about the cyclomatic complexity. Which when we wrote code as humans would be something that we would have very hard targets on. And nowadays I notice when I’m reviewing code of someone who’s generated the code completely with AI, if they don’t have say these constraints, then you might see cyclometric complexity 125 or something. That’s huge, right? That’s yeah, yeah that feels like a seismic event, or something compared to say what we would have at then. So, for those kinds of cases is what you’re saying that you can actually add things in your guides to restrict the agent? That’s what you’re saying. So, you can actually have sort of a constraint that you can place on the agent for your particular repository like coding guidelines?
Birgitta Boeckeler 00:17:28 Yeah, so the sensor triggers another little loop. So, I define my constraints in the configuration of the static code analysis. For example, I say what my maximum allowed cyclomatic complexity is, right? Then the agent writes some code and then the analysis tool says over there cyclomatic complexity is too high. And then it also has this additional little prompt that tells it more about how we want it to handle cyclomatic complexity with all you are allowed to increase it if you make a judgment call it then rely on the LLMA lot again. But it just makes it more explicit where you need to review and then that starts another little loop where the agent tries to self-correct and then asks the sensor again, right? So, it’s an extra little loop that you start. And the cyclomatic complexity definitely another great example of a very typical failure mode of AI generated code.
Birgitta Boeckeler 00:18:19 So that one triggered a lot for me as well, even with the big models and not just the complexity but also sometimes just very brittle, hard to read Boolean expressions, like combinations of different that maybe as a human I would’ve turned the does not equal around to an equal or just doesn’t feel very expressive and very readable. And some people say, oh but it doesn’t matter if it’s readable because AI changes it in the future. But when you have these weird Boolean expressions, it’s a lot riskier for AI to change it as well in the future and have an unintended side effect that breaks something else. Right.
Priyanka Raghavan 00:18:57 So what do you mean by AI change it in the future? Do you mean by a refactoring attempt?
Birgitta Boeckeler 00:19:02 Yeah, so some people say that all of these things about code readability or code quality are not as important anymore because as a human I will not have to deal with the details anymore. I just deal with a little bit higher level and AI will change the details in the future. So, it doesn’t matter to me if I’m able to read this Boolean expression, but it turns out that AI needs a lot of the same things that we need to make safe changes to stuff.
Priyanka Raghavan 00:19:27 Okay, got it. So, what about even the practices unit testing and we generate, obviously use a lot of AI agents to generate the test, but one thing I’ve noticed is sometimes when I write the test as a human, then the code becomes better. Is that something that you have seen in your experience? Because I’m able to describe the behavior better when I write a unit test and therefore, I feel the code gets generated better.
Birgitta Boeckeler 00:19:55 Yeah, I mean tests are definitely an important sensor, but they have two overarching roles. I mean there’s probably more but I have two in mind, right? one is we try to use them to check the functional correctness of what we’re building. Is it correct? And there if we just have AI generate all the tests and the code, then other than doing manual tests and observing what the application does, it’s not a 100% guarantee that it actually does what we wanted to do, right? Because we didn’t write the test. Test suites also have another function as a regression sensor, right? Just telling us how good they are at telling us that something broke, right? That’s more a maintainability concern. But that’s also an interesting part of a sensor that we can also talk about next if you want to. But yeah, this whole thing about the correctness of tests, I mean the test might all be green but just test things that we don’t want, right?
Birgitta Boeckeler 00:20:46 And in the past, you said, we often, that’s why it’s called test driven development, right? It wasn’t just about making sure we have tests, but it was about writing the tests first. Because the tests are the ultimate specification, right? They made us think through what we actually want. And so, this for me is still a largely unsolved problem let’s say, because you might argue that maybe we could say as a human, my main role is to review the tests and okay, let’s say AI has gotten good at the code, I have some sensors in place, maybe I’m exaggerating because that’s not enough. But let’s say, okay, I don’t want to look at the code anymore, but I want to review the tests. But tests are very tedious to review when you haven’t written them yourself, right? They’re very verbose and you have to, it’s a lot of detail and you actually want to think about the big picture like have I thought about everything, all the combinations and stuff like that, right?
Birgitta Boeckeler 00:21:35 So I recently talked a lot to my colleague Mateo Vacarti about this who has been on some teams where he’s trying to look for the good acceptance test level and then he’s trying to find a way to write those acceptance tests and their input and outputs in particular in a way that is really easy to review so that he can really think about the input-output data that is very relatively easy to do. For example, when you have HTTP, when you have APIs, right? There’s always a request and a response, and you want to have different scenarios of requests and response combinations that you test in other parts of the code base, it’s a lot messier. There’s different combinations and unit tests, it’s much different. And then when you write these acceptance tests, they’re often not as detailed as the unit tests. And so, do you just leave the unit test to AI?
Birgitta Boeckeler 00:22:22 I mean, ultimately it’s a bit of a risk assessment, a risk decision. In your situation, how good do you think AI will be at writing these unit tests? So, I think it’s a very unsolved problem in a way, but a very, very important one, right? It’s all about thinking about our feedback loops for the correctness of what we’re doing. And they can be very different depending on what type of application you build, right? Is it an API, is there a UI? Are you writing to a database? Are you writing about events that need to land in some other system? All those situations need very different types of testing. So, it’s also hard to find just one simple way to do this.
Priyanka Raghavan 00:23:00 Okay, sounds good. I think let’s move on to some more practical implementation tips and maybe we’ll cover some of the questions I had. The first thing I want to ask you in this practical implementation segment is what does a minimum viable harness look for a new team? Can you give some tips and tools to add?
Birgitta Boeckeler 00:23:17 It’s actually hard to answer because this is still so much evolving. So also, in the experiments that I’ve been doing, I’m just poking around and trying to use different sensors and to even see which of these are valuable, which are not, right? And that’s definitely one of the big questions here, the tech questions as well. When you build this harness, when you have these guides and sensors in place, how do you know that they’re effective? It’s like on the guides side because the guides are often inferential, so they’re often mark down files that get interpreted by a large language model. So, on the guide side, there are now some ways to do skills evals, right? So, to have your skills and then write some scenarios and run them one time with the skill, one time without the skill and see if the skill actually makes the results better or if the model would’ve been good at it by itself, right?
Birgitta Boeckeler 00:24:04 That’s the whole thing I said earlier about the models getting more powerful and when you use a very powerful model, you might not even need some of those instructions, right? Especially when you think about that many skills these days are actually generated by large language models. So, people have large language models generate the skills and then I’m sometimes, yeah, but then if the model already knows all of this, sometimes it’s still useful. I’m not saying it doesn’t make sense, right? But it sometimes makes me wonder if there is some kind of cycle that I should consider if it actually needs it or not, right? Unfortunately, models are changing all the time, their capabilities as well. So evals, that’s one way to check if your skills are actually being effective on the sensor side, especially the computational sensors the code analysis and all of that that I was talking about, there’s not really tooling for that yet, right?
Birgitta Boeckeler 00:24:51 So I’ve been experimenting a bit in a speculative kind of way what type of tooling I would want for this. I’ll write a bit about that publicly soon. But there I was also thinking about, okay, during my coding agent session I was giving it access to all of these sensors in one big suite and one big let’s say sidecar that was running next to my agent. And so, I had it write a history of every time the agent checked for the state of the sensors, I had it write a history and then I could see across the coding session, how did the test coverage evolve? How did the number of analyses violations evolve and different security static code analysis and stuff like that as well. And I would sometimes see that in the middle of the session some issues would come up and then they would go away again.
Birgitta Boeckeler 00:25:34 So that’s maybe the first step to have some observability of what’s going on with the sensors. And then it’s a similar dilemma to our pipelines. It’s very similar to our integration pipelines, right? If my pipeline is always green, is that a good thing? It’s actually not, right? You want it to go red sometimes because that gives you the good feeling that oh yeah, it’s actually a safety net that works, right? If it’s always green, I would get suspicious, but if it’s red all the time, that’s also bad, right? So, it’s kind of this dilemma of what level of the sensors firing and actually giving feedback is good or bad. But that would be something that would be great in the future if we had more tools that would introduce this type of observability. There’s a lot of that happening in the base harnesses space, right?
Birgitta Boeckeler 00:26:17 So around what people are calling agent traces. So, there’s a company founded by the ex-GitHub CEO, called Entire and they just as their first little thing they released a CLI that helps you track all of the stuff like that is going on in a coding agent session so that it can be analyzed later and there’s some other tooling popping up at the moment around that. So that’s another way to look at how well is my harness working once we figure out how to analyze all of these traces of what’s going on in the session and can then maybe compare when I have these guides and sensors in place, I actually got to my goal a lot faster than when I did not. Or can we, I don’t know how to practically measure this, but can we have metrics around how long the agent goes until I had to intervene? Or how much feedback would I give before it could actually create a commit or, so I think that’s also an emerging space of monitoring and analyzing how well our coding agent sessions are going. Which would tell us about the harness effectiveness, both of the base harness that we’re using. So again, Cloud code, Pi, Cursor and of the harness that we put around it.
Priyanka Raghavan 00:27:21 Yeah, because sometimes I also notice that the, when you ask it to change a particular file based on constraints, as you say, this changes all over the place and sometimes it feels you’re losing control of the file and every time I’m always doing the diff pretty slowly on the agent almost stopping it at every time and checking the change. So, I think an observability log on the changes would also be good sometimes because sometimes it feels it’s all over the place and it also completely rewrites things. Because I’ve had a case, it was last Friday where I was rewriting a bunch of unit tests, the unit test was not functioning okay. And then I was trying to go and investigate it and when I investigated, I found the exact part where it was failing and I asked just to give instructions to redo that part with the correct logic and then the entire stuff got over it, you know? Other previous functions because I had forgotten to say that, you know, please do not change anything except this even though I had it in my guide. I don’t. So that can be sometimes where you feel you are losing control, the basics are just very fast.
Birgitta Boeckeler 00:28:28 And that also goes beyond single file, right? We talked about static code analysis before a lot and the type of analysis that I gave examples of was mostly related to the file level, right? Or a function level or something like that. Cyclomatic complexity and stuff like that. But we also usually as developers think a lot about one level higher about cross file, cross module, what is our coupling and stuff like that, right? Because we know that when that gets messy, changes will get harder and harder and harder and riskier and riskier in the future. So, for example, often when I use a coding agent, let’s say build up a POC, that I actually want to be maintainable for a little while at least, right? I start noticing that after a while my change sets gets bigger and bigger because when I change something it has to change suddenly.
Birgitta Boeckeler 00:29:16 I did a change recently where I just added a new query parameter to one of the endpoints and it changed 15 files, I think. So, this is a relatively small application. So that was actually quite a lot. So that’s a smell usually for, oh, there’s some repeatability here, there’s something that’s not well abstracted, right? So, and that’s then about sensors, thinking about the sensors that help us look at that coupling and modularity level, right? So, I tried that a little bit with, again, more computational static analysis, you know, finding the, what are the files that get imported a lot by other files or stuff like that. And it’s actually not that helpful, it finds some of the hot spots, but in this area with modularity, I’ve had much better results with LLM sensors. LLM judges, right? That actually have really good prompting about what coupling and balanced coupling means.
Birgitta Boeckeler 00:30:10 Right? Let me just quickly look up, I don’t, I don’t want to botch the name, but I use this skill by Vlad Kononov who wrote a book about balanced coupling, right? And I’ve had really, really good results, which shows when you have an expert, right? Really good description about also like trade-offs and semantic analysis of what that means. You can get some good results from that. It’s not just doing lots of stuff in one file. Inside of a file, when I feel confident in my test, I sometimes don’t mind as much, although it is a little suspicious and then I might go and look at that in particular. But across files it’s even more problematic potentially.
Priyanka Raghavan 00:30:48 So can you walk us through a real-world example of a harness in action?
Birgitta Boeckeler 00:30:51 So, I actually published a little YouTube video recently on an example, like my example application that I’m working on where I’m trying to put a harness like that in there. And in that one, because I focus a lot on sensors, I don’t have a lot of guides in there. But I would say typically what I see lots of teams do these days is on the one hand have this informational part of the guides, right? So almost a knowledge base inside the code base that gives the general context of the basics of what you’re working on so that the agent becomes aware of that really quickly and doesn’t have to reconstruct that every time you start a session by looking at the code base. And then the other part is definitely some coding conventions or pitfalls that are maybe in Cloud.md or Agents.md file.
Birgitta Boeckeler 00:31:36 And then maybe a few skills that are about coding conventions in places, right? On top of that, there’s of course that whole space of spectrum specification driven development, right? Where people then also use workflows and stuff like that which in a way is also extra guides, right? But I know that lots of people are building their own workflows in form of guides. First do the planning, then do the design, then do the, you know, all of those type of things. So that’s also quite typical, but also a space that is very emerging and lots of different stuff happening and there’s some backlash to it already, again about how it’s too much, right? With spectrum development. So that’s on the guide side, right? On the sensor side, I haven’t seen that as much yet, but in my application that I’ve been using, I’ve set up a set of sensors that happen during the coding agent session.
Birgitta Boeckeler 00:32:25 So for example, the static code analysis we talked about, it’s looking at the test coverage, I have mutation testing set up as well, although that’s a bit resource intense, right? But mutation testing with the coverage actually tells me how good my regression level is, right? So, there’s a bunch of things happening in the session before I even do a commit. I should say I come very much from the tradition of trunk-based development, and you know, pushing to main as much as possible. So, I’m very controlling about a commit, and I always wanted to be perfect. So, I know that some people are using much smaller commits and then think more about the PR. I just wanted to give that as a caveat, but this exactly goes to one of the things we have to do when we think about the guides and sensors is where to place them.
Birgitta Boeckeler 00:33:04 Where do we want them to run? Because they have different costs in terms of how long they take to run, how much energy they used to run, how many tokens they used to run. So those are all things to consider about where we place them during the session before we even integrate during a PR review or in the CICD pipeline. And then there’s also one place in our path to production that is kind of a repeated running of sensors. There’s this article by an open AI team about harness engineering where they call this garbage collection. So, they say they worked on a code base for five, six months just having the agent touch all the code and them just working on that harness and they still saw technical debt compound, you know, probably also things I experienced about the modularity and stuff like that.
Birgitta Boeckeler 00:33:54 So they also have things running on a schedule, right? I don’t know, they didn’t share details about how often they do that. But in my application, I also have some skills that are, one is this modularity review, one is a security review that is based on our internal InfoSec guidelines at Thoughtworks. And then there’s one that checks kind of the freshness of the dependencies and gives me recommendations of where I should replace libraries or something like that. And I just go in there and I currently manually just run those once a week and see if there’s anything new. You can of course then expand that even more, have the agent create tickets for the findings that it has or even created pull requests for the findings that it has, right? So, it all depends on your team processes and how you’re maintaining the application. But you can already start by just having it create a report once a week and having a human look at it. It has definitely reduced my maintenance work on this application, which is a real-life application with a very small user base. But you know, it needs some maintenance and it has definitely made it possible for me to do that with half an hour a week or something like that.
Priyanka Raghavan 00:34:58 Okay. That’s interesting to know. And are there any frameworks of patterns building harnesses today?
Birgitta Boeckeler 00:35:03 No. On the one hand I think it’s more like conceptual thing. So, there’s probably not, here’s the one technical thing that is the harness and at least not in this expanded harness. You know, of course for the base harness there’s the one technical thing that is the coding agent harness. But yeah, I’ve also been thinking about that. For example, you have guides and sensors that have to be consistent with each other, right? You might have a guide that describes, this is a very simple example that’s probably not real in reality, but let’s say a guide said you should have very low cyclomatic complexity and then there’s a sensor that checks the cyclometric complexity, right? I mean this is a very simple example, but just to say that the guides and sensors, they have to be in sync with each other, and you might have them all splattered across the code base and not. So, I’ve also been wondering if there is a way to maybe package them together somehow or is there a way that the base harness can take some of the responsibility of running the sensors with some plugins or something like that, right? I think there’s definitely potential for more tooling in some of these things and more configurability, but overall, I think it’ll always stay more a conceptual thing like architecture or what are the things that you have in place to increase your coding agent’s quality of results.
Priyanka Raghavan 00:36:18 So this comes before you actually do the, like you run your GitHub actions or anything on the main branch, right? So, when as soon as you get a result from the coding agent is when you run these sensors as well?
Birgitta Boeckeler 00:36:29 Like I said, you have to decide where you put which sensor, right? And I definitely think there are definitely the kind of low hanging fruit, quick hygiene type of sensors that also run really fast. Why not run them before you even create a commit? I see a lot of the base harnesses always assume there’s already a PR or you’re on a branch or something, right? But I want to have a lot more stuff like that happens on my local change diff, right? Because if you can do it after the commit, you can also do it before it, right? So again, maybe this comes from my history of trunk-based development, but we don’t want to start pushing quality, right? We want to keep it as far left as possible whenever we can, right? So especially the cheap stuff, let’s just try to run it, you know, as soon as possible and not two days later when we have a PR review.
Priyanka Raghavan 00:37:17 And here is what I want to ask you a question also in terms of when you run these tools for doing the, a static code analysis or security like say send rep or tools like that, what happens is sometimes you’ll get a finding where it’ll tell you to go and fix it, but then the thing is that sometimes it might not really make sense to fix it in your particular project either because of it could break something else or it’s really not that important and there’s almost a judgment that comes from a human to tell you to fix it, right? How does that work with the AI agents?
Birgitta Boeckeler 00:37:49 So this goes back to what I was talking about before where I was asking it to make a judgment call in my self-correction guidance, so to say, right? And so that again is, you know, remains to be seen, okay, how much we trust the models to make this judgment call, right? We might trust it more with max lines per function than with security thing, right? So, I actually also do have send rep set up for my application and I think in that guidance, I don’t tell it to make a judgment call. I tell it to let me know, I don’t quite remember. But in any case, it’s a lot easier to do this suppression, you know, usually these linting tools, you can put a comment in the code and then in the comment you can also say why you are suppressing it, right?
Birgitta Boeckeler 00:38:33 So that also gives information for the future when you’re doing this. So, harnesses for me are all about improving how as a human I can prioritize where to put my review. So always thinking about how I can triage that almost, right? And depending on the risk that I see in my use case in this area of the application and so on. But the thing that I find even more interesting than that, and I don’t know yet where that is going to go is what if you have sensors that contradict each other, right? Trade-offs are all over software delivery, right? So, I have been in my application watching out for these, are there any contradictions, trade-offs, what does that mean? The only small thing I’ve found maybe so far is that because my agent has been doing a lot of splitting things into smaller pieces because of, you know, it kept violating max-lines per function, max-lines per kind of things.
Birgitta Boeckeler 00:39:26 So it puts some reasonable exceptions in there, but sometimes it just breaks my React components down even smaller and smaller, right? And then suddenly I have a hierarchy of, I don’t know how many levels of React components that contain each other and that they all suddenly pass through eight or 10 properties all the way down to the bottom. And most components don’t even need that, right? So that’s maybe the only example I’ve seen of a trade-off so far where the, you know, it’s very eager to break everything down into smaller pieces because that’s what the sensor said, but what if I now switched on another sensor that tells it more about the react component health and not making it too granular. So that will be interesting to see when you switch on lots and lots of sensors, will it just go into overdrive and keep ping ponging between different judgment calls?
Priyanka Raghavan 00:40:13 That’s very interesting. So then how do you kind of build a trust in systems where agents are generating and also modifying so much of the code?
Birgitta Boeckeler 00:40:22 Yeah, so that’s kind of the goal of this, right? Thinking about increasing the trust in the agent. And this is of course not the only thing, right? We were saying before, the test suite is green, that might not actually mean anything, right? But it is a lot about probabilities and risk assessment, right? So, I always think of it as probability impact and detectability that’s always the formula of how you do risk assessment. So, I think about how probable do I think it is that AI will do something wrong or will do it right, however I want to think about it. And so that probability assessment is based on my experience with using the coding agent, my experience with using the model, or how much do I know about the tool. So, it’s very you can’t just look it up at a table.
Birgitta Boeckeler 00:41:08 It’s very much the skill that we have to own because it’s such a weird technology, but probability that it does something wrong might also be, oh, this is a very, very messy code base. So, it’s a lot higher probability that it will do something wrong than when it’s a very, very well factored code base. And then for impact, it’s all about the use case. Am I building a POC or a spike or is this a super critical business flow that I’m working on? And then detectability is all about how easy will it be for me to see that it did something wrong. So, what are my feedback loops? What do I have in place? And then when I look at all of those things, then I decide how much do I let it go unsupervised? How much review do I want to do? Do I want to look at every single line of code or just a high-level thing, just check the sensors and I’m done. And yeah, to go back to the probability, right? Probability then is also a lot about how much confidence I have in my guides and sensors. How much visibility do I have into the things that I care about for this particular application and for this change that I’m doing.
Priyanka Raghavan 00:42:05 So in a sense that trust gets built because of your trust and also your harness in a sense, right?
Birgitta Boeckeler 00:42:11 Yeah, yeah, yeah. We just now have to find out how far can we push these harnesses? And again, it might depend on the use case, right? In some use cases I’m fine with just a little linting and maybe there’s some modularity review once a week. And in other use cases it might just not be enough for me. And I really want to look at everything.
Priyanka Raghavan 00:42:28 Can multiple agents coordinate within a harness?
Birgitta Boeckeler 00:42:32 So I was talking before about the coating harness itself is the base thing that does everything right. And that coating harness might actually have the ability to spin off multiple agents and then my expanded harness, I can of course, you know, all of those agents can potentially access it or some of those agents might be part of my harness, my expanded harness. So, there might be a code review agent, that is one of my sensors. So yeah, again, with the terminology, it’s a little tricky right now, but I think we’re getting there. It’s getting a little better. I would love it if there was a better word for this expanded outer harness, but nobody has come up with one yet. So, I think it would be helpful if we had a separate word for it that out part of the onion basically that we engineer. Yeah. And these separate code review agents are actually a very, very common sensor that people set up where they spin off another subagent that has its fully own context and no history of the conversation. And then with that quote unquote unbiased view, it looks at the code and reviews it. Sometimes people to use a different model even for that subagent, right? And so that’s a very commonly used sensor that people put in place.
Priyanka Raghavan 00:43:38 In your article, you talk a little bit about the role of human in the software development life cycle and also the accountability, which humans have to do things, right, which agents might not have because they don’t belong to an organization or have an organization.
Birgitta Boeckeler 00:43:54 Yeah, they definitely don’t, they’re machines, right? They don’t have accountability.
Priyanka Raghavan 00:43:58 So now we are placing so much of trust on these agents to do a lot of the work. Without the accountability. So how does this whole fit into the software development lifecycle?
Birgitta Boeckeler 00:44:09 Yeah. And also to add to that, sometimes some of the anecdotes and stories you hear, it’s also putting people in unfair position because some organizations on the one hand side put a lot of pressure on people to use coding agents and to become faster and more output and you’re, you know, you’re being measured by that. But then it’s when something goes wrong, then suddenly it’s your fault. But when you put a lot of pressure on people, oh, now that you have AI, you have to be X percent faster, you have to create 50 PRS a week or whatever, right? Then you know, if that’s what you’re being measured on, people are going to cut corners and it’s going to lead to more incidents, it’s going to lead to more instability, right? Google does this Dora report on software delivery performance and now they’re also focusing a lot on the AI assisted, they call it I think AI assisted software delivery.
Birgitta Boeckeler 00:44:55 And that is one of the factors that they have found so far in their data that instability goes up, right? So, or stability goes down, however you want to put it. And yeah, there were some prominent incidents over the past few months where in the communication around it you could start seeing some organizations having this reflex to throw people under the bus a little bit, right? And so, I’m always torn because on the one hand it’s absolutely clear to me the human has to be accountable, right? But because of the environment that we put the agents in and that we put people in, I feel for people for cutting corners when you’re only being measured by activity, by throughput and all of that. So, it’s a tough situation. I think it has to be done consistently if you still expect accountability, if you hold humans accountable for what’s happening, you also have to give them the space, the environment, and the incentives to be able to do that.
Priyanka Raghavan 00:45:47 That’s true. So, in this regard, I also wanted to ask you from your perspective, how is the role of the software engineering change because of agents? Coding agents and harness engineering. How does the whole thing change?
Birgitta Boeckeler 00:46:00 I mean, from the beginning of all of this hype starting, right? People have said, oh, it’s just we don’t type the code anymore, right? I mean, that’s what people mean these days when they say, oh, all of our code is written by AI or 90% of our code is written by AI, they mean it’s being typed by AI, right? Of course, a lot of it is also reasoning about and creating it, but the engineers sitting in front of the coding agent or deciding what task they give it, how they describe, a lot of the prompting that is being done is very technical, has a lot of technical details, right? And this is particularly easy for experienced software engineers, right? Because when we know what we need ultimately and what good looks like, we’re also a lot better at prompting and saying what we need, right?
Birgitta Boeckeler 00:46:43 So, but there’s, from the beginning when there’s talk of, oh, the obstruction level goes up, right? But I don’t think it goes up in the same way that it went up with compilers and stuff like that. It’s a typical comparison that people do. Oh, it’s at some point at first people didn’t trust assembly code and then at some point they just trusted it and never read it again, right? But I don’t think it’s the same thing because a large language model is not a compiler. A compiler is repeatable, deterministic to very, very high nines. So, and a large language model is very semantic, interpretative, probabilistic, all of those types of things. So, I think it’s a different type of abstraction level raising that will happen here. I also think that the details will matter less, especially when we find good ways to control the details even more with tools like linting and stuff like that.
Birgitta Boeckeler 00:47:34 But then, you know, I said, this modularity review, that’s already where I came to the point where just having the data and the computational sensors was not enough anymore. You needed more semantic interpretation, which, and LLM helps us with a little bit, but then we also maybe want to look at and still understand what’s going on in our code base. So, it’s easier for us again in the future to tell the LLM what we need. So yeah, I think there’s definitely some kind of raising of the abstraction level, but we don’t know yet where that’s going to sit. Might also depend on the domain. And then the other thing is this thinking and risk assessment that I talked about before. So, I recently thought about the best quality analyst I ever worked with who was actually a career changer from recruiting. So not a computer science graduate.
Birgitta Boeckeler 00:48:18 She could sit with US developers and she got into coding a little bit and she could sit with us, she could read the code and she was the person on the team who understood the whole system inside out better than anybody else on the team. And she would know when we would do story kickoffs, she would say things, oh, you’re changing something in the database. And then she would know exactly what to poke at, what to test, what the risk profile of that was, what that meant. And so, this type of thinking in, oh, we’re changing this, so that means that we have to test and quality assure these things and maybe do we have sensors for this? Whatever, right? So, this type of risk assessment or also how much do I review based on probability impact detectability. That’s something that will be a very important skill and that often developers haven’t honed that much. The fact that we need to have quality analysts on teams tells us that. We’ve seen so many teams go, oh, we need a quality analyst. You know, it’s six developers sitting there and they all say they need a quality analyst. So, it tells us that it’s not a skill that so far, we’ve always widely developed. So that’s another shift I think we’ll have to make. Like you said before, how do I know when I can trust it? It depends.
Priyanka Raghavan 00:49:30 So I think that’s a good skill. Looking at quality engineering, I mean, which we used to have before, but then we’ve all fused into the T-shaped engineer. What I think, what looks, this has to come up a bit in the T shape.
Birgitta Boeckeler 00:49:42 So I highlight that again. And quality assurance in is very, very much about risk assessment. Where are the critical paths? What actually matters? The testing pyramid, how will that change? Will we indeed have more acceptance tests? And then how do we feel about the risk of that, right? So, it’s very much this risk assessment and thinking and probabilities and being comfortable with that.
Priyanka Raghavan 00:50:02 I think the quality angle is going to become all the more important when writing code is not going to be so tough. So yeah.
Birgitta Boeckeler 00:50:08 And then now we’re also like to determine quality, we actually have to describe what good looks for our situation. And that’s also not something that we’re historically always good at. Oh, it just feels more clean, you know?
Priyanka Raghavan 00:50:23 Yeah. Metrics for that as well. Yeah. I’d like to end the show by asking you a little about what do you think about the future of harness engineering? Adding more things to make the AI agents more predictable. Right? Can you talk a little bit about that?
Birgitta Boeckeler 00:50:38 Yeah, I mean, it’s just like maybe recency bias, but I’ve been experimenting a lot with these sensors because I think they’re still quite underused and there’s also not a lot of tooling in that area. I just don’t see the future as being like 50 markdown files in our code base. I mean, that can’t be it, right? And then I said before, in every markdown file we have very important. Do the following, never do the following. I mean, that can be it. Can we still call ourselves engineers if that’s how we’re doing stuff? I don’t know. I mean, we’ll be a part of that definitely. But yeah, so I’m just like really interested in this whole ecosystem and all these integrations into the base harness and how we can make that better and also balance it, right? It also cannot be just throwing a hundred tools at the agent and 50 sensors that kind of overload it.
Birgitta Boeckeler 00:51:21 Or we can think about, okay, if I have these sensors, do I need those guides anymore? How do we balance that? Yeah. And unfortunately, then we’ll probably never have a really, really good way to measure what’s better and what’s worse. So, it’ll be about the feeling again, right? But in short, I see a lot of tooling potential there still, and a lot of things we still have to figure out about how to balance this. This whole idea of regulation, I really like it. Like a system that you try to regulate that it doesn’t get too much or too little or you know. So, I really like that image to think about harnesses.
Priyanka Raghavan 00:51:58 And it’s also a good way to test the models which are coming out in the market as well. Right? If you have good harness,
Birgitta Boeckeler 00:52:03 If we know how to determine that it is good, then it’s
Priyanka Raghavan 00:52:05 Okay. Okay. Okay. Yeah, you’re right. I mean in the sense that you,
Birgitta Boeckeler 00:52:09 It’s all moving pieces, right? Right. It’s all of these moving pieces. The models change, the base harnesses change. I mean, they all churning out releases crazy, right? And then our own harness changes, right? So, there are so many moving pieces on the board. I mean, the ultimate measures we have are our software delivery performance, right? What is stability? What is throughput? And then the ultimate, ultimate measure is user value, right? Is this actually what we’re building? Are we just building more crap faster? Or is it actually bringing the user’s value? Is it bringing us revenue? Is it bringing profit? Right?
Priyanka Raghavan 00:52:41 I think building something of value is what is most important, and I think using all the tools we can to get to that state is what we should aim for. Thank you. This has been a very great conversation. Where can people find you in cyberspace Birgitta? LinkedIn or email?
Birgitta Boeckeler 00:52:58 Yeah. Most of my writing I do on my colleague Martin Fowler’s website, there’s a series there exploring GenAI and also some articles. I have a personal website, Birgitta.info, where I just always list all the podcasts, the YouTube conference talks, writing, just a list of content.
Priyanka Raghavan 00:53:17 Okay, great. I’ll add that to our show notes. Thank you for coming on the show.
Birgitta Boeckeler 00:53:21 Yeah, thanks for the conversation, Priyanka.
Priyanka Raghavan 00:53:23 This is Priyanka Raghavan for Software Engineering Radio. Thanks for listening.
[End of Audio][/tt]
