Hans Dockter, the creator of the Gradle build tool and founder of Gradle Inc, the company behind the developer productivity platform Develocity, joins Giovanni Asproni to talk about developer productivity. They start with some definitions and an explanation of the importance of developer productivity, its relationship with cognitive load, and the big impact that development tools have on it. Hans describes how to implement developer productivity metrics in an organization, as well as warns about some pitfalls. The episode closes with some discussion on Hans’s views on the future of this discipline, as well as some near-term developments and expectations. Brought to you by IEEE Computer Society and IEEE Software magazine.
Show Notes
Related Episodes
- SE Radio 554: Adam Tornhill on Behavioral Code Analysis
- SE Radio 524: Abi Noda on Developer Experience
- SE Radio 327: Glynn Bird on Developer Productivity with Open Source
- SE Radio 317: Travis Kimmel on Measuring Software Engineering Productivity
Related Articles and Resources
- DORA website
- Microsoft Developer Experience Lab publications
- The Programmer’s Brain book
- Measuring Developer Productivity via Humans
- Are Most Of Your Product’s Features…Useless?
- Team Topologies
- The Next Generation of Developer Productivity
- Book: Order Out of Chaos: Man’s New Dialogue with Nature by Ilya Prigogine and Isabelle Stengers
- The SPACE of Developer Productivity
Transcript
Transcript brought to you by IEEE Software magazine and IEEE Computer Society. This transcript was automatically generated. To suggest improvements in the text, please contact [email protected] and include the episode number.
Giovanni Asproni 00:00:18 Welcome to Software Engineering Radio. I’m your host, Giovanni Asproni, and today I’ll be discussing developer productivity with Hans Dockter. Hans is a software engineer and the creator of the Gradle build tool is the founder of Gradle link, the company maintaining and developing Gradle, and which is behind also Develocity, a developer productivity platform. Hans, welcome to the show. Is there anything missed that you’d like to add?
Hans Dockter 00:00:40 No, I think it was a great introduction. Thanks for having me, Giovanni, and looking forward to our conversation today. I’m very excited about it.
Giovanni Asproni 00:00:47 Well, me too. The pleasure is mine. I’m very excited as well. I find this a very interesting subject. So let’s start. What is developer productivity?
Hans Dockter 00:00:55 You start with a hard one. Yeah, so for me, developer productivity is a theoretical potential for how productive a developer can be, ? And then there is the actual productivity of a developer, ? And for me, the developer productivity is about how close is the actual productivity to the maximum productivity your development team could provide to you.
Giovanni Asproni 00:01:20 So in the intuitive times, what we mean with productivity is kind of number of feature develop per unit of time is something else.
Hans Dockter 00:01:27 Yes. I mean, you know, the whole space framework, ? Or I think it is multi-dimensional, ? So for me, every metric that we can quantify is only a reflection or a hypothesis, ? How this could affect the actual productivity, ? I think we cannot reduce productivity to a metric. What we can do though, there are a lot of metrics that indicate that your actual productivity is far from your maximum productivity. So when you know the feedback cycle times are two hours, when you know every second build fails with flaky test, ? And you have a lot of unnecessary interruptions, there is a huge impediment, ? That reduces your actual productivity. And that is what we are focusing on. What are the roadblocks? What is by definition, affecting negatively the productivity? Let’s say if a development team works in a perfect environment, then kind of from perspective, then the machinery has done its job, ? That’s what we are focusing on. But then to measure what is really productivity, I don’t think at the moment we have the metrics only, let’s say in thought experiments. Yes. If someone doesn’t commit any code, it’s probably over a year. You know, there’s probably not much productivity, but otherwise, all those metrics that you talked about could be gained.
Giovanni Asproni 00:02:45 When we talk about productivity, developer productivity, now there is this whole field, maybe, I don’t think it’s very new, but certainly now since to be gaining more traction, the developer productivity engineering. And what is this about? It’s about finding matters to improve this developer productivity.
Hans Dockter 00:03:02 Yes. So the way I look at it, in theory, it’s not new for all practical purposes, it’s new. And why I say it’s new, I think when you look 10 or 15 years back, ? The people who worked on productivity, let’s say the people who worked on CI or the people who worked on build systems, or on the builds, were considered not good enough to work on a production code. So that’s why I would challenge it a little bit. It’s not new. I think as a discipline in software, it was a completely esoteric and niche thing hardly any organization focused on and appreciated, ? So that has fortunately changed. And DPE, ? So there are many dimensions that affect developer productivity. Could be the food in the cafeteria or you know, many, many things. But that is not what DPE is focusing on, on all the dimensions, ? DPE focuses on what I think is the biggest impediment to develop our productivity. And that is the tool chain, definitely in an enterprise context. There’s a lot of other important vectors that affected, but that is at the heart of the, I would call it the developer productivity crisis that we are in, ? That we slowly awakening to that is at the heart of it. And DPE is a practice that tries to professionalize how we look at the developer tool chain and how we improve it.
Giovanni Asproni 00:04:25 Okay. And now why is developer productivity so important?
Hans Dockter 00:04:28 Because the actual productivity is far, far, far from where it could be. So there is a huge waste. If that weren’t the case, then we wouldn’t need to have that conversation. But the reality is that there is a transformational opportunity for organizations to get order of magnitude improvements when it comes to the output, or at least many factors of improvement of their code output if they invest into this area. And some organizations are showing that this is possible, ? That’s, that’s why it’s so important. That’s why I’m intrigued working in this area because it’s transformational in my opinion. Yeah.
Giovanni Asproni 00:05:04 Okay. Yeah, I agree that that is a lot of waste in software development in many projects. I mean, I’ve seen many of those cases where we waste time and effort. And there is a lot that can be done to improve the situation. But now I’ve got also another question because when I think productivity, I’m thinking also about developer experience. They will, you know, you mentioned the tool chain. So what is the relationship between developer productivity and developer experience?
Hans Dockter 00:05:29 Yes. I think it’s at the heart of the developer experience crisis, if you like, is the developer productivity. I mentioned this wonderful book by Ilya Briga Gene, one of the great physicists of the last century. He calls it Men’s New Dialogue with Nature. The way he looked at this, ? As a scientist, you formulate a hypothesis, ? And then you enter a dialogue with nature by our experiments to get feedback on that hypothesis as a developer, ? We write code, which is a hypothesis, and then we need to enter a dialogue with the tool chain. Hey, compiler, hey, unit test, Hey, functional test, Hey, regression test, ? What do you think about this hypothesis? Give me feedback. Is it’s doing what it’s supposed to do? Is it breaking something? Is it fast enough? All of that, ? And that dialogue is at the heart of what we are doing.
Hans Dockter 00:06:18 And now we come to the experience question. When I started coding C64, back in the days, it was like, my print hello world, ? It was fast, there was no waiting, ? So I could formulate my hypothesis, I would get instant feedback. And I don’t know if I would’ve gotten into programming if I had the experience of an enterprise developer. Nowadays, oh, it takes 20 minutes to get any kind of feedback. 20 half of the time the feedback is not correct. That would’ve destroyed all the fun where I got hooked up in software development because it’s the flow, ? Hypothesis feedback, hypothesis feedback, let’s try this, let’s try this. I have this question. And that’s so much fun when you can move quickly with that. And when you are now bombarded with a slow, unreliable tool chain, it’s the one of the most frustrating things in the world.
Hans Dockter 00:07:08 And it takes so much fun out of the job. Furthermore you have pressure to deliver. And the space frame of people write a lot about that. They say, when the impediments are so strong, then the only way you can deliver what is expected of you is by doing heroics, basically burning yourself out. Doing for extra hours to even get their kind of, to be expected stuff done. So, and those two things, the flow, the instantaneous experience between hypothesis and feedback, plus the pressure to deliver and then having a blunt knife. That I think both, there’s nothing that affects the experience more than that in a negative way in the life of a software development in most environments.
Giovanni Asproni 00:07:51 Okay. Now let’s talk about the key factors affecting productivity. You mentioned already the tool chain, maybe you mentioned in passing slow builds or long feedback loops, basically the tool chain. But are there other factors that affect productivity for developers? Because I was thinking about, things that sometimes you mentioned in some of your talks like cognitive load or maybe other things like, I don’t know, we’ll talk maybe a bit more later about this. But for example, things that have not much to do with developers, with the developers as in with the writing the code, creating the system and be, you know, unclear requirements, things like this.
Hans Dockter 00:08:28 Yes. So I would say sure, the code base is in a really bad state, ? And now you have to work on this, ? Or toxic work environment. I mean, which is not probably a complete exception. So yeah, a hundred percent there are many factors. So let’s go into the cognitive direction. The brain is the, I would say at a leadership level is not a very well understood production means for code. And there are a lot of naive assumptions that I think create negative outcome for the organizations. What do I mean with that? So when you look at what neuroscience tells us, there are two types of cognitive activity. One is where you have pre-established neural pathways completely learned cognitive activities. So for example, professional chess players when they do the opening of a chess match.
Hans Dockter 00:09:20 They don’t have to think much. You can wake them up at 3:00 AM in the morning and they will do an opening move. Usually and because this is learned, they don’t have to think about it. And other thing is like when we are crossing the street and we are looking in the direction, unless we are in England, and I don’t have to think about it. Oh, where do I need to look? Those are could be very complex cognitive activities, but they’re completely learned. And then we have activities that require cognitive control. And those are the really interesting one. Cognitive control activities are where we know the goal, where we look at the inputs, but we don’t know yet how we get there. Like in the middle game of a chess match, there are exponential opportunities. There is nothing pre learned anymore.
Hans Dockter 00:10:03 So you need to look at the inputs and what I’m trying to achieve. And then you need to find a way to get to this goal. And there is a lot of research around that in neuroscience. So what you have to do is you have to dynamically create a model in your prefrontal cortex where you model the inputs, where you model the goals, and then you figure out how to get there. And all the valuable coding activity falls in that area. You need cognitive control. There’s not a pre learned thing. You have a requirement and now you want to have a certain outcome with software. Now you need to think about how to achieve that. So, and activities that require mental activities that require cognitive control, they lead to cognitive fatigue,right? So, and now what does it mean in the context of developer productivity?
Hans Dockter 00:10:49 Ideally, I have a hundred percent conversion of cognitive fatigue in code. I start in the morning, I have the perfect environment and all my cognitive fatigue gets basically converted into great code for the organization. That would be ideal. And then after eight hours or whatever, I go home, I’m fatigued, but you know, I got paid for that fatigue and the company has hopefully, you know, a lot of good codes. That creates business value. And now the way I look at this, a lot of non-productive stuff that is happening that is accelerating cognitive fatigue without getting any business value out of that.
Giovanni Asproni 00:11:29 So if I’m understanding your point correctly, of course there is this cognitive load and cognitive fatigue because I mean what we do is transforming our thinking into, to something that transform a machine, I guess. And you say this kind of the tool chain we use can be a problem. You mentioned flaky test, but even long feedback loops. So say the build that is too slow or other factors that something not working well will actually affect the feedback loop and this will actually cause some cognitive fatigue because we are there waiting for something to happen. Potentially doing something else. Okay. Now what do you think about the cognitive fatigue also due to the fact that sometimes, maybe we are stretching ourself a bit too much. So for example, a lack of the necessary skills or the fact that somebody is using for the first time some tools.
Hans Dockter 00:12:16 So a good example is, so many organizations they’re now, I know Microsoft is a proponent of that. They’re looking, when we hire a new developer, when do we see the first commit, the first tool request? And they consider this as a primary measure for how good their onboarding is. And that goes to what you are saying. How much cognitive energy do I have to burn to start getting productive when I start working on a new project?
Giovanni Asproni 00:12:44 So Microsoft does that. So to decide if how to improve their onboarding process.
Hans Dockter 00:12:49 Yes. And they have some very interesting data. Brian Houck, he’s presenting on that regularly. He’s one of the co-authors of the space paper. He’s working for Microsoft Research. They say the interesting thing they found is when the first poll request is earlier, then all the other poll requests that come are also earlier. So this is for them now a strong measure that, that they will get, that this is a sustainable productivity gain is indicated by an early pool request.
Giovanni Asproni 00:13:15 Oh, okay. And I think also you mentioned before about also cold quality. So I was wondering also if the cold quality as you know, acts as a potential productivity blocker because you know, especially developers that work maybe on legacy systems or systems that have been created without any test documentation. But in that case it’s not about the tool chain that much, it’s more about what the material you’re working with.
Hans Dockter 00:13:40 I’m biased. Yes and no. So what we often see is, if you have no test, there’s a problem. But what we often see with legacy project is that the tool chain experience is terrible. But then the organization saying, oh, this is not the future. We do not invest into the tool chain. And so let’s say you have a very hairy, very coupled tool chain. So you have to do complex refactorings, let’s say, to extract some microservices out of that. And now when the bill takes a long time, when the test quality is poor, either the reliability of the test or the test coverage. Then it’s so much harder to work with a legacy code base. Even with the best tool chain for badly designed legacy code base that affects your productivity. But often it goes together a bad code base with a really bad tool chain experience. And that makes it so hard then to improve it. So I’m biased, I always have a tool chain perspective. That’s just kind of, that’s why I’ve seen it so often as the primary bottleneck. So I agree with you, there are other things here, but even here, the toolchain, when we look at this in reality often is a main contributor why it’s so hard to work with this legacy code base.
Giovanni Asproni 00:14:47 Have you got any examples you can give us? Of course. Without naming names.
Hans Dockter 00:14:51 Yeah. So for me, and that is probably most larger companies in the world. It starts with the reliability of the feedback, which I think emotionally is even worse than the waiting time. So you have a pull request built and it fails. So, and ask any company. And I like to do this as a survey when I present somewhere, how often in the last four weeks you had a failed pull request built, you looked at it and you said, I have no idea why this has failed, let’s run it again and if it passes, good, then I move on. There is so much. So we see many companies where five or 10% even of their CI builds are failing because of flakiness in the pipeline. And all of that creates a disruption. Oh, I have to stop doing what I’m doing. I need now to look why this is failing. But it’s not your code that it’s failing, but you don’t know, you just see it’s failed. Is it my code or is it something in the CI environment that is not reliable? And this is just
Giovanni Asproni 00:15:53 Actually, have you got any information about how often is the in the code and how often is the CI environment? I mean are there any statistics out there about this?
Hans Dockter 00:16:02 Yes. In this case. So we measure some of that and that’s why I’m saying five to 10% of the failures are CI environment related. And that’s a big number. We’re talking about, millions of bills. But the problem, but that is not even, let’s say it’s 50-50, the problem is? The developer doesn’t know that the worst thing, it doesn’t know is it the code or is it the environment? And then what do you do then you say, let’s file a ticket for the infrastructure team. The infrastructure team says it’s a failing test. What have we to do? You look at this. No, you have to look at it. It’s not me. This is a daily dance in 99% of the enterprises of the world. I know organization that have just that, have a whole team on the infrastructure side working on triaging, failed CI builds, developers fight tickets for, we’re talking about 10, 20,000 tickets a year.
Hans Dockter 00:16:56 It’s terrible. And then the developers, they need to get their, want to merge their poll requests. So they’re stuck. They need to wait until the ticket is looked at. And the problem is the percentage is high enough. If the percentage that developers, that they think it’s very likely that it’s not my code, and then they pass it on. So the number is very high. Of those of pipeline flakiness and it’s creating so many disruptions, so much waste. It’s one of the biggest problem that we’re seeing out there and hugely frustrating for everyone involved.
Giovanni Asproni 00:17:25 Yeah. To me, from what you say, since that the machinery there can be either of help if it works well, let’s say with the legacy code, if you have build pipelines and build tools and everything and you know, maybe tooling for a factory that actually work well can enable you to improve the code base. But if they’re flaky or don’t work well, they actually worsen the situation are kind of amplifiers in a sense.
Hans Dockter 00:17:48 Giovanni, and now an interesting thing is, and again I guarantee you that every company in the world, this is happening every day. Your CI build failed to pull request build, you run it again, it’s green, okay? This can be merged. So the obviously is flakiness. You haven’t changed the input. Now the result is different. Now it works, it’s green. But you made a good point. In many cases they don’t know do I now ship a flaky product to my customers or do I just deal with a flaky CI pipeline? We don’t know. The customer will discover that and this is happening every day. Many, many charts.
Giovanni Asproni 00:18:27 How does the team structure affect productivity? I mean, not the single team. When you are in a multiple team situation, are there any effects on developer productivity directly measurable?
Hans Dockter 00:18:38 Yes. So when you’re saying multiple teams need to work together on a certain software deliverable. Yes. So that is a very interesting questions. Because you could argue isn’t that in one team. You know what I mean? So are you talking about time zones? Because for me, if they’re all working together, we should look at them as one team.
Giovanni Asproni 00:18:57 What I’m thinking of, let’s say a team, like people that work on the same service. A team of say eight, 10 people. Usually these teams are around what, in between five and ten one team. But then we have the situation of teams. So we have two or three or four or five or 10 or hundreds. I’ve seen, or I worked in projects with really hundreds. And so each of these group, these teams of 10 people work on a subset of the bigger system. But all these teams together pretty much create the entire system. And so I was wondering if this kind of, the structure of this setup can actually affect productivity as well. I tell you when I’m coming from, because I’ve seen situations of teams waiting for other things to be done with their work. This is one example.
Hans Dockter 00:19:43 Yes. Oh, so it’s a very interesting topic. And it leads us again to the tool chain to a degree. Because how is this work organized? So for example, do you work all in one repository or do you have now many repositories that integrate somehow? And for example, one problem we are seeing, so with the whole microservices movement, I think things went over the top and people started to create many, many repositories. Which is a pretty heavyweight abstraction. With very small amounts of code. And now you have to sync the work between all those repositories and you have a very complex dependency craft because now all the output is versioned independently. That complexity has created a lot of problems. Another thing is, in that situation you have described, when you have many, many repositories, so let’s say you are producing a library that is consumed by a lot of other teams.
Hans Dockter 00:20:38 In the organization or service doesn’t matter. So what is really a problem is, so now you change something, how do you make sure that you haven’t broken the consumers or you don’t even know your consumers in that world? You know the stuff you are consuming, you have no idea who is consuming you. So now you basically, you change something and now the consumers have to run into problems. And let you know because the test you are running, because you don’t know your consumers, you’re only testing your stuff,right? You know that test your functionality but not how your change is affecting the consumers. If everything were in one repository that would automatically happen. You would run all the tests of all the consumers, all the producers. And then they’re green and everything works. So the microservices world has really created problems. In that respect, as I’ve just described. And it makes collaboration more difficult.
Giovanni Asproni 00:21:31 Maybe those problems I think, are solved problems, because with microservices, if you really want to use them is you can always have versioning of the interface. The API. Yeah. So it’s like version one, you create version 1.1 or version 2 depend on. And so you can of course serve the old consumers and the new ones. But this creates additional work probably because you need to manage versioning of the APIs and potentially additional tests because you have to keep testing different versions of the same API and then it come up with a deprecation mechanisms and all sorts of stuff.
Hans Dockter 00:22:06 I agree with you, it’s solvable. I’m just saying it has created complexity. For having teams working, you know, in multiple repositories that has created productivity issue. So kind of, yes, I’m not saying it’s not solvable but it has definitely created integration problems that makes that work less efficient. Otherwise, I think another area is that people have different, when different teams have different development pH. So that is now I’m going to reverse. Let’s say you work in one repository and with people that have very different perspectives on testing. So for example, you have now team members that say, I don’t care about flaky tests. And basically you’re now running a build and you’re now always test failing. That are not your test, but you cannot get it through the pipeline. If those tests are failing and you now have members maybe in a different team that work on the same repository that just say, oh, just ignore it, just merge it anyhow, for example. That is another thing where we see a lot of frustration that there is just not a joint development philosophy. And because it’s different teams with different leadership, you can also not escalate easily. With them to fix that. So that is definitely something that can be very frustrating.
Giovanni Asproni 00:23:18 In the situation, I guess that the machine area some limitations world can do because there is an element of actually people agreeing on something.
Hans Dockter 00:23:26 Absolutely. This is where the tool chain can only say, I wouldn’t say the tool chain, this is where you want no, I fully agree with you. But what we need here, at least as a potential way to, I wouldn’t say to solve it, but to kind of pinpoint the problem, is let’s say you’re now in a situation where you are going to your VP and saying, man, those flaky tests. For that, that are introduced by a subset of the group is killing us. And then the answer would be, well how often is that a problem? And then what do you think, how many people would tell you, hey, we had so and so many flaky tests over the last 30 days. No one. And then it becomes a gut feel and the other team saying, oh they’re exaggerating. It’s not that much of a problem. So the lack of observability. So I would say if you have strong, you can derive very interesting data from the tool chain that can point to organizational problems and it allows you to quantify them versus making it a finger pointing exercise that is arguing from the guts.
Giovanni Asproni 00:24:22 I can tell you that in a very large scale project that I consultant for years ago, they had such a problem with the flaky tests that they decided that if more than 90% of the tests were passing, they declared the product good for release. But there was an additional problem that was this 90% was not always the same test because the flakiness, so okay, it was the total percentage, but the test failing could be different as each time. So it was really a gamble.
Hans Dockter 00:24:49 Yes. And I think, and the other aspect of that is then it really demotivates people to write tests.
Giovanni Asproni 00:24:55 Yes. Yeah, I agree. I agree. Good machinery, good observability can help in figuring out, well at least in highlighting the organizational problems. Can be helpful.
Hans Dockter 00:25:06 Yes. And then of course if you have a dysfunctional organization, nothing will happen in any case. But then it’s time maybe to leave. Yeah, true. When you have a chance.
Giovanni Asproni 00:25:14 I can ask you also in your experience, you know, what are some key metrics or indicators that teams can use to measure productivity effectively?
Hans Dockter 00:25:22 Yeah, I think for me, so going back to the dialogue with the tool chain. I think how many dialogues the developers do is almost like a survey every day. How much you like your tool chain and how high the tax you mentally think you are paying for getting feedback. So, and this is fascinating. I mean we all know this horror stories. We know one organization still big enterprise software company, their turnaround time for full feedback cycle is 24 hours. 24 hours and everyone understands that’s a disaster,right? Or three hours or two hours. So that is where everyone emotionally understands that’s terrible to wait so long for feedback. Okay.
Giovanni Asproni 00:26:04 When you mention feedback here, you mean a feedback on say a pull request feedback on yes. Okay.
Hans Dockter 00:26:09 Put feedback on a pull request. Good question. Yes. Because they have very different level of feedback. Thanks for asking that. So, okay, now we did some research. With projects that are in a much healthier state, so one example was we had an organization, they, one team was having a one-minute build the other team similar tool chain, a four-minute built and the team with the, and both sounds like what’s your problem. That’s great for minutes. Hey, who’s complaining about that? And they weren’t complaining about it, but what we could see is that the team with them one minute build time had twice as many builds and test cycles than a team with the four-minute build time. So the four-minute build time was already a tax. They were, less willing to pay than the team with the one- minute build time.
Hans Dockter 00:26:54 And that’s I think where we as an industry need to develop. Yes we have all those nightmares hours and everything is terrible, but even if things is in a pretty good state, at least from a current perspective, improvement still makes a lot of sense. So that is another thing that Microsoft has basically done some serious research on where they’re saying we can now correlate basically every second of improved feedback sign type of cycle time goes into more code. Anyhow, so I think this kind of, how often do you ask for feedback is only a very important metric and the differences we see between organizations and teams are astonishing. Two orders of magnitude, sometimes how often do you ask for feedback? And that is one metric that I think is a key, is a key metric basically.
Giovanni Asproni 00:27:41 So here we are talking about, well the feedback loop for pull requests, how long it takes, then the build time as well. It show
Hans Dockter 00:27:49 Also local build. How often you run local builds. Your pre-commit build before you even submit a pull request. Okay.
Giovanni Asproni 00:27:55 And is there also frequency of commits? Is it a good metric or not?
Hans Dockter 00:27:59 It’s a very good metric because there is a really toxic correlation. The longer your build time, the bigger is the change set of your pull request because hey, I don’t want to pay this tax, I just 50 flaky tests I have to deal with, it takes ages. Then I have to deal with all this stuff. This is a flat tax. So let me get a bigger change set going to make it worthwhile. And the problem is with all of that, even with the long build time, the happy path is not so much a problem if your builds are always succeeding. You have a lot change sets, you have a long build time, it’s not so much of a problem, it’s fire and forget they’re always succeeding. Everything is good. But unfortunately they don’t, now you have flas combined with a very large change set? I mean the debugging nightmare that is causing, right? Because there’s so many dimensions that are now responsible. So it’s a huge problem. Yes. So I think number of commits, a lot of commits, small same sets, walking to the pipeline is for me one of the most important health metrics.
Giovanni Asproni 00:29:02 Okay. Can I ask you, maybe let’s say off the cuff numbers that you think are good numbers. Like, what is your experience, what have you seen in the best companies for example that use pull requests? How long is the feedback loop? You know, we are talking minutes, hours, days.
Hans Dockter 00:29:18 I would say we see many organizations where when you take everything into account, the flakiness, how often you have to run it, the tickets you file until they’re fixed because you see IBU green again. It is off hours and for the best companies it depends of course. But this is in the area of minutes. So that’s where I’m saying there’s really two orders of magnitude. Within the, I would say ignoring the extremely pathological cases. Where this is even worse.
Giovanni Asproni 00:29:47 Yeah, well yeah, I’ve seen pathological cases of really one or two months for a poor request. Of course, yes. Then when you match, that is a hell to much anything before.
Hans Dockter 00:29:57 That’s because, so you’re making a very good point because the big change sets now the likelihood of major merge conflicts. And that is for me, the worst thing in the world. Merge conflicts are just, oh my god.
Giovanni Asproni 00:30:11 Actually now that you’re talking about pull requests and me and have you got any maybe video tools matrix or have you seen any metrics with your customers around maybe comparing teams that do, if you have found any do that do per programming and trunk based development compared to teams that maybe do more pool requests and more solo working? Have you come across any interesting numbers?
Hans Dockter 00:30:34 No, unfortunately we haven’t started to instrument that. So that would be super interesting to look at that with different types of development philosophies. Or approaches that would be super interesting. So one thing, the story I wanted to share is there was this company, and it’s a while ago but it had a big monorepo in a bill time was six to seven hours? If it passed, and leadership was saying we need to do scrum, we need to have weekly sprints. And I said, okay, let’s do this. And the problem was they were not even able, usually if they were lucky, they got one working built a week and that’s your point, it took them month. So, and sometimes and the only way often they got a working bill for a week, they had to tell everyone, freeze no commits anymore. And then how do you do scrub and then the testing team had nothing to do initially. And then when they were getting close to their release date, it was like all help broke loose. It’s exactly what you’re saying. So the whole idea, hey we want to introduce Scrum when you have that kind of feedback cycles and before you get a pull request through, forget about it.
Giovanni Asproni 00:31:41 Yeah. Well you have to do to make some deeper changes to everything pretty much.
Hans Dockter 00:31:46 Yes, exactly. Exactly.
Giovanni Asproni 00:31:47 Okay, so just now before move on, I’m interested. So in terms of interesting metrics, we said the frequency of committees is an interesting one that length of the feedback loop for pool request seems to be another good one. Are there any other that you suggest, as in, you know, generally good ones to look at?
Hans Dockter 00:32:04 For me, super, super important is how early do people ask for feedback. Because we want to shift left. We want the developers work in small increment and ask feedback.
Giovanni Asproni 00:32:16 So when we, say how early they ask for feedback, what do you mean exactly with the asking for feedback?
Hans Dockter 00:32:21 Iíll explain. So for me, a very good metric is number of pull request build versus number of local builds. So if we see that developers stop running local builds often because the experience is to do this. Then they say, ah, it’s such a pain. Let’s do this on ci, which means they will do it much less often and later then you’re shifting the whole thing to the or because maybe the machine, when they run their tests have to run it on their machine and then it consumes so much load from the machine that they cannot work in the IDE. That is something. So you want to shift them left and everyone says, every organization you are shifting left is great. We have to do this. This is our philosophy as the reality is though it’s shifting at this point in time. Because the code bases are growing. So I think it’s super important that people take care of the local feedback experience and it’s often completely ignored because you have zero data about it.
Giovanni Asproni 00:33:18 Yeah. That is quite interesting actually. It’s quite interesting. And I think, you know, to add a bit to this point, I’ve seen situations where in theory the company management wanted to do the thing, okay? In practice they gave developers underpowered machines that were not able to do much. So maybe not enough disc, not enough memory, not enough processor power. And so I’ve seen situations where for the developers pretty much a local build was not really an appealing option.
Hans Dockter 00:33:45 Yes. I can tell you a funny story. And so before I tell what I think it’s a funny story, but then you need to think about, so I really think that the MacBooks, the mx, process are game changer for development. And they really expanded what is possible to do on a local machine significantly. So that is very exciting but there are still limits obviously. So that is where I think technologies need to come in. Like hey, you run your build locally but you can run your test. Fully trans, you just run your Mave and your Gradle, your SPT, whatever build and but the tests we take and run them on distributed agents. To take that load away from you. Or there is Facebook has pioneered that. There is technology like predictive test selection where we use an AI model.
Hans Dockter 00:34:41 Or where you use an AI model that predicts, hey, for this change, let’s say 900 out of your a thousand tests with 99% likelihood will succeed. They’re not sensitive to that change. So let’s weaken the quality gate a little bit. But with the advantage that you get much faster feedback with much less resources. So there’s really interesting stuff and that’s where DPE comes in, ? To think about this, to model this, to create this kind of backend, ? So that you can actually get early feedback. And the funny story I wanted to share, there is a company that the build time was going about 50 minutes, 20 minutes, 25 minutes. And the developers were supposed to always run local builds before they commit and push something. And one point they were escalating to the vp, we cannot do this anymore. It takes way too much time, ?
Hans Dockter 00:35:30 And you will like that story. So then they established a task force, ? And task force was trying to fix it and the goal was to get it under five minutes and they got it down, I don’t know, from 28 to 22 minutes with some improvements. It was far away from the five minutes. And then they came up with the solution and they were quite convinced that this was a good solution. They said, Hey, just stop running the test locally and boom, everything
Hans Dockter was under five minutes and they were pretty proud
Hans Dockter about it. And I was like, okay, whatever. ? Great. Yeah,
Giovanni Asproni 00:36:02 Yeah. It doesn’t seem to be a good solution,
Hans Dockter.
Hans Dockter 00:36:05 No it doesn’t. ?
Giovanni Asproni 00:36:06 Okay. And now let’s talk about steps to improve productivity in a company. So say an organization, the problems, maybe they’re not measuring much at the moment, but they have the feeling that everything is low, they want to go faster. This is a typical thing in most of organizations. Everybody wants to go faster. So what are the first steps you would suggest?
Hans Dockter 00:36:26 It’s very easy. And now we’re coming back to the management. So the thing that was eye-opening for me years back was when, so Android is a, you know, mobile in general is a more complex platform than doing let’s say microservices with Java for the backend, ? So because there’s so much stuff that needs to be done for building mobile applications, build time is particularly a problem, ? And then the Google Android team developed then a lot of improvements, ? For making this faster, but then quite a few of those improvements required that you needed to reconfigure and redo your build and some things in your code, ? To get benefits from this. And they did it because all the servers, were they the developers saying that’s our biggest problem by far now they’ve offered significant improvement and now less developers than they thought were making use of those, ? And then they did a big survey, ? Why is this the case? And the developer said, our biggest problem is to make the business case to the management that we get resources to do this.
Giovanni Asproni 00:37:28 I see.
Hans Dockter 00:37:29 So, and let me give you a concrete example, ? How it works in most organizations. So many organizations move to Athermal ci, ? Let’s always throw the CI environment away before you run the next build. So all your local state, your local caches are thrown away and it’s culturally, maybe we have time to go into this, ? But why this happened and, and, but the CI build time has drastically increased because of that including pull request build. And the big part of that is now downloading all the dependencies over and over again. So you have no idea how many people came to me and said, oh, dependency download time is a big problem. And then I was asking the most naive question in the world, well what is your average dependency downward time? We don’t know across the board, it’s not a single organization almost on a planet that could tell you what is the actual dependency download time.
Hans Dockter 00:38:19 Although everyone is complaining about, so if someone would come to me and my company would say, man, dependency downward time is a huge problem, we need five people to fix it. I would say, well how big of a problem is it? ? And then, oh, we don’t know. Everyone is complaining about it and that puts you in a paralysis state. So am I now willing to invest into this when I cannot quantify it? So then it’s , maybe let’s put half a person on the problem
Hans Dockter, ? But the reality is now when we measure this at some of those organizations, we see easily 30% of the decide time. It’s just downloading dependence. And now you can have a conversation with the leadership saying, look, this is 30% that cost us compute resources because the CI machines are idle while downloading dependencies, ? It cost us 1 million minutes per whatever period of time Now you can have a conversation, now you can make a business case.
Hans Dockter 00:39:07 So my, the answer for me is therefore observability is everything. Don’t even start on anything gut feel wise. You need to measure to see how big the problem is and the particular problem. And then to measure what is the improvement, ? So people think always thinking, oh there are some silver bullets, ? So let’s introduce a new technology without any observability. So for me it all starts with observability. With instrumenting and that’s how we would do performance improvement of an application. Yeah, ? Yeah. But that is the kind of immaturity we still have in that area of the software industry, ? That we are not applying best practices we would apply in other areas.
Giovanni Asproni 00:39:44 Am I correct If you, if I interpret this as we developers, technical people should actually do our homework first, get the numbers and then go when we complain, go there with appropriate information so we can have a conversation.
Hans Dockter 00:39:58 A hundred percent. Otherwise it just developers whining again, all those developers, ? They’re never happy
Hans Dockter. There’s another, it’s one of my favorite stories, ? Because it shows the cultural kind of mismatch. So we were talking with the big car make, they wanted to talk about developer productivity and I’m kind of, I mean it’s a little bit mean. I always ask those questions, ? Very simple. How many feedback cycles do your developers run per day? What is the average time of them and how often do they fail? I mean the most innocent simple questions you can imagine. And the answer I was getting is we don’t know. In this case the answer was, well we need to do a survey to figure this out, ? And then I said, well if I would go to any of your factories and would ask how long is assembly part A taking to get from there to there? How long is it sitting in a shelf? And I wanna get an answer on that in five minutes, the person would be fired, it would be culturally disgrace, people would be ashamed. It’s even beyond, it’s beyond just, you know, rational. It would be like this is not who we are, ? And that’s where we need to get as a software industry for our own stuff. You put it perfectly ? We have to do our homework, you know, when it comes to that. Yeah.
Giovanni Asproni 00:41:09 , also one other thing I was thinking at least, I mean you can tell me this if it matches your experience. In my experience and, and apparently also in literature, there is a lot of work that developers do. So let’s say that they have the perfect machinery. Yeah, that everything works fine. But then if you look at the literature and at least also my experience and probably you’ve seen this with some of your customers, is even for the features we produce, usually there is a kind of paral rule thing that is pretty much 20% of the features we produce are the ones that are used. The remaining 80%, we could have avoided them. So I was wondering from a management perspective, maybe looking at that first, which is probably a bit more difficult than measuring developer productivity. The machinery looking at that could be another attack point because that would free up some productivity for developers to do something more useful.
Hans Dockter 00:42:04 Yes. So I absolutely agree. And the question is, I think the likelihood of this to happen that you do not, the most important stuff increases with cognitive fatigue, ? If you are fatigued, you rather do incremental stuff, ? Something you feel familiar with, then thinking about what is actually the most important thing, ? So I mean there could be many reasons for that, but fatigue can be one major reason. I see it even in our company, ? When you’re fatigued, you don’t see the forest for the trees, it’s hard to kind of think about the bigger picture because that requires a lot of cognitive control, ? So if you are in a fatigued organization, I think it’s very unlikely that they work on the most important stuff. So
Giovanni Asproni 00:42:45 If I understand correctly, you’re saying that fatigue has acts as a kind of an amplifier for this problem again, yes. Because Yes, exactly. It’s like brings you to look at the familiar or in general spend less time thinking about what is important. Yes. And what is good.
Hans Dockter 00:43:00 So one thing I found fascinating is how do they measure cognitive fatigue in the cognitive science experiments, ? And because they’re still not a hundred percent sure of the biological root cause of fatigue, ? They only know it exists, ? So how they measure it is in many of those experiments, ? So let’s say they let you do a lot of context switching, ? To kind of accelerate fatigue, ? Some math problem here, then a word problem here, you get the idea, ? And then they always give you two choices, ? One is low cognitive effort, low reward, the other is high cognitive effort, high reward. So let’s say $10 if you solve the difficult problem, $1 if you solve the easy problem. And when people start switching from difficult to easy, that’s the indication, the no fatigue. And that kind of correlates well to what you just described, ? In software.
Giovanni Asproni 00:43:50 Okay. Yeah. I never thought about this from this angle. That is quite interesting. Now another question also, we talk about management and developers and how to do, how our role work to put things to managers. Now, let’s say that in a company people, they said okay, we’ll do some developer productivity engineering and then we put the tools in place to measure everything and to have the metrics that are of interest to us to improve the situation. Yeah. Who owns this metric? Who owns the developer productivity as a concept in an organization? Developers, managers, both.
Hans Dockter 00:44:23 Yes. It has to be both. So what we see, and that is the very promising thing, ? Even in the, let’s say non-tech industry, let’s say financial world, ? We now see , developer experience and developer productivity metrics have c-level ownership. I think that’s extremely important. Otherwise you’re not getting anywhere, ? If that is not at the highest level considered strategic, it will always be deprioritized, ? So I think that is super important. But, and that leads us to an interesting topic. So we talked about metrics, ? That let’s say are now owned even at at sea level, ? That indicate improvements to developer productivity. I think what has happened in the last couple of years that metrics were perceived as measurements, how good the developers are. So basically there is now a c-level team that now starts instrumenting developer productivity to find out who are the bad developers like naively said, or the non-productive ones, ?
Hans Dockter 00:45:20 And use that and also to make, , personal decisions, ? And I think not even talking about the ethical aspect here, I’m just talking about the economic aspect here. This is absolutely wrong, ? Because when that starts happening, those valuable metrics, let’s say number of builds, people will start gaming it. At the moment, people are honest with how many builds they’re running. It’s a survey about how good institution experience when they learn about, oh this is now
Hans Dockter how we getting measured, ? People will start gaming and completely destroy the value of that metric. So that’s why I think there need to be some form of contract. Yes, those metrics are very important to us, but we understand that we cannot draw , naive conclusions, you know, to the quality of the teams or the developers based on those metrics. But those metrics are important and the developers need to be part of the metric discussion to understand what is the root cause that this metric is bad, how can we improve it? So the metrics should be a guideline to improve the machinery and the environment versus looked at it, hey, this is now a tool to measure how good you are, ? I think that’s super important in my opinion.
Giovanni Asproni 00:46:30 Can I ask you a question on that? So we already said that your company created a developer productivity platform this develop. So, but this question is about now the, this aspect that you just mentioned, are there, is there a way to put in these platforms provision so that it’s difficult to trace back to individuals what the metrics, you know, trace back the metrics to individualness?
Hans Dockter 00:46:53 Yeah, so some companies, especially in Europe, , they even have legal requirements to do so and we can hide all the personal information for example. But I think it’s a pity that we have to do that. And it’s, there’s something culturally wrong when we have to do this to protect developers because we lose very interesting information, ? I want a developer to see for themself, ? Oh my God, ? Why is everyone else in my team? Why am I the one with the build time that is five times slower than everyone else? Oh, I’m the only guy in Singapore and there is the Cashs are so far away from me. So there is so much value in being able to break down the metrics if they are looked at it from a collaborative perspective, Hey, we want to improve the environment, ? You, you lose information. But yes, there are ways to do that, but you pay a price for that. And it’s sad, ? That you have to do it to protect developers. Something is wrong with the culture in my opinion, if that is necessary, ? But it is possible, yes.
Giovanni Asproni 00:47:51 Yeah. I have to say that I’ve seen in the past I consultant for a big company on a big project head and there was a major cultural problem there in terms of trust and the CTO there are mandated that developers had to do a certain number of commits per month or else Yeah, engineering manager said to do at least one per month as well because he wanted an organization of technical people. But it’s interesting what happened there was exactly what you mentioned before. So some people that were not really doing much development every day because they were doing other useful work, they changed inconsequential files, you know, text files, something that could appear in a commit because they knew that people would only look at the number of commits, not what was changed, see not the quality of the work or the particular effect. So that was quite dysfunctional.
Hans Dockter 00:48:40 I mean with all respect to the leadership there, but I mean I think the person in my opinion that need to get fired is not any one of the developers, ? Because this is so obviously stupid and ineffective and setting the wrong incentives, ? It’s, but yeah. And or people, you know, measuring how, how much you move your mouse cursor, ? There are quite a, I know quite a few organiz and with the remote from home situation, ? They, you heard about it, ? The productivity paranoia in the leadership. Are people still doing work or they’re just hanging out on a patio, ? Looking at the sunset.
Giovanni Asproni 00:49:11 Okay. And then can I ask you, let’s say that night now we are in a situation where the culture is fine and we, and the teams want to, and the company wants to improve, improve their productivity, they’re employing some, so are there any pitfalls that the teams encounter when trying to improve productivity?
Hans Dockter 00:49:26 There was an organization that had a very successful organization but they had for legacy reasons, they had a situation where they had tens and tens of millions of yml files, ? So it’s very, but it’s, it sounds very kinda okay, what is interesting here, but tens of tens of millions of YAML files that they were assembling, ? In their archive files, ? For deployment, ? And that made their build time very long, ? We’re talking about just copying those the way they did it, ? And working with them was taking 15 to 20 minutes, ? For every, you know, every build, ? So and they were on build system A, I don’t even mention any name, ? It doesn’t matter, ? And then we did something, ? We wrote them some logic to get this time down to seconds with build system a dramatic improvement.
Hans Dockter 00:50:14 I mean obviously, ? But then they said, oh, but we wanna migrate to build system B and if we now make build system a much better, then we might lose the mandate to move to build system B. And we see this all the time, they’re holding something hostage so they don’t wanna do the low hanging fruit because then people might say, oh this is good enough, ? Don’t do anything more long term. So they were not applying this change and the migration is taking years, ? While people still have to use build system A and giovan, I guarantee you I see this over and over and over again people because of the immaturity, ? They use a big problem to say, oh, you can only solve this by doing something strategically and then they get ED to do something strategically. Although they could fix it two orders of magnitude faster, but they don’t want to because then they can do something more strategically. That is one pitfall that I see
Giovanni Asproni 00:51:05 That seems to be a problem with the developers themselves. The technical people seems to be, they’re more interested in doing something new than improving something that is already there are working and delivering value. Exactly.
Hans Dockter 00:51:16 Exactly. A new toy, a new silver bullet, ? Absolutely. And that is one part and I agree that is probably more than 50% of the story. And then the other part is that it might be the only way to get resources, let’s say, for doing something which they think is actually important. So it’s a mix of both. But you, you make a very good point.
Giovanni Asproni 00:51:34 Yeah. I think we are going towards the end of the interview now. So I will ask you to the future, you know, how do you see the future of developer productivity engineering evolving.
Hans Dockter 00:51:43 Yes. So when you look at, let’s say chemical process engineering, you can study this at every university. There are probably hundred, thousands PhD thesis about every aspect, ? Of chemical process engineering. So there’s a deep cognitive roots, tremendous body of research, of empirical research, of theoretical research, ? And it’s in the DNA of the companies to apply that, ? That’s what I would be looking for, for developer productivity engineering, that this becomes really also an academic discipline. I mean, it’s a trillion dollar industry, ? And we’re treating the production like it is. We we’re building a A lemonade stand, ? I mean, seriously, ? It’s changing now, ? That’s I think the foundation, the academic foundation, and the depth of thinking and modeling, because especially you talked about it our I models, ? How do you measure things, ? I mean, we just at the beginning and academia is not involved yet. Bridging the divide from all the interesting research in cognitive science to what does it mean for software development? Such a body of research there we’re not applying yet, ? So those are, yeah, building this foundation and then all the rest will follow. .
Giovanni Asproni 00:52:54 Okay. And are there any interesting developments at the moment, you know, in your opinion, like some technology cut advances or improved methods that we’ll see in the short term? Because what you just mentioned for the future seems to be more kind of long-term vision. But is there something coming in the shorter term in your view that we don’t have yet,
Hans Dockter 00:53:12
Hans Dockter? Well, of course we cannot finish this podcast, ?
Hans Dockter talk about generative ai, ?
Hans Dockter. So, yeah, I think generative AI is there, ? And that I think it increases the pressure. So here’s how I look at this, ? So let’s say you get the windfall from generative ai and every developer writes twice as much code. Well, the code needs to be built, needs to be tested, yada, yada, yada. So the tool chain pressure, if the experience is now, good luck when everyone writes twice as much code, ? And then you will not get that windfall from generative AI in terms of getting features to the customer faster, et cetera, et cetera. The other thing I think it will also, it should push us in the following direction. So imagine we talked about cognitive fatigue, ? So let’s say there’s an organization where in average the cognitive fatigue sets in three hours at 11:00 AM ?
Hans Dockter
Hans Dockter 00:54:01 In the morning, ? So those people are not going home. There is still busy work to do. Rename this, refactor this, write some simple algorithms, ? So, which adds some value, although not the 80% value we are talking about before, but it adds the 20% value. So now imagine a world that we’re probably getting close to where all this busy work is done by ai and what are those people doing starting
Hans Dockter from 11:00 AM, ? So I think the companies that push the cognitive fatigue out to later, ? Will generative AI will increase the competitive advantage they’re getting from that. So that’s something companies should think about. And then what generative AI can do for the tool chain, ? To make it easier to improve the tool chain, ? One reason is, let’s say for example, if everyone had a perfectly modularized code structure, it would have a huge benefit on bill times on how can you paralyze and cache and reuse, ? Dramatic, ? But hardly anyone is tackling that because modularization is just a hairy issue with the cyclic dependencies and things like that, ? So if AI could tackle some of those hard problems, ? To reorganize your tool chain into software in a way that it’s a much lower cost, that would be amazing, ? For the productivity of software developers.
Giovanni Asproni 00:55:28 Okay. Thank you very much. I think we’ve come to the end. I think we’ve done a good job introducing developer productivity. Thanks, Hans. Is there anything else that you would like to mention? Something that we missed that you think would be important to mention?
Hans Dockter 00:55:40 No, you asked all the questions that have covered all the stuff I wanted to share. So yeah, thanks a lot. This was a great experience from my side. I really enjoyed the conversation.
Giovanni Asproni 00:55:49 Okay. Thank you, Hans, for coming to the show then. Has been a real pleasure for me. And this is Giovanni for Software Engineering Radio. Thank you for listening.
[End of Audio]
