In this episode, Abi Noda, founder of Pull Panda and DX, discusses developer experience with SE Radio host Brijesh Ammanath. They examine the basic concept of DX and why it matters before diving into a wide variety of issues, including methodologies for measuring DX, the main factors that influence it, and strategies for overcoming common barriers in improving DX. Abi also suggests coping mechanisms developers can use when it’s not possible to improve DX. In the last section, they consider developer productivity and the various measures for it — the ones that work and those that don’t.
- Twitter: @abinoda
- Linked In: https://www.linkedin.com/in/abinoda/
- White Paper on “An Actionable Framework for Understanding and Improving Developer Experience”
- “The elusive quest to measure developer productivity”: Talk on issues with popular engineering metrics today
- “DevOps metrics” by Dr. Nicole Forsgren: This is a thoughtful overview of the trade-offs of system-based metrics vs survey-based metrics
- How to Measure Anything: a wonderful book that helped inspire this journey
Transcript brought to you by IEEE Software magazine.
This transcript was automatically generated. To suggest improvements in the text, please contact [email protected] and include the episode number and URL.
Brijesh Ammanath 00:00:16 Welcome to Software Engineering Radio. I’m your host, Brijesh Ammanath, and today my guest is Abi Noda. Abi led engineering teams for over six years before founding Pull Panda, a developer productivity tool used by over 7,000 developers, which was acquired by GitHub in 2019. At GitHub, he led research collaborations with Dr. Nicole Forsgen, McKinsey and Microsoft research, which was the impetus for founding his new company DX. Abi, welcome to Software Engineering Radio. Is there anything I missed in your bio that you would like to add?
Abi Noda 00:00:49 No, I think you covered it. Thanks so much for having me.
Brijesh Ammanath 00:00:52 Thank you. We will be talking today about developer experience, commonly referred to as DX. Once we have gained an understanding about what DX is, we will jump into various topics covering measurement, influencing factors, strategies to improve DX, the barriers encountered, coping mechanisms adopted by developers and developer productivity. Let’s start with the basics. Abi, what is developer experience and why is it important?
Abi Noda 00:01:15 Sure. Well, just in the past couple years, I think DX or developer experience has really become a buzzword. And so it’s really getting thrown out a lot. Now, I think the most common definitions of developer experience really boil down to two. You often hear people referring to developer experience in the context of vendor solutions and external tools, meaning these are companies for example, companies like Stripe, that build products for developers, and they talk about developer experience in terms of the user experience of their products for developers. The other context in which developer experience is used is internally. And so if you look across to industry, there are more and more teams that are called developer experience teams. And these teams look internally within their companies and at the experiences of their employees who are developers. And so when we talk about developer experience today, we’re talking about this latter category. We’re talking about the holistic lived experiences of developers and their day-to-day work, working on professional teams. And really, I think developer experience is about all the different points of friction that these developers encounter in their work. And these things span from tools to the processes, to the culture of their organization. And so improving developer experience is really about empowering developers to do their best work so they can ultimately deliver the best results for their companies and teams.
Brijesh Ammanath 00:02:44 I like the phrase lived experience of developers. So to put it another way, I would say it’s the quality of engineers, quality of life for engineers. Would that sum it up?
Abi Noda 00:02:54 Yeah. Quality of life for developers, quality of life means different things, right? Again, developer experience is really a collection of things that affect those live experiences. So quality of life, I think would be an yeah, I think that would be an acceptable way to sum it up.
Brijesh Ammanath 00:03:11 Right. Does DX become more important as teams work in a remote or hybrid model?
Abi Noda 00:03:16 Well certainly. So I think developer experience like many aspects of sort of the lived experience of employees becomes more difficult to understand and stay aware of as teams shift to hybrid and remote working models. And so really, I think developer experience is important regardless of whether you’re co-located or remote because developers are ultimately the lifeblood of today’s modern digital economy. And of course, companies invest so much money into developer’s salaries and tools. And at the same time, we know there’s so much room for improving engineering efficiency and developer happiness. A few years ago, Stripe published this study that global GDP is reduced by over 300 billion per year due to developer in efficiency. And so developer experience, isn’t just this kind of feel good topic about quality of life, so to speak, but it’s also critical to the bottom line for business.
Brijesh Ammanath 00:04:15 Interesting. So besides the bottom line, what are some other benefits of an enhanced DX?
Abi Noda 00:04:21 Sure. So first of all, when we say bottom line what does that mean? So we have research showing that developer experience is a top predictor of developer productivity and satisfaction, which of course these things correlate to the bottom line of companies. So we see that companies with top quartile developer experience, not only outperform their competition in terms of productivity and their ability to innovate faster, but also ultimately that business performance, whether that’s commercial or non-commercial goals. Aside from kind of things pertaining to productivity and money, itís also today this big war for talent going on. And so this ability to attract and retain top talent, is probably just as if not more important than how quickly you can deliver. And developer experiences is key to keep your developers happy and engage within your organization.
Brijesh Ammanath 00:05:16 I did have a question around the retention by using DX as one of the differentiating factors by companies. So is that a leak table for DX by companies? You can, if you are looking for a job, you can look up how, how is that company performing on DX and that influences your decision whether to take that job or not, and equally from a different angle, if you are a company, how do you go out and tell developers that you’ve got a great DX?
Abi Noda 00:05:47 Sure. Well, today there’s not some formal way in which DX is tracked and shared to candidates, but I think sort of unofficially or informally DX is really important to candidates who are looking for new job opportunities. I think developers are always sort of aware where other good developers are going to work and what they’re hearing about those companies. And much of what you hear is about the developer experience. It’s that, it’s that ability to do great work and to work collaboratively and be successful as a team and be empowered with great tools and the ability to work highly effectively. And so while it’s not something that’s necessarily shared and tracked in some kind of objective way, it’s definitely something that’s talked about a lot. And you, I think you’re seeing that trend more and more. Companies are really looking for ways to differentiate themselves. And of course you hear companies sharing we got ranked top 10 places to work, things like that, but within sort of one-on-one conversations and the recruiting process the internal developer experiences, definitely a huge point of emphasis in order to make a place attractive for developers to work. And I’m sorry, could you repeat the second question?
Brijesh Ammanath 00:07:16 So the second part was, if you are a company and you really move the needle in terms of improving your DX internally, how do you make sure that you use that to advertise and attract more talent?
Abi Noda 00:07:29 Sure. Today, a lot of that, again, as I was mentioning is captured sort of informally. So throughout the interview process, it’s usually becomes sort of a two-way information sharing sort of process where candidates are being interviewed, but candidates are also interviewing the company and they often get to interact with developers on multiple teams and leaders in multiple parts of the organization. They get to see blog posts and Open Source code written by people who, who work there. So I think candidates are able to get a pretty good pulse or sense of what the developer experience looks like just by asking questions. And as we will kind of I’m sure talk about later asking questions is really the key to how we understand developer experience within organizations. There are certainly sort of system based metrics that you can look at. I mean candidates could ask how quickly do your build finish? But really to understand developer experience holistically, you need to look at self-reported data from developers.
Brijesh Ammanath 00:08:36 So build time is a question that you could ask. What are some other questions that a potential candidate could ask the companies they’re interviewing with to get a gauge or an understanding about the DXs?
Abi Noda 00:08:49 Sure that’s a great question. I haven’t been in that position in a little bit but you based on our research, I can say that there’s a whole range of factors that affect developer experience and a subset of those are things that are very top of mind for developers. So I think a common question would be around the development environment. What is the process you have to go through to actually set up code locally, run it, work on it and create a change. Then beyond that, I think there’s the organizational side of making changes. So what is the review process? What’s the approval process? What steps you need to go through to actually take something you’ve completed and released it to customers. What’s that feedback loop look like? So a lot of it has to do with this kind, the feedback loops that are scattered across the development process and experience and asking about those and really asking not just about necessarily the time it takes to complete those different aspects, but moreover are they frustrating? What’s the experience like? Do they, is it a joy to do work at this company or is it, does it feel like a slog?
Brijesh Ammanath 00:10:05 Some excellent tips. Thanks. Now that we have a good understanding about DX, let’s take a bit deeper starting with measurement. What are the different methodologies for measuring DX?
Abi Noda 00:10:15 Yeah, this is one of the reasons why DX is so important, right? Because in general, this problem of measurement or measuring productivity has been such an elusive problem for engineering leaders for decades. Companies spend millions of dollars on developers, but they don’t have clear indicators on how effective their developers are or where they need to invest in order to improve. So just generally speaking, we as an industry really desperately need more effective approaches to measurement. When you talk about developer experience specifically, there are really two ways to measure it. There are certainly aspects of developer experience that can be understood by looking at our systems. So the example we’ve already talked about for example, how long developers wait for builds to complete? That’s something you can, if you have a well-built pipeline, you can look at the stats and it’ll tell you how long builds take to complete.
Abi Noda 00:11:14 But really the only way to measure experience holistically is to get self-reported data from developers. And I’ll give you one example. So code review, right? So we know that the time spent waiting for code reviews can be a major point of frustration and delay for developers. And this is an example where there are two acceptable ways to measure it. You could look at systems, so you could try to look at your JIRA board or your GitHub pull request data to understand how long it takes for things to kind of move through the process and move through the systems. But you could also ask developers to share self-reported data on how long they wait, or maybe not just how long they wait, but also how long they’re blocked or how much they’re frustrated by the process. And that’s really so important because one of the problems I think we have today and the way we attempt to measure, not only experience, but just things in the development process as a whole is that we miss the business context. We miss the root cause, right? When we look at certain types of metrics, they tell us what’s happening, but they don’t actually take into account what the world looks like for a developer. And as a result, a lot of these measures don’t really provide an accurate or meaningful in the trenches view of what’s causing friction for teams and developers.
Brijesh Ammanath 00:12:41 So if I got it right, there are broadly two different methodologies. One is the system metrics and the other one is self-reported measures. But it would be wrong to just use the system metrics because that will not give a true picture of what’s happening on the ground. So you need to, in addition to the system metrics also need the self-reported measures.
Abi Noda 00:13:00 Yeah. I mean, I would go one step further and, and say that really there’s, there’s a very limited amount of information you can capture from the system metrics alone, both because of the challenge and accurately instrumenting our systems, but also because the systems only touch really a fraction of the things that affect developer experience, , earlier we talked about how developer experience was not just about the build tools, but it was really about the end to end experience of building and developing and releasing software and working cross the team or multiple teams to accomplish that goal. And so when you think about what’s involved in the developer experience as a whole, it goes far beyond just the time spent waiting for builds or the time it takes for a pull request to go through a system. You only get a very limited understanding of the developer experience. If you only look at system data. And so really organizations need to move toward getting self-reported data from developers in order to get a holistic understanding of developer experience.
Brijesh Ammanath 00:14:04 How should leaders think about the importance of measuring and focusing on developer experience versus other metrics they may already track?
Abi Noda 00:14:12 Yeah. Well, if you talk to most leaders about engineering or develop their productivity type metrics, most still feel pretty lost and frustrated with the status quo. Right? I mentioned earlier that we really, as an industry need more effective approaches to measurement. And it’s, I think it’s really interesting to look at the way we, how we measure his evolved. If you look back to the 90ís or even 80í and 2000ís, there was a big emphasis on measuring output. Things like lines of code or velocity points. Those are the most common ones, but developer understood in many leaders quickly understood that, those types of out output measures don’t account for the complexity or nuance of engineering work. For example, shipping something that has more lines of code is not better than shipping something that is less lines of code. And it doesn’t tell you how difficult that task was.
Abi Noda 00:15:08 Moving forward more recently, there’s been this shift towards process metrics or delivery metrics, right? So DORA is a great example of this metrics like lead time, pull request, throughput, pull request cycle time. This is what I see most companies today focus on measuring. And as mentioned earlier, this a huge problem with these metrics because they don’t account for the context or root cause. For example, your data might tell you that code reviews are taking three days to complete, but if you go talk to the team, they might tell you that that is perfectly acceptable to them because they work on multiple tasks at once or lead time, for example, or it’s appointment frequency. The DevOps annual report says that elite performers release things daily, constantly. But what if you’re an iOS team that has to wait two weeks for your app to get reviewed by Apple. Right, but that lead time metric doesn’t really speak to you in terms of the reality of how you work.
Abi Noda 00:16:12 And so there’s really, I think, a big need in the industry right now for a better way to measure and impact engineering, productivity and performance. And I think that’s what developer experience has the potential to offer right? Experience provides the true in the trenches indicators of the bottlenecks and performance of developers and their teams. And this is so important to leaders because as we talked about earlier, not only of course it’s a top priority for them to maximize sort of output and productivity and performance, but it’s just as much of a priority for them to retain their talent and keep their developers happy. And really, there’s not many other ways to do that then to focus on developer experience and measure and improve it
Brijesh Ammanath 00:17:01 From what you’re saying, you’re saying that developer experience measures will be different for each team because each team is unique and they’re working on different emergent problems. And if that’s the case, are we saying that it’s not possible to have a standard set of measures for DX?
Abi Noda 00:17:16 It is possible to have a standard set of measures for DX, but it’s also important to understand that every team’s different and every team has their own challenges and their own unique points of friction and not only just teams. So if you go down to the individual level, you’ll find that people on the same team can have very different experiences as well, depending on what they’re working on. So an example would be, if you’re on a team, you might have a senior engineer who’s really in the role of developing features, but also supporting the rest of the team mentoring the more junior developers doing a lot of the code reviews. So their biggest points of friction. Friction might be the amount of time that is taken away from them to do mentoring type work or code review work, or just the workload in general. Whereas if you were to go talk to a junior engineer on that team, they might be really struggling with understanding the code base or understanding requirements for work getting clear scope or being able to reduce their work down into sort of manageable sizes in terms of batch size. So really problems come down to the individual level and to understand them, you need to look at the individuals, you need to look at the teams and then you can look holistically at the organization and what the patterns and major themes are.
Brijesh Ammanath 00:18:41 Can DX be compared across teams? Or is it similar to velocity, which is unique to each team and hence should not be used to measure different teams, but does the individual teams’ performance over time
Abi Noda 00:18:52 Developer experience can definitely be measured across teams, but like any measure, you have to be careful when doing that, right? You don’t want to create unhealthy competition between teams. You also don’t want to accidentally create incentives for teams to sort of game their metrics, right. To alter their metrics because there is a reward for doing so. And so you can certainly compare developer experience across teams to help with learning, both learning for leaders, to understand where investments or support may be needed. And also learning for teams to understand best practices and learnings from other teams that are doing things well. But you have to be careful whenever you are comparing, because it can create unhealthy dynamics, competition, and ultimately lead to kind of ruining the measures themselves.
Brijesh Ammanath 00:19:49 I thought that was a good discussion on measurement methodologies. Moving on, let’s talk on some of the influencing factors that impact DX, what are the most important factors that affect DX?
Abi Noda 00:20:00 When we talk about factors, first of all, we need to think of, there’s a few things at play here. So there are factors that affect developer experience. These are things like code complexity or ease of release test efficiency, or having clear direction, having good requirements. But we also know that these factors themselves are highly dependent on the individual. Like we were just talking about, and we have an understanding of what affect, how these different factors affect an individual. And these things come down to things such as seniority. Like the example I provided earlier, where more senior developers might be dealing with a totally different set of problems than junior developers. We also know that the presence of problems is, has a huge effect on developer experience. Meaning developers feel the pain of problems much more than they feel the joy from there being a lack of problems.
Abi Noda 00:21:01 So developers will be able to easily identify and feel the friction from things that are affecting them day to day. Another aspects of this has to do with just kind of individual interests and expectations. When you hire people in an organization, they come from varying backgrounds, different previous job experiences. And so people come in with just a different set of expectations for one developer coming from on-prem development and switching over to, for example, cloud API development, they might feel like deploying code once every two weeks is incredible. It might feel like magic, but to someone coming from working at a startup, SaaS startup, they would find that two weeks might feel really slow to them. And so a lot of developer experience does often boil down to the individual perceptions and expectations of what good looks like.
Brijesh Ammanath 00:22:00 What factors are most influenced by senior leadership and how can they play a positive role in improving these factors that impact DX?
Abi Noda 00:22:09 So when we look at developer experience within organizations, there’s a really interesting set of dynamics at play. So what we typically find is that most issues affecting developer experience are local team issues, meaning they are specific to the areas of the code that the local team is working in. It is specific to the way that local team works. It’s specific to that the way that local team interfaces with other teams. However, there are also some aspects of developer experience that are, tend to be more global. So I think release process, local development environment, test infrastructure. Those are things that tend to be shared across an organization and therefore, and oftentimes also owned by a centralized team. So when earlier we talked about these developer experience teams, they’re also often called developer productivity teams or enablement teams, most medium to large size companies have a group that’s responsible for kind of owning and improving internal tooling.
Abi Noda 00:23:21 And those tools are typically used across the company. And so when senior leaders are thinking about how do we improve developer experience within our organization, it really has to be a two-pronged tack. There needs to be a huge emphasis placed on enabling these local individual squads and pods to understand their local points of friction and improve those. At the same time, there needs to be an examination of patterns across the organization or shared tools that may be affecting everybody. And those may be things that can be uniquely impact in a high leverage way by senior leadership.
Brijesh Ammanath 00:24:02 Okay. So if I understood that try the local factors will be primarily influenced by the team itself, whereas the tools and the horizontal teams, which support the team, which support many teams, those are the touchpoints, which can be influenced by senior leadership to improve DX.
Abi Noda 00:24:22 Yeah, there’s things that, for example, senior leadership could invest in bringing purchasing the new tool that makes releasing easier, right? That would be an example of something that senior leadership could easily influence, but there’s a lot of things. For example, teams that are struggling with how they work in terms of process, right product management process, or the way they collaborate and communicate, or the workflows they have to review and approve changes. Those types of things aren’t, I mean, senior leadership can’t prescribe a one size fits all solution for the entire company. That’s not how engineering organizations work. There’s a huge emphasis and value put on enabling teams to be autonomous and grow on their own. And so what senior leadership can do to affect those types of issues is to provide those local teams with a way to both measure and understand their local team problems and provide them the support they need to make progress in improving those local issues.
Brijesh Ammanath 00:25:33 Right. How important is the code review process for DX?
Abi Noda 00:25:37 Code review process is something that comes up frequently and there’s a lot of different facets of code review process. There’s the portion of it that involves the person who’s getting their code reviewed. So developers often have frustration with the amount of time they have to wait to get feedback or the back and forth that’s involved in the code review process. There’s also the quality of the code review. So developers can feel maybe short change or frustrated with not getting thorough feedback about the work they do. Or on the flip side, sometimes developers feel like the feedback they get is, is too harsh or too strict, right? It’s, it’s not, there’s not a clear set of expectations around what is an acceptable level of quality or standard for the code they’re writing. And as a result code reviews can sort of stall. Then there’s also the experience of the reviewers.
Abi Noda 00:26:34 There are often people in roles that involve doing a lot of code reviews. For example, if you’re a senior engineer or someone who’s maintaining a tool that receives contributions from across the company, you have to do a lot of code reviews and there can also be this frustration with the experience of going and reviewing other people’s work. Is that work well described? Is it, is the change clear and is it, what do you do when a change isn’t to a certain standard or what do you do if a change just seems totally off?
Brijesh Ammanath 00:27:07 So I guess the most important thing is to ensure that, the code process itself is well documented? People understands its importance and the reviewer is appreciated for taking the time doing the code review.
Abi Noda 00:27:20 Yeah. There’s definitely a set of tradeoffs and that’s one thing that’s common across developer experience is that it’s really just a lot of tradeoffs. So for example, with code review, there’s this clear tradeoff between the time it takes for reviewers to complete code reviews and the quality of the feedback, right? Reviewers can just hop into a poll request or a change request and just give a thumbs up sign and that will allow that change to be approved and released. However, was that a thorough code review, right? No. And so there’s this constant tension between quality and flow and that’s of course, common across software development. And so really finding that right balance does involve as you said, setting, having clear process and expectations and standards around the code review process and how it’ll be conducted.
Brijesh Ammanath 00:28:17 Flow is seen as a key dimension for developer productivity, you just touched on it. Can you help define flow and what can be done to improve flow besides the code review process?
Abi Noda 00:28:29 Sure. Yeah, well, I think there’s a couple different ways the industry thinks and talks about flow. So sometimes when leaders talk about flow, I think they’re just referring to output or throughput, how much stuff are we out outputting, right? Whether and how they think of that might be in terms of commits or poor requests or features. The other way the industry thinks about flow has to do with really the, the psychology definition of flow, which has to do with this sort of Nirvana state of creativity, immersion, and engagement, and that anyone doing creative work can find themselves in. And so, and of course the two are related, right? When you’re developers can be in this flow state, they are often more productive and able to release more work leading to more output and throughput and flow. And so when we talk about that latter definition of really helping developers feel immersed, and engaged and in the zone, if you will, again there are a number of factors that affect this, but I think one of the most common ones is just interruptions.
Abi Noda 00:29:48 So we know that interruptions take developers and really anyone doing creative work out of the flow of their work and deeply reduce their, both the psychological state of how they feel while they’re doing the work. But also the output that they’re able to produce. In addition to uninterrupted time, flow is also affected by things like how stimulated developers actually feel with the work. So are they working on a boring mundane task? That is something they’ve done a thousand times or are they working on something that’s new and where they’re learning and feels stimulated? Do they get to learn as part of their work? In addition, autonomy is a big element of flow as well. So if you’re a developer, you’ve probably been in a situation where anytime you try to make a change, someone comes in and tells you to either rewrite your code or tells you a different way of doing things. And this can be incredibly deflating, right? To feel like you don’t have this freedom to create and produce in the way that you feel is best. And so that autonomy over how things are actually built is also a huge aspect of enabling developers to feel in the zone and immersed in their work and ultimately as be as productive as they could be.
Brijesh Ammanath 00:31:11 Now, we’ll move on the next section where we’ll talk about the barriers that are there in that start companies or the teams from improving developer, what are the common barriers in improving DX?
Abi Noda 00:31:26 So there’s a number of barriers to improving developer experience, but it really begins with a lack of visibility and awareness. As we talked about earlier, there’s this huge problem in the industry around just what to measure. And as a result experience is not something that most organizations are measuring today. So they don’t actually have visibility into the types of problems that we talked about. Like how many organizations have a good pulse on how much developers are getting interrupted, or whether developers have sufficient autonomy in their work or code review quality. These are things that aren’t measured today. And therefore, they lack that visibility and awareness and when problems don’t have visibility, they aren’t prioritized. And so that’s what we see really with developer experience. If you talk to developers across the industry and just ask them about their work environment, you’ll often hear them just lament at the inefficiencies and the barriers that they face on a day-to-day basis and just trying to do their work.
Abi Noda 00:32:35 And of course these frustrations ultimately lead to them leaving their jobs, or becoming apathetic in their work and disengaged, but those same sort of complaints and problems often aren’t raised and surfaced within the organization and they’re not prioritized. So this sort of inability to understand and see and quantify problems leads to a lack of these problems being prioritized. And that’s really, I think the first set of barriers that organizations face. Now, once problems are understood because sometimes even if things aren’t being measured, there may be an outspoken developer or group of developers who are pointing out problems, or there may be a team like a developer experience team looking at friction points that exist. Then there’s this other problem around buying and ownership, right? So problems take time and money to actually improve. And organizations need to really understand what the return on investment may be.
Abi Noda 00:33:38 And that’s often really difficult for developers to advocate for things that sort of affect their work environment, but are a little disconnected from the sort of day to day objectives of teams, which is the ship features the customers. And in even complicating that further, a lot of problems as we talked about earlier are a little muddy in terms of ownership. So there may be these tools that have a clear owner, for example, the build systems may be owned by the developer experience team and a company, but a lot of the problems, for example, around collaboration or both within a team or across multiple teams, there’s not necessarily a clear percent organization who’s in charge of being a steward of that problem. And as a result, because there isn’t a clear owner, it can make change more difficult people don’t like, where do people send their complaints? Where do people have conversations and where can people go to, to have improvements championed? And so hopefully this is a nice little overview of the types of challenges and barriers organizations face when trying to improve DX or, or even getting started with knowing that they should improve developer experience.
Brijesh Ammanath 00:34:53 Completely agree, lack of visibility, no ownership, are very tough, challenging barriers. Do you feel some of the barriers have become more pronounced in a fully remote work setup? And also in contrast, have some barriers disappeared due to a remote setup?
Abi Noda 00:35:11 Yeah, that’s a great question. I think problems with visibility certainly are affected by the shift to remote working. A lot of that visibility and awareness is stuff that is more easily picked up when you are working in a co-located environment and you can have water cooler conversations with coworkers and hear how their day is going. And what’s frustrating them. These types of problems don’t often come up as much in, for example, asynchronous conversations, or even in retrospectives. In our conversations with developers, we find that retrospectives are often very focused on sort of sprint objectives. They don’t necessarily go into kind of systemic problems with how teams are working or how the organization is working or things like technical debt, right? Those are things that sort of persist across for months or even for years. And those things don’t really get raised. In terms of some advantages of remote work,
Abi Noda 00:36:21 I do think that elements such as work life balance, uninterrupted time, for example, in some cases have improved due to remote working. However, you do actually see some evidence of the contrary where people are actually feeling more interrupted because of tools like Slack and are having more difficulty with the work life balance because of there’s not this boundary between the office and where they work. And so largely I think developer experience and visibility into it has become even a more difficult and challenging problem as organizations and teams have shifted to hybrid and remote.
Brijesh Ammanath 00:37:01 OK, now we’ll move on some strategies that companies and teams and developers can use to improve DX. Let’s start off with what are the common strategies employed by companies to improve DX?
Abi Noda 00:37:14 Sure. So many companies there’s been this shift recently towards setting up developer experience teams. And I think there’s this organic trend in the industry right now where organizations are placing more emphasis on developer experience. And one of the things that’s unique about how they’re approaching developer experiences, that the key element to it is that they treat their developers as if they were their customers. So in the same way that we gather feedback from our customers and understand their satisfaction with different elements of our products or services, companies are doing the same with their developers. They’re asking developers about their satisfaction and different areas of their experience, and then working to systematically measure and improve those. And so organizations that are doing a good job at improving developer experience have some kind of systematic approach to it. They have a systematic approach to measuring on some cadence, whether it’s monthly or quarterly or biannually, they’re gathering feedback from their engineers across the development life cycle and across all the different types of factors we’ve talked about.
Abi Noda 00:38:28 And then they have a process for how they move the needle on those issues. And as we were talking about earlier, a key to that last piece about moving the needle has to do with that balance of global issues and points of friction and local issues and points of friction. So the organizations that are making the biggest improvements to develop our experience are really empowering their local teams to make local improvements themselves. Not just the global developer experience team, making a couple improvements tools each quarter, but empowering the entire organization in every team to be continually improving. And they’re doing that by providing measurements and feedback systems to those local teams so that they can have the information they need to guide where to improve and what actions to take.
Brijesh Ammanath 00:39:24 Interesting. I was just thinking back about the definition you gave initially, when you mentioned DX can be considered from tool lenses. One is where you have companies building tools for developers, and then you have developer experience, which is internal to the company. So when we think about DX and the strategy being employed to developers as customers, it’s almost that DX is moving into the CX world, where you’re seeing developers as your customers.
Abi Noda 00:39:51 Yeah. It has a lot of similarities to customer experience, certainly user experience. I think where that definition of developer experience that you see product companies and vendors use. The difference between that and internal DX is the vendors are only looking at a partial scope of the overall developer experience, right? Experience with tools is really just one of many, many factors that affect developer experience. So really they are talking about the same thing, but just a different scope, right? And when you’re thinking about developer experience holistically, it’s, it’s really everything, everything in the work environment that affects how developers feel and how they approach their work.
Brijesh Ammanath 00:40:37 Yeah, I get it. I think what, what we’re saying is that you could have a vendor build great developer tools, which gives a great experience to the developers who are the users, but internally that product that’s being built, the developers who building it, the culture might not be great, or the collaboration might not be there. And the internal developer experience could be very different to what their CX is.
Abi Noda 00:40:59 Absolutely tools is just a fraction of the picture, right? I mean, you can have great tools, we talk to companies all the time where they have great tools and teams don’t even use them. Or they have great tools, but only half the teams use them because half it’s hard to build tools that suit the needs of everyone across the company. And so, and when we talk to developers tools, don’t often necessarily come up as their top points of friction. Give you an example. One of the things that we found slow down teams and developers the most is, lack of having clear scope and requirements on their tasks. This leads to working on the wrong things or working on things, and then discovering that they weren’t designed appropriately and having to do rework. And so when you, when you’re looking at internal developer experience through the lens of how do you create the most effective engineering organization possible, tools may surprisingly not actually be at the top of the list of opportunities to actually drive improvement to your overall productivity and performance.
Brijesh Ammanath 00:42:12 Right. What strategies can an individual team member adopt to improve his or her DX?
Abi Noda 00:42:18 That’s a great question. So much of developer experience is about the team. It’s about the shared tools, the shared knowledge, the interactions. But as we talked about earlier, there’s also this individual element as well, where individuals have their own unique frustrations and point of friction that they experience. And so there’s a few strategies that individuals can employ to improve those. One of the most common we see is what we call job crafting. And that means that developers actually sort of tweak their own roles and job descriptions, if you will, to meet the demands of, and deal with the friction that they’re encountering. And so a common example of this would be a senior engineer who is, wants to get, spend more time on mentoring the team, but is dealing with the frustration of the balance between their day to day job requirements and being able to support others. And so someone in this role may actually go to their manager and have a conversation about changing their role slightly to actually provide them, let’s say 20% of their time during the week to block off, to support the rest of the team. So you see this constantly, I think on teams, individuals kind of redefining their role and their expectations in order to better meet the demands of, and needs of their team, and also have a more productive and satisfying work experience individually.
Brijesh Ammanath 00:44:00 That brings up a very interesting point. Job crafting seems like a very valid concept, something which can really improve developer experience, but all of that is based on developers speaking up. And one of the issues that we see commonly is teams and team members, developers not speaking? And that reduces the overall engagement, which is very counterproductive to include DX and has an overall detrimental effect on team culture How do you stop this thing from happening? If youíre already in a team, what steps can be taken to revert this behavior?
Abi Noda 00:44:33 Well, the dynamic you’re describing is extremely common. I mean, you talk to teams and managers all the time where they kind of jokingly talk about how their meetings and retrospectives go. And it’s a lot of silence, right? Not everyone speaks up and participates. And there’s an element of that. That is just somewhat related to the types of personalities that are common in engineering, but there’s also a big element of that, which has to do with psychological safety and people feeling comfortable enough to speak up and share their honest opinions and thoughts. And that’s one of the things you find with developer experience is that, although there are all these different process and tool related points of friction, none of those things matter as much or can be improved without developers having a degree of psychological safety, where they feel comfortable speaking up, speaking up about those problems and, or having candid conversations about how to improve those problems.
Abi Noda 00:45:39 And so really enabling developers to speak up again, is I think comes down to a, creating a culture of psychological safety, both within teams, but sometimes across teams. So developers feel safe doing so. And there’s another part of it, which is just that not all developers feel comfortable speaking up in certain types of social settings. So maybe live meetings, isn’t the best forum for developers or voice their concerns. So finding different methods for developers to be able to share concerns, whether it be through surveys or through asynchronous discussions or threads can really help developers sort of share their thoughts in a medium, through a channel that feels most comfortable to them.
Brijesh Ammanath 00:46:27 Thanks. I think that transitions us into the coping mechanisms that developers and team develop, if the strategies to improve DX to not really work out. So, so what are the ways coping mechanisms developers can use with a poor DX?
Abi Noda 00:46:42 We have industry level data on sort of how developer experience affects things like retention and attrition and productivity. But we also have insights on what this looks like at the individual level. So through our research, we’ve found that there are several different common coping mechanisms, or in other words, what developers do when areas of their developer experience aren’t improved, or if developer experience as a whole, it’s not to their satisfaction or isn’t being improved. A few of these, these are really funny. Well, some aren’t funny, but for example, one of the common things that has come up is a focus on personal projects. So developers who sort of get frustrated with their work environment, aren’t getting as much fulfillment and satisfaction out of that. So they actually start looking to personal side projects for that satisfaction or for that learning. Right? Another thing related to that is just reduce engagement.
Abi Noda 00:47:43 So we’ve seen many developers who are frustrated with elements of their work environment simply sort of become more apathetic and less excited about their work and even worse consequences gaining the system. So we see developers who, for example, if they feel like their estimation process at work is unfair, or if they’re being held to unreasonable deadlines, they will intentionally misreport their estimates to create more buffer time for themselves. And that of course doesn’t serve anybody, especially not the business. And lastly developers often talk about leaving or looking for new work. And we know from talking to both developers and leaders, that the developer experience is one of the top reasons why developers leave. It’s not as many people think just about salaries and pay, but it’s often about feeling like they’re in an environment where they just can’t get stuff done as efficiently as they’d like, or where they’re not set up to succeed individually or with their team. And so ultimately there are a lot of sort of coping mechanisms that manifest themselves before someone leaves, but all of those have bad consequences for the business. And especially once people leave, of course, it’s so expensive to find higher and onboard developers these days that that presents an enormous challenge to the businesses.
Brijesh Ammanath 00:49:12 Completely agree with that. The research you’re referring to is the white paper that you have co-authored titled, An Actionable Framework for Understanding and Improving Developer Experience. I’ll make sure we add a link to that in the podcast notes.
Abi Noda 00:49:25 Sounds great.
Brijesh Ammanath 00:49:26 We’ll move on the last topic, which is around developer Op30, which is one of the key facets or outcomes out of an improved developer experience. How do you define developer productivity?
Abi Noda 00:49:40 Wow. That is the elusive question of the last three decades for everyone in engineering. Really developer productivity doesn’t have any single definition. And if you look at, for example, the work of one of the co-authors of this paper, Margaret Ann’s story, she’s published dozens and dozens of papers about this topic of the differing ways that engineers and managers and people in other roles, view productivity. One of the interesting things that came out of one of our recent papers is that developer’s perception of productivity actually does heavily revolve around their perception of the amount of output or the amount of activity. For example, the number of tickets they’re able to complete. Whereas the perception of managers actually has a lot more to do with the performance of the team. Are they delivering on their commitments and projects? So that really highlights how there’s really varying definitions out there of productivity.
Abi Noda 00:50:45 I think if you’re asking me individually on my definition of productivity, I believe that productivity is ultimately around how developers feel? Meaning that because software development is not an assembly line. It’s not a factory where you can just count the widgets coming out. People try, right? People try to count things like lines of code or poll request or tickets, but anyone who’s in software development knows these things don’t capture the size or complexity or nuance of that work. And so they’re sort of misleading signals. And so to really gauge productivity, to me, it has to involve the perceptions of the developers, their perception on how much is getting done. And when we say perception, think of it as estimation, we don’t just mean their high-level feelings, and intuitions, but their judgment and their estimation of how much work is getting done. And whether that amount of good is that amount of work getting done is sufficient or good or as good as it could be. So for me, developer productivity is very perceptual and that, and the perception of developers, the sentiment of developers is the best signal to understanding how productive your engineering organization is.
Brijesh Ammanath 00:52:06 That’s an interesting way of looking at productivity. So how would you, what would you think are some good measures for productivity?
Abi Noda 00:52:13 Sure. So some examples would be how much of their time is lost due to inefficiencies. I mean, that’s something that anecdotally comes up in conversation a lot with developers in one-on-ones and when consulting companies come in and do, try to understand what’s slowing down a company, but just getting an estimate on a regular basis from developers of how much waste there is in their system, meaning their processes, their tools. That gives you an estimate it’s perceptual, but it’s an estimate no different than when developers are estimating, how long something will complete or giving something an estimated number of points. It’s an estimate that’s, self-reported based on the expertise of your developers, but that gives you a real quantifiable signal, right on the amount of inefficiency and waste in the system that could be improved. And in fact, when we’ve run that measure with companies, we find that that number is upwards of 20 or even 30%. And when you think about that in terms of headcount, right, that means that these organizations have as much opportunity to improve their output and their performance by looking internally as they would hiring 30, 20 to 30% more engineers to do work. So I think that spells the size of the opportunity, again, for leaders to focus and improve developer experience.
Brijesh Ammanath 00:53:42 And what are the commonly used flawed measures of productivity that are currently seen across industry?
Abi Noda 00:53:49 Sure. I think the most common ones are, of course, the output measures such as lines of code and number of velocity points. I think more recently, you’re seeing kind of a resurgence of those types of metrics. For example, number of poor requests has become a popular metric, but really number of poor requests is no different than number of lines of code has the same flaws. And really a poor request is just a group of commits, right? So you’re essentially counting something very similar to commits. I think even going beyond that, I spoke earlier about some of the problems with process metrics, things like lead time and cycle time the problem with looking at those types of metrics as north star, if you will, or team level metrics is that they lack context. There’s no single definition of what a good cycle time or lead time looks like, because it really depends on how a team works and what they’re working on. I think really today, both output metrics and process metrics have flaws. And I think the industry needs to move towards measuring experience because it really solves for the shortcomings of these existing types of measures.
Brijesh Ammanath 00:55:01 Thank you. A couple of questions before we wrap up, you have founded a company focusing exclusively on DX. What solution does your company provide and what is the engagement model?
Abi Noda 00:55:12 Sure. DX provides a solution for any organization that wants to measure and improve developer experience. We provide a solution that helps systematically measure over 40 different factors across developer experience. So things like ease of release, test efficiency, uninterrupted time, and we provide a solution that surfaces those metrics, not only for leadership, but for those local teams, because as we discussed earlier, it’s so important to provide these types of measures back to teams so that they can make their own local improvements. Typically, our engagement model is that we are brought in by either a CIO or CTO or the developer experience team or dev productivity team. So someone pretty high up within an organization that is either already looking at developer experience specifically, or is more broadly concerned with just improving velocity or retention of their developers. And we partner with these leaders to implement our solution and also provide the support that’s needed to drive a program of continual improvement and measurement to see tangible improvements in not just developer experience, but those bottom line metrics and signals. We’ve talked about such as attrition and output and team performance.
Brijesh Ammanath 00:56:37 Before we conclude the catchall question. Was there anything I missed that you would like to mention?
Abi Noda 00:56:43 No, I think you asked great questions. I think we’ve captured it all.
Brijesh Ammanath 00:56:45 Thanks people can follow you on Twitter, but how else can people get in touch?
Abi Noda 00:56:51 People can feel free to connect with me on Twitter or LinkedIn or just email me. My email address is a [email protected].
Brijesh Ammanath 00:56:59 Abi thank you for coming on the show. It’s been a real pleasure. This is Brijesh Ammanath for Software Engineering Radio. Thank you for listening.
Abi Noda 00:57:07 Thanks for having me. [End of Audio]