M. Scott Ford, the CTO of Corgibytes and host of the Legacy Code Rocks podcast, discusses managing dependency freshness. SE Radio’s Sam Taggart speaks with him about why dependency freshness is important to ensure that your code has all the latest bug fixes, how exactly to measure dependency freshness, and some of the insights that teams can gain from monitoring freshness over time. Brought to you by IEEE Computer Society and IEEE Software Magazine.
This episode is sponsored by Miro.
Show Notes
Related Episodes
Other Resources
Transcript
Transcript brought to you by IEEE Software magazine and IEEE Computer Society. This transcript was automatically generated. To suggest improvements in the text, please contact [email protected] and include the episode number and URL.
Sam Taggart 00:00:18 For Software Engineering Radio. This is Sam Taggart. I am here today with M. Scott Ford. Scott is the co-founder and CTO of Corgibytes. He regularly conducts training courses on O’Reilly’s learning platform in LinkedIn Learning. Scott is the host of the podcast, Legacy Code Rocks and manages their Slack group, which I’m a part of. That is how I met Scott. Scott is here to talk to us today about a tool he has built to manage dependency freshness. So first of all, Scott, let’s talk about defining dependencies. What do you mean by the word “dependencies”?
M. Scott Ford 00:00:51 Yeah, so I think in general it’s best to think of it as code that you didn’t write that you’re pulling into your application from somewhere else.
Sam Taggart 00:00:58 So how are you typically pulling that code in? Is it through a package manager or both?
M. Scott Ford 00:01:02 It’s varied across history of programming. So I think in the days of yore, it would’ve been like copy and pasting from another source or copying a file into your project directly or buying a third party product that had to be installed into your system with an installer. So there are like DLLs that were installed that way. And then also in some ecosystems those pieces of code that you’re consuming from elsewhere are either part of the operating system or can be installed with an operating system package manager. But for most modern programming language environments, there is some kind of dependency manifest file that allows you to specify the packages and libraries that you’re depending on and optionally what versions of, of each of those libraries you want to consume.
Sam Taggart 00:01:53 So you mentioned versioning plays a very important role in managing freshness. So can you talk a little bit about that?
M. Scott Ford 00:02:00 So freshness, there are multiple ways to define it. First, let me take a step back and say that the term dependency freshness was introduced in a paper by Joel Cox et al. I forget the name of the paper, but the idea is how out of date the version you’re using is, and there’s different ways to measure how it’s out of date. So you can look at the version number and you can compare components of the version number that you’re using to components of a later version number and get a distance between the version you’re using and another version. Whether it’s the latest version or the latest for the series that you’re targeting or whatnot. The metric that I like the most is one that’s called libyear and it’s also referred to in the paper as a time-based kind of temporal distance. So it’s looking at the release date of the version of the package you’re on and comparing that with the release date of the latest version. And you get a distance in years. And so if it’s been six months between the version that you’re using and the release date of the latest version, then the libyear for that package will be 0.5, because half a year being six months.
Sam Taggart 00:03:20 Now I see that giving some problems because certain projects have more churn or are releasing more often than others. There’s probably some projects that are stable and only get revised once a year or something when they find a bug versus projects that maybe are in development or are constantly churning out new versions. Is there a way to account for that or this method just doesn’t take that into account?
M. Scott Ford 00:03:42 Unfortunately this method doesn’t take that into account. There are some of the other freshness measures if you key them to stay on the major version that you’re on, but you want to make sure that you are always on the latest minor version. You can look at distance measures that will help you compute that, but Libyear just on its surface is always comparing against the latest. So that is a little bit of a weakness in that. Another weakness is that freshness doesn’t really communicate quality of the dependency itself. So there are other factors to consider as well. Such as is the package that you’re using still being updated by community support or a vendor or has it been abandoned? So you might be on the latest version so your dependency freshness score would look really good because you’re on the latest version, but that version is eight years old or something like that.
M. Scott Ford 00:04:47 And that’s not necessarily a bad thing. Like you said, it could be really stable, it could be just an easy to understand problem. The things that it depends on don’t change often. So it doesn’t have a need to change. Or it could be that there’s just not any more work on it and it’s not receiving bug fixes, it’s not receiving security updates, it’s not getting operating system updates to support newer environments or newer processor architectures. Those are also possibilities. So I think, sometimes these things warrant investigation and I think there are different trends that you can start to look for and you can start to spot. So if you’ve got a dependency that you’re on the latest version, but that version’s more than five years old or four years old or three years old, whatever your threshold might be, maybe it warrants investigating. Has the community abandoned that package and is it time to look for an alternative? So I think there are signals that will start to show up in the data that you can look for and then start to use those as targets for further investigation.
Sam Taggart 00:05:52 So I think this brings us to a very important question, which is why is dependency freshness important? What possible things have you seen go wrong when dependencies are old and out of date?
M. Scott Ford 00:06:04 It shows up in three different ways. There’s three factors, and depending on your organization, each of these factors will have a varying degree of importance. One is staff turnover. So most teams that I’ve worked on, the developers on that team get frustrated when they have to work with libraries that are consistently out of date. It becomes a productivity challenge as well. That’s the second factor. Keeping with that first factor, as people get more and more frustrated, they will eventually not want to work on the project anymore and then look for a job elsewhere. So that is a factor and it is a very real factor for some teams, especially when there are lots of opportunities for software developers to seek other forms of employment at other organizations. That can be a really big challenge for teams managing turnover. And so I think it does create a little bit of turnover risk.
Sam Taggart 00:07:08 Probably recruiting on that part too? So seeing people want to leave and new people come in and if they’re interviewing and you say we’re using 10 year old technology.
M. Scott Ford 00:07:19 And I’ve heard it joked that a lot of, a lot of developers perform resume driven development where they like to pick the tools that’re going to make their resume look good for the next gig. And if the work that they’re doing right now isn’t going to help them further their career and make it easier for them to get the next opportunity, then they feel like they’re starting to take on some career risk as well. So I think that is a real problem. I think it’s really hard to quantify. I think it’s hard to notice when you’re getting into that situation, it is real. Now, the frustration that those individuals are experiencing is often in the form of it being harder to work with an older dependency because of the information that is out there about the projects. So typically my experience has been, and I can definitely speak from personal experience on this. When I’m working with an older version of a library, and I’m trying to investigate how to do something with it. So there’s a problem that I’m trying to solve or I’ve encountered an issue that I’m trying to work around, or I’m trying to solve a problem in a new way and I’m doing a lot of research, what I often find is that the resources that I’m directed to are for newer versions than the version that I’m using. Because what often happens in libraries and packages is as challenges are encountered by members of the community, the people maintaining these packages start to solve some of those common problems in subsequent versions. So you can often find yourself in a situation where the problem that you’re having has been solved by the community, but it’s been done so in a version that’s newer than the one that you’re on, and if there are impediments to you being able to upgrade, then that becomes really challenging.
Sam Taggart 00:09:11 So my reaction to that was actually a different reaction in that I’ve had the opposite problem using Linux a lot because for newer versions of Linux, some of the problems that you encounter, because it’s newer, not as many people have run into that problem. And so I Google, how do I do x, y, Z in Linux and all documentations first of older version of Linux where they’ve now updated all the tools and it works differently. So that’s just an interesting data point.
M. Scott Ford 00:09:36 And then, the third component for why freshness is important is security. And then for a lot of organizations, this is the one that’s most important and that’s, that security or vulnerabilities don’t get fixed in the exact version they’re using. You’re targeting a specific version of a third party project. And so by incorporating that into your code base, if there’s a vulnerability in that third party dependency, there’s now a vulnerability or potential for vulnerability in your code base. Now, you might not be exercising in the library in a way that would expose your project to the vulnerability, but the risk is certainly higher than it would be otherwise if there wasn’t a vulnerability in the third party dependency. So you will need to be able to upgrade to a version of the library that doesn’t have that vulnerability. So looking at freshness and starting to consider a measure of freshness being how many known vulnerabilities are there in a package? So the idea being that if your library is supported by a community that’s very active and stays on top of security vulnerabilities and patches them relatively quickly, then the latest version should not have any known vulnerabilities. There should be at least a version of that project with no known vulnerabilities and you can target that version. And so, the number of vulnerabilities and the levels and severity of those vulnerabilities can start to be used to kind of form an additional freshness metric separate from the version number itself.
Sam Taggart 00:11:19 Yes. That’s interesting. So you’re not only relying on the new version, but you’re also relying on the maintainers so that in a way when there’s a bug, the maintainers actually have to go and issue a new update? So if the bug is found and nobody fixes it, that also doesn’t bode well.
M. Scott Ford 00:11:38 Yeah so if there’s not community support for that, you’ve adopted this, this dependency into your project. And so when you become aware of the fact that there’s a vulnerability, you now have a responsibility to make sure that that gets fixed. And it could be really understanding the vulnerability and making sure there aren’t any code paths in your application that bring it to life, so to speak. Or it could be finding a way for you to fix it or patch that library yourself by forking it and creating your own version.
Sam Taggart 00:12:07 That speaks to the value of having open source dependencies.
M. Scott Ford 00:12:10 Yes. Because that becomes possible. Whereas with a closed source, third party dependency, it’s much less possible. It’s not impossible, it’s much less possible to get in and fix things yourself.
Sam Taggart 00:12:23 Yeah. In that case, the best you can do is try to prevent that code path from being executed or do some kind of mitigation. So how do developers currently manage this freshness and keeping stuff up to date without talking about your product yet Freshli? If people aren’t using that, how do they handle this problem?
M. Scott Ford 00:12:43 Yeah, so there are a variety of approaches and there is starting to be better and better tooling introduced to help teams stay on top of the challenge and stay on top of the problem. Most package managers have a command that you can run that is usually outdated or something along those lines. So there’s like npm space outdated, there’s bundle outdated. I think there is at least, or if not there’s a plugin for it. So there are different package management and communities, there’s NuGet outdated as well. There are different package management communities where you can get a list of the versions that you’re depending on and what are the latest versions that are available and they’ll usually present it in a variety of ways. So it’ll show you the way that the version has been specified in your application and you can specify versions usually in a multiple number of ways. You can specify an exact version number or you can specify an expression that will resolve to an exact version number. And most of these tools that let you know whether or not there’s a newer package, they’ll let you know first if there’s a newer version that satisfies that expression that you’ve defined for the version number. And it’ll also show you if there’s a version that’s even newer than that to kind of help you kind of assess which might be easier to upgrade to.
Sam Taggart 00:14:10 So I have a question for you. You work with a lot of legacy code bases. Do you generally prefer to specify exact version numbers or minimum version numbers and what are the trade-offs there?
M. Scott Ford 00:14:21 Yeah, so I like there to be some mechanism that specifies the exact version number. Usually that’s in the form of a lock file. So in the yarn and npm ecosystems, it’s either packages that lock or yarn.lock in the Ruby ecosystem, it’s gemfile.lock. Not sure what it is in the .net ecosystem, but I know there is a lock file that does pin down the exact version number. And those are important for applications at least to be able to specify the exact version number that you expect to install. Especially when comparing a local development environment with a production environment, you want to be able to have those two environments relatively in sync with the version numbers that are getting installed. Otherwise you can run into potential problems with production, have an older version of a package and the bug you’re trying to fix has actually been fixed in a dependency, but you’re not able to observe that bug because you’ve got a newer version of that dependency locally. So it is important to have some kind of mechanism for keeping those in sync. And if the package manager that you’re using doesn’t provide that mechanism, then I think you do have to specify exact version numbers.
Sam Taggart 00:15:40 That’s the “works on my machine” problem.
M. Scott Ford 00:15:43 Exactly. So when you do have ranges available to you, so if you’re able to partner ranges with exact versions, then I think it makes sense to look at the way version numbers are being specified for the dependency that you’re using and specify the version range in such a way that you’re able to absorb any new versions that don’t have any breaking changes. That’s a promise that the packaging maintainers are making to you is that certain version number increments won’t have breaking changes included. There’s the sim ver or semantic version in convention that specifies which parts of a version number are allowed to change based on what changes have been introduced as part of that version. So if there’s a breaking change, the first digit and the version number is supposed to increment as an example. So if the project that you’re consuming complies with sim ver then you can say that the major version that you’re on, you want to keep that the same, but then all newer versions are acceptable. And then you might specify a minimum version within that category. So it could be that you start a feature that your application needs was introduced in say 3.5. And so anything less than 4.0 is okay, but it needs to be greater than three five.
Sam Taggart 00:17:03 Okay. So how common do you think it is for companies and projects to have out of date dependencies? Is it widespread?
M. Scott Ford 00:17:12 Crazy common crazy common. Yeah, there’s a report, the O-S-S-R-A report from 2023. So nearly half of commercial code bases contain a known high-risk vulnerability that has been patched and it’s 48% of those code bases. So that’s a data point where we can point to and say like, these are known vulnerabilities that are present and there’s a fix for these vulnerabilities. So that means those packages have to be out of date? It doesn’t happen any other way. I don’t know of any specific data for how many companies have, what percentage of their code base is out of date or whatnot. But I think that number tells us a lot in and of itself.
Sam Taggart 00:17:56 Yeah. And if people want to know about the consequence of that, they could probably just go Google Equifax because that is the shining example that everybody knows about.
M. Scott Ford 00:18:06 That example is what really got me really interested in this area and dependency freshness in general. And it was specifically when I read an article that was published by Ars Technica where they reported that it was Apache Struts. So it was a vulnerability in the Apache struts libraries, the Apache struts web framework that was being used by Equifax. The vulnerability was published and at the same time that the vulnerability was published, the fix was released. And it was two months after that publication that Equifax was hacked and the vulnerability that was exploited was one that was fixed prior to that two month window. So the development team had two months to patch it. Now I’ve worked on plenty of development teams where upgrading a major dependency such as Apache Struts, something that’s absolutely foundational for a web project and presents a high level of risk for upgrading.
M. Scott Ford 00:19:12 But their inability to do that within two months to me communicates a pattern of practice within the development team where it had become hard to do upgrades and it wasn’t routine. And so I think teams need to get to the point where doing upgrades of these sort becomes routine. And so you were asking about tools earlier. Dependabot is another great tool for this. There are other other tools that have similar capabilities, but it will notice that there’s a newer version of a package that you’re depending on and it will open up a portal request for you that patches that package and you can look and see if your test suite passes. And so if you’ve got a strong enough safety net to help you notice when upgrading a package is going to break something, then you can start to build some trust that upgrading isn’t going to break anything. That needs to be said with a giant caveat, however, because there have been some sophisticated vulnerabilities that have been inserted exploiting that practice. And these are known as supply chain attacks where a bad actor has gotten access to the source code repository of a popular library and they’ve inserted malicious code and released a new version. So pretty much the only change in that version is the inclusion of this malicious code.
Sam Taggart 00:20:36 So there is a previous episode of software engineering radio that covers that. And I do not have a number off the top of my head, but we can throw it in the show notes. So you start to answer my next question, why do organizations struggle with out of date dependencies? I think you hit on one reason: it’s hard to upgrade and people are afraid of breaking their code when they upgrade and they don’t have the test infrastructure around that. But what other reasons. From an organizational standpoint, is it lack of info or visibility? Is it incentives? Is it sometimes they just don’t care or don’t know?
M. Scott Ford 00:21:09 I think it’s sometimes all of those things or all of them at once or any number of them . I think that awareness is a big part of the problem. When projects get created from scratch, they’re almost always used in the latest versions of things and then they just continue to work. Those versions that you’re depending on, they’re just continuing to work and very much within the vein of it’s not broken, don’t fix it. There’s not really a big need or a big motivator to keep things up to date. And so it can be easy to not maintain that awareness as a result. I also think that on teams where there is awareness that there are problems and that those problems are caused by things being out of date, there’s often a situation where leadership doesn’t see the value in investing and making a change that doesn’t provide any new functionality.
M. Scott Ford 00:22:05 So upgrading a third party to dependency if done correctly, , it’s not a user visible change. Or if there is a user visible change, it’s because things are faster. Hopefully, not things are slower, but usually there’s a performance impact or maybe there’s a bug that gets fixed, but you’re not going to get a new feature as a result. You’re not going to upgrade a third party. It’s going to be pretty rare that you upgrade a third party dependency and now you have new capabilities that you’re able to offer your customers just by upgrading the dependency. That newer version might allow you to then build upon the features in that newer version to provide value to your customers, but there are a lot of business leaders that you don’t see the value in that investment. And then the issue that I mentioned before, that a lot of teams kind of view it as a risk in that a lot of these dependencies are, many of them are foundational or they’re only used in a very small part of the code and both have problems.
M. Scott Ford 00:23:03 So the ones that are foundational, you have the risk that it’ll break everything. And that’s often what people are afraid of. There’s also the risk that it’s going to break something in a very nuanced minor way because just because the team that’s put out the newer version, they’re shipping bug fixes, they’re shipping performance enhancements, they’re shipping security vulnerability remediations, it’s still possible that they’ve introduced new bugs. There’s no way for us to solve the halting problem. So introducing change into your system is introducing risk. So that ends up becoming a risk that you need to manage. And there are a lot of teams that are pretty risk averse when it comes to things breaking and I think that’s largely because they don’t have very thorough tests around whether or not their dependencies are doing what they expect them to do.
Sam Taggart 00:23:53 So that brings me to a question. When it comes to dependencies and testing, do you typically recommend that you write tests around the libraries that you depend on? Or do you just test those as part of your test for the rest of your code? Some path is calling that dependency and therefore it gets tested?
M. Scott Ford 00:24:09 The approach that’s best taken depends on the kind of a dependency that you’re working with. I think for a foundational dependency such as a web framework that is best tested just as part of like having end-to-end acceptance tests. So tests that exercise the full stack because that’s going to be exercising that dependency and the work that dependency is providing you will get exercised as part of that. If it’s a dependency that’s providing you one specific capability, maybe it takes a Word document and generates a PDF or something of that sort. It has a very small nuanced task. I think in those cases it makes sense to have a small integration test that exercises that dependency and makes sure that it continues to work the way you expect it to.
M. Scott Ford 00:24:56 You don’t need to exhaustively test it, the team that’s releasing it is going to be testing it, but test what you need out of it. Test that it’s providing you what you need out of it. And again, those don’t need to be extensive, but really trying to think about if there was a breaking change that was introduced, I would want this test to fail. If you’re using a dynamic language where the compiler isn’t able to notify you about breaking changes like that, then you would want your test to fail if breaking changes like that were introduced or looking at the output of what gets generated and making sure that it’s equivalent to what you expect it to be. And if it’s not, then that means a bug was introduced as part of that dependency and you probably only need maybe one or two of those for those type of minor dependencies.
M. Scott Ford 00:25:45 But for the more foundational dependencies, it’s next to impossible to really exercise everything you’re utilizing from it, because often the design of those foundational dependencies really influence the design of your application. In testing your application, isolation of those dependencies really requires a completely different architecture for your application than is commonly used. So if you’ve adopted such an architecture like a hexagonal architecture, then you can leverage some of the test points that are available within that. But if you haven’t, if you’ve just kind of like you just quickly built on top of it, then there’s not a lot of utility in testing those dependencies independently.
Sam Taggart 00:26:27 Okay. So let’s move on and talk just about Freshli itself. Can you give me a brief summary on what Freshli is? How does it relate to all this stuff we’ve been talking about?
M. Scott Ford 00:26:37 So it’s gone through a few iterations and it’s definitely something that I am continuing to work on and continuing to reimagine and evolve into something better based on feedback from people I’m sharing it with. But it really started out as an experiment for graphing how dependency freshness metrics changed over time. So taking the data out of a dependency manifest file, computing dependency freshness metrics for the dependencies that are specified in there, but not leaving it at that. There are a lot of tools that compute libyear. If you go to libyear.com, there’s a good list of tools and there’s probably one for the language that you’re using. It’ll read your dependency manifest file and it’ll spit out the libyear value for all of your packages. What I was really interested in is how those values change over time and what, what they might be communicating about the team’s behavior and about how things are trending for them, what they might communicate about the community of dependencies that are being relied on.
M. Scott Ford 00:27:43 So what Freshli does is look at the history within Git, and for as much history as it can get access to it computes the libyear metric across that time. By default, it does this in one month intervals and it computes the metric as it would’ve been at that point in time. So if it’s looking at January 1st three years ago, then what would the dependency freshness libyear metric look like at that point in time. And so then able to graph that for teams and then show how these values were trended over time. I also wanted Freshli to be supportive of ecosystems and, and project structures where there were multiple dependency manifest files that are present. And I wanted it to be a tool that supported multiple language communities.
Sam Taggart 00:28:37 So which languages do you support?
M. Scott Ford 00:28:39 The first initial version we had, which you can play with by going to Freshli.io, that’s kind of the initial alpha that has support for PHP via Composer, Ruby via Bundler, Python via requirements.txt, .net via looking at csproj files and I think that was it. And then maybe Perl using a CPAN file. The initial approach that was taken there was to write code in the host language. The language that Freshli was being built with which we ended up, we picked C# but write code to parse the dependency manifests, file formats in these other ecosystems. And that worked pretty well for those initial targets. Ruby, PHP, Python, that worked pretty well. And then we were writing inside of the .head ecosystem, so APIs were available for us to do this work in .net.
M. Scott Ford 00:29:44 One thing we did start to notice is that we felt like we were duplicating effort. We knew that there were libraries in these host ecosystems that already knew how to parse these files and already knew how to do the dependency resolution for an expression and things like that. But we were having to re-implement all that logic and there was a risk that we would get it wrong and we did have a lot of initial bugs at first where we did get some of those things wrong. So that had us wishing that we could leverage code that already existed in those ecosystems. And then when we went to add support for the JavaScript ecosystem, specifically yarn’s lock file format is non-trivial to parse. And then also when we started looking at the Java ecosystem specifically with Maven, the POM.XML file and the various ways that versions can be specified and the various ways that POM.XML files can refer to other POM.XML files, that just seemed like way too big for us to re-implement.
M. Scott Ford 00:30:44 And so we ended up pivoting to an approach where we have a core CLI application that knows how to do things like compute libyear based on a common file format, but then we have language agent programs that are written in the host ecosystem and are able to leverage APIs in that host ecosystem. And then the CLI communicates with those. So we’ve got under that approach, the newer version of the CLI is done for the most part. There’s obviously still things that we’d like to add. And then we’ve got a language agent built for Java and we’ve got a language agent built for .net and we have ones in progress for Go and Python.
Sam Taggart 00:31:27 So is the language agent center like a plugin type system?
M. Scott Ford 00:31:30 It is very much like a plugin like system. It’s a plugin that isn’t executable. So in that executable, when you start it runs a gRPC service that the central CLI is communicating with. So then the CLI is responsible for walking through the history of the Git repository and figuring out what points in time need to be analyzed and what Git Shas need to be looked at. It’s responsible for doing that. And then it asks the language agents to detect any dependency manifests in a particular file tree and then it asks it to create a software bill of materials. We are using the CycloneDx SBOM format for that.
Sam Taggart 00:32:58 Let’s continue just talking about the plugins and the different ecosystems.
M. Scott Ford 00:33:03 The central CLI is also responsible for communicating with the API that the data gets transmitted to. So you can then view the graphs on the website. That’s how that architecture is set up. So the language agent plugins are really only responsible for looking at detecting dependency manifest files in a directory tree and then converting those into an SBOM format so that the CLI can then work with it. So what the CLI does is it uses the SBOM file to compute the libyear metric.
Sam Taggart 00:33:33 We’re talking about SBOMs, you had talked about security earlier and I think this kind of ties into that. So what is an SBOM?
M. Scott Ford 00:33:42 So it’s a software bill of materials and it is a structured document that specifies all of the third party dependencies that an application is using. It can specify things other than that.You can craft an SBOM to also communicate with the third party web enabled services. And then SBOMs can also communicate other forms of metadata as well, such as the license that’s being used. So they’re a pretty good file format for different kinds of analyses, especially the CycloneDx file format because the CycloneDx file format has the ability to also include information about vulnerabilities that are known for the packages. So from a single file format, you can do analysis for what licenses are being used across your project. So you can do some due diligence around whether you have exposure for license violations, like are you consuming code that has a copy left license that you may or may, may or may not be complying with, or maybe your organization has a policy that a particular license just isn’t allowed.
M. Scott Ford 00:34:43 That file format and the tools that can generate for you can help you do that analysis. It also includes in it a tree of the dependencies that you’re using and the tools that generate these can be usually configured as to whether or not they include dependencies that are used for development versus development dependencies that are used for production. So depending on what you’re interested in, you can adjust them that way. And so you can kind of see the version numbers that you’re using. Also, there’s usually a hash of those versions, so you get some kind of cryptographic check of what that code should look like. So then you can do tests against different deployment environments to see if your dependencies have been modified. So that’s another thing that it can be used for is to kind of make sure that the correctness of those files is still good. That those dependencies haven’t been modified by a bad actor.
Sam Taggart 00:35:36 So the SBOM then lists all the dependencies in a tree. It lists the version, it has a hash, it has potentially vulnerabilities.
M. Scott Ford 00:35:44 Yeah, potentially vulnerabilities. It has license information. It’ll likely have author information links to documentation. There’s a lot of metadata that can be embedded in there and it really kind of depends on the tool vendor that you’re using and how much data they provide and also how much data they’re able to to get from the projects that publish these packages.
Sam Taggart 00:36:03 Okay. And the reason SBOMs are in the news is because there’s talk at least to the government requiring them for certain things? Or maybe they already are required.
M. Scott Ford 00:36:12 Yes. So there was an executive order signed a couple years ago by the Biden administration that mandates that they’re created in different scenarios. I don’t know all of those scenarios nor do I know, I’m not super aware of what the requirements for what an SBOM needs to have in it in order to comply with that. But it is a file format that teams are starting to be aware of. And one thing that I think is really nice about it is that prior to these SBOM based tools being introduced, you had lots of different language ecosystems with fragmented ways of defining these dependency trees and the SBOM file formats and there’s two competing, two pretty prominent competing standards. The SPDX and the CycloneDx, or is it SPDX, the one that I’m not used to working with , um, CycloneDx is the one that I’m, I’m more familiar with. That’s the one that we’re using under, under the covers and Freshli. So now you don’t have this huge fragmentation across language boundaries that you had before.
Sam Taggart 00:37:13 Okay. So if I needed to generate an SBOM for my project, I could potentially use Freshli to do that.
M. Scott Ford 00:37:19 Yes. And that’s something that we’re going to make more visible. Right now, the SBOMs are getting created in a folder, in a .Freshli directory. And again, because we’re going back in time, we’re competing the SBOM as it might have looked, we’re not a hundred percent exactly sure that it’s correct, but as correct as we can make it for what the SBOM would’ve been at that point in the past.
Sam Taggart 00:37:41 Okay, so what does that look like? How do I use Freshli? Is it a service that I connect up to my GitHub account? Is it a program that runs locally on my computer? How does that work?
M. Scott Ford 00:37:52 Our initial envisioning of it was that it would be a software as a service thing that you would, you would connect your GitHub account to and then do the analysis. We got some feedback from some different organizations that were nervous about a third party service having access to their source code. And we didn’t want to be a target for getting hacked. So as a way to kind of minimize both the risk to the code base owners and minimize the risk to ourselves, we pivoted away from that approach. And so now the CLI is what does the collecting of the data, it’s what does the analysis and then the, the metrics that it’s computed and the SBOMs are getting sent up to the Freshli API and then you’ll be able to log into the Freshli website and then see the analytics and graphs and recommendations based on that.
Sam Taggart 00:38:41 Okay. So what type of reports and graphs can I see in Freshli? And then the following question is what kind of insights can I get from that? Because you mentioned earlier that instead of looking at one point in time, you look over time. So what does, what does that timeframe tell you? Maybe you can give some examples of things that you’ve noticed and then what the solution was or what problem it pointed out.
M. Scott Ford 00:39:03 Yeah I can certainly give some examples. So the graphs that we’re currently producing are based on libyear and we’re looking at projects total libyear. So we take the libyear value for each package and we add them all together to get a composite aggregate value for your whole dependency manifest file and graph how that’s changed over time. Now there are multiple reasons that that number could go up. It’s highly sensitive to the number of dependencies. So the more dependencies you have, the chances that that number is higher is going to go up. So it’s going to likely scale relative to how many dependencies you have.
Sam Taggart 00:39:42 So question though, can I look at that then as a way of measuring my exposure in a way? Because if I have one really old dependency, I have a lot of exposures, but if I have a whole bunch of different dependencies, even if they’re all relatively up to date, that’s also more exposure. Am I thinking about that right?
M. Scott Ford 00:40:00 Yeah so I’ve been thinking about that total libyear as a stand-in for risk. I wish I knew how to convert that risk into dollars, like how much money is on the line, how much financial risk there is. But I do think it is a measure of some kind of risk. I’m not sure what yet , but as that number goes up, your risk has also gone up and there’s multiple ways that it can go up. Like you said, it can go up a little bit across a lot of packages or it can go up a lot in one package. But the level of risk that you’re carrying in your project has increased. So we look at the total and then for applications, the metric on the libyear website, they say on their website that they try to keep application totals to under 10 years. So for a particular application, under 10 years in their opinion is pretty good. And so you can imagine like if you’ve got like a hundred dependencies, you can hit that 10 year figure by having an average of 0.1.
Sam Taggart 00:41:04 That was going to be another question I had. What’s a good number if I’m looking at the number? ? So, that’s a question. Are you more interested in the number itself or the trend over time?
M. Scott Ford 00:41:16 I’m more interested in the trend. I think that the trends and what they can tell us is more interesting than the numbers. And so the other graphs that we look at, we look at average and average pretty much has the exact same shape. Like if you were to squint, you wouldn’t really be able to tell the difference between the total graph and the average graph. The scale’s different. So you can get a sense of what the scale might be. And then we also graph the maximum value. And I think that that’s pretty interesting and that’s one that I think having some hard thresholds on would be a good idea for teams. And that’s something that we plan on adding support for is being able to integrate Freshli into a CI/CD pipeline and failing to build if the maximum value has crossed a particular threshold. So has the maximum value crossed a year? So is there at least one library that’s more than a year old in your code base? And that could be like just a hard threshold that your team sets.
Sam Taggart 00:42:10 I imagine if you had one that was stable that you didn’t care about, you could potentially keep that out of the running or say if there’s one that we care about, you can ignore it or something.
M. Scott Ford 00:42:21 You could create an allow list or libraries that you ignore, things of that sort. But like having some kind of way to plug into the CI/CD pipeline and say that these are the thresholds that we care about, like whether it’s a threshold on the total, a threshold on the average or a threshold on the maximum. And then the final graph that we put together, it shows all of the packages over time. So every single package your project has ever depended on and what the libyear values have been over time for those. And that’s the graph that I think has shown some interesting insights. And I’ll talk about those and I’ll talk about some insights that can be gained from some of the others. I’ve given talks at a couple different Python conferences and have shown graphs from different open source applications in, in the Python ecosystem.
M. Scott Ford 00:43:08 What’s interesting for a lot of them is the trajectory that libyear goes up before Python version 2.7 was declared dead is a very different trajectory than after Python 2.7 was declared dead. And so you can almost like draw a line on the graph and it kind of shows up visually, like there’s a huge spike in the graph for Python 2.7 when support officially ended. And what I suspect happened is you had an ecosystem effect where across the entire ecosystem of packages, package maintainers went in and dropped support for Python 2.7 and then published a new version. So they were likely able to jettison lots of code that they’ve been holding onto that they were frustrated with or might’ve been a source of a bunch of bugs. And so they were able to release a new version that dropped support for Python 2.7. And then interestingly enough, past that point, the libyear values tend to go up faster than they were before that. So I think that means that the velocity across the open source projects increased after dropping support for Python 2.7. It’s almost like across the ecosystem teams started putting out more frequent versions of their packages because they were able to drop support for Python 2.7. It’s neat to kind of see that signal show up across multiple different applications.
Sam Taggart 00:44:28 So I’m thinking about that. Was that cause or effect that they dropped 2.7 now they could go faster or was it that they dropped 2.7 but when they dropped that support they introduced some new bugs that then they immediately had to fix?
M. Scott Ford 00:44:38 I think both are plausible. So yeah, cause versus effect there can be difficult to tease out. But I think that gets to a point that I try to keep in mind with a lot of these graphs and with a lot of metrics in general is the value isn’t good or bad on its own. It can be high or low, but low and high might necessarily have good or bad connotations. Its high values warrant more investigation than low values is the way that I look at it. And so you want to make sure that you can understand the reason for whatever value you have. Like if the value is low and it’s been low for a while make sure you understand why it’s low. It’s low because the team’s doing a good job keeping an update with the packages or it’s low because all of the packages that the team’s depending on aren’t being updated anymore .
M. Scott Ford 00:45:28 Really looking at that and figuring out and making sure that you understand why the graph you have has the shape that it has. Another thing that shows up on a lot on the total and the average graph is when teams put in investment to do a major framework upgrade. So in a lot of Ruby on Rails applications that I’ve analyzed with Freshli, the total libyear does this stair step up into the right and then there will be like this drop where like there’ll be like a 50% reduction in the score when the team is upgraded to the next major version of Rails. And then it kind of continues to go up at the same rate that it was going up before and then there’ll be another investment and it’ll drop by like 50% again.
M. Scott Ford 00:46:11 But something that I’ve noticed on these graphs is that it never gets brought back down to really low. The trend is still up into the right, the trend is still ever increasing even though the team has been putting in an effort to bring it down at different points in time and that effort shows up and can be observed on the graphs and on projects that we analyzed for clients of ours, we’re able to then correlate based on our time sheets from that point in time what we’re working on. That is when we were working on a Rails upgrade. That big drop in the graph is when we did this work. And it’s a way that can show up and be a measure of how much return you get on the investment that you put in.
Sam Taggart 00:46:53 Yeah, I wonder if the constant increasing is just a form of entropy. Time just keeps marching on and as time goes on you’re adding more features so you’re potentially adding more dependencies and it’s just kind of inevitable that it eventually just keeps going up.
M. Scott Ford 00:47:06 If you think of freshness as a technical debt metric, then I think it’s unique amongst those technical debt metrics because with most of them, they only change when the source code changes. So if you think about different ways to compute complexity, whether it’s architectural complexity or individual code method complexity, those values will only change if the source code changes. But with freshness, freshness will continue to change even if the source code doesn’t change. So the whole attitude of let’s not touch it, it’s not broken, does start to show up. There is extra risk that you’re carrying even though you’re not making changes to the system.
Sam Taggart 00:47:49 Well I think that carries with my understanding of technical debt because if you wrote something 10 years ago and it was written perfect at that time and you look at it today, most people would consider that technical debt even though the code hasn’t changed since they wrote it then. So that tracks.
M. Scott Ford 00:48:05 Some things that we want to get better at and we want to find some good solutions for are being able to track the versions of the major tooling that’s being used. So the version of the compiler that’s being used to build a project or the version of the runtime that’s being used. There’s not always clear manifests for that information, but where we can get it, we’re going to start surfacing that as well. It’s not a traditional dependency that you might think of in terms of a package that you’re relying on, but most compilers or interpreters do ship with a standard library. You can think of that standard library being the dependency and sometimes there are vulnerabilities that do get exposed in those standard libraries.
Sam Taggart 00:48:49 Well that’s interesting because I do a lot of work in LabVIEW. We talk to a lot of hardware and sometimes the hardware drivers, there’s that LabVIEW package that talks to the hardware driver, but then the hardware driver has its own version and then sometimes even the hardware itself has its own firmware versions, which would probably be interesting and good to track as well.
M. Scott Ford 00:49:07 That use case of communicating with an external device or communicating with anything external, anything your team doesn’t control. I think a practice that I learned from working in aerospace is to build out some tests to qualify those dependencies. So some tests to qualify, yes, I can actually talk to the hardware and yes, I’m confident that I can talk to the hardware even if it’s the kind of test that requires a little bit of manual intervention such as it’s a test that runs and then ask you to look at a display on the device and then type in a value and then incorporate that value into the test suite. So it could be that the driver was commanded to display four on a display or to do something that would’ve had the number four show up somewhere on the device and then you then enter that value to kind of close the loop for the test.
Sam Taggart 00:49:59 That’s definitely something I do a lot of. Although I have to say a lot of that tends to be manual, sometimes I get some automated tests around those . Yeah, it’s definitely worth doing those types of things. So you do a lot of consulting. Is Freshli something that you use with all your clients and that you recommend that all your clients use? Is there a time when you wouldn’t recommend it or when it doesn’t add a lot of value?
M. Scott Ford 00:50:20 Not that we’ve noticed so far. Our big limiter on using it with all of our clients at this point is how many languages that we’ve built support for with the new architecture. So we don’t yet have support for JavaScript – that’s high on the list to add in the future. And then we want to add support for PHP and then once we have those two, then we’ll kind of have all of our current clients covered. We’ll have .net and Java and PHP and JavaScript and Ruby. Ruby’s the other one we need to add support for. But having all that built out will then allow us to support our current customers more. But this has kind of largely borne out of a repeated problem that we’ve seen over and over across clients and it’s something that we plan on doing more and more often at Corgibytes is taking repeated patterns that we’ve seen and trying to productize solutions to those problems and Freshli is really our first attempt at doing so.
Sam Taggart 00:51:15 I have a question then. You mentioned that one of the limiting factors is support for different languages. I’m a LabVIEW programmer. Is there a way, is the specification open enough that I could go write my own plugin to talk to LabVIEW or is that something you guys would have to do on your end?
M. Scott Ford 00:51:32 The CLI is an open source project and the language agents that we’ve written are open source. The only kind of closed source component is the website where you can view the data that’s been collected in the graphs and get that analysis and learning from that. But yeah, absolutely you could try and create a language agent for LabVIEW if you wanted. I would be really interested to talk to you about that if you’re interested in giving it a try. The documentation around that is pretty poor at this point, as it is with many open source projects. But that is something we do have on our roadmap to, to definitely improve, improve both the tooling around making it easier for people to create language agents and then tools for kind of qualifying whether or not the language agent provides the capabilities that that’s needed by the CLI.
Sam Taggart 00:52:19 Very nice. So one last question, that is, assuming one of our listeners is using one of the languages already supported, how do they get started?
M. Scott Ford 00:52:28 A couple different ways. If you go to Freshli.app, there are instructions for how to download the Docker container that you can use to run an analysis on your project. The version that we have in that Docker container is a little outdated, so if you happen to run into a bug, then following the instructions for how to build a project from source at our GitHub project for the Freshli CLI would be the next thing to tackle. And those languages that we have language agents built for are Java using Maven, we don’t have Gradle support yet. And then .net, we’ve got that support for .net for both csproj files that have package references in them, and then also the packages JSON format that was used for a little while by NuGet.
Sam Taggart 00:53:08 So all those language specific things are included in the Docker container?
M. Scott Ford 00:53:15 The Docker container that you’ll get if you do the download right now only has the Java language agent in it. We’re working on cleaning up the next version. It’ll be 0.6.0 and that Docker container will have both the Java language agent and the .net language agent installed inside of it. And that’s one of the reasons why we’re taking the approach of using Docker for the delivery is that following the steps to set up an environment correctly for all the different language agents can become a non-trivial task.
Sam Taggart 00:53:45 But if I wanted to, I could go to your GitHub and clone it and then build the Docker container from there and include my own LabVIEW.
M. Scott Ford 00:53:53 Absolutely.
Sam Taggart 00:53:55 Very cool. Well, thank you Scott. That was a great and enlightening conversation.
M. Scott Ford 00:53:59 Thanks. I appreciate it. It’s been fun.
Sam Taggart 00:54:02 For Software Engineering Radio, this has been Sam Taggart.
[End of Audio]
