SE Radio 505: Daniel Stenberg on 25 years with cURL

Daniel Stenberg, founder and lead developer of cURL and libcurl, discusses what it’s been like looking after them for the past 25 years. Host Gavin Henry spoke with Stenberg about the history of cURL, libcurl, whether C was the right choice, portability, key events in those 25 years, implementing protocols, why HTTP is not so simple, rust libs, the Polhem Prize, security issues, feature requests, random support requests, code on Mars, Apple OS adoption, cars stuck in production lines, Android OS, 8-week release cycles, release cycle joy, breakdown of bug types, 1000 committers, 250 cli options, user bases, determination, json, libSSH2, c-ares, HTTPbis, HTTP/2, QUIC, Mozilla, OpenSSL, WolfSSL, DNS, FTP, the cURL book, testing, CI/CD, favorite command line options that you might not know about, and making sure that you don’t give up on that idea or project you are working on.

Show Notes

Related Links

• Twitter – bagder
• Daniel Stenberg – https://daniel.haxx.se/
• About Daniel – https://daniel.haxx.se/about.html
• Daniel on WikiPedia – Daniel Stenberg
• Polhem Prize – Polhem Prize
• cURL project – curl
• cURL history – curl – History
• cURL book – Everything curl
• cURL man page – curl – How To Use
• Timeline – gource · curl/curl Wiki

Transcript

Transcript brought to you by IEEE Software magazine.
This transcript was automatically generated. To suggest improvements in the text, please contact [email protected] and include the episode number and URL.

Gavin Henry 00:00:17 Welcome to Software Engineering Radio. I’m your host, Gavin Henry, and today my guest is Daniel Stenberg. Daniel is a founder and lead developer of cURL and Libcurl, an internet protocol geek, an open source person, and a developer. He’s worked on HTTP implementations for over 25 years, has been active in the IETF for over a decade and worked on that HTTP stack and Firefox for several years. He currently works at wolfSSL. Robert, welcome to Software Engineering Radio. Is there anything I missed in your bio that you’d like to add?

Daniel Stenberg 00:00:50 Thank you and good to be here. No I think that’s sort of covers the basics, but possibly I could add I’ve done a lot of other Open Source stuff as well, but sort of cURL is my baby. That’s my primary focus.

Gavin Henry 00:01:03 Excellent. This sure is going to be quite different for me and for the podcast in general. We’re going to be talking about the cURL project, history, war stories, loads of different things so it’s going to be really exciting. We’re going to have a chat about five or six topics related to the cURL project for around 10 minutes each. Let’s start. Daniel, please take us through the 25-year journey, if 25 years is correct, you can bring me up on that, of cURL and 10 minutes or so, or do your best. Go.

Daniel Stenberg 00:01:38 Well, 25 years, it is. If we count the projects I did before I renamed it cURL. So basically my journey with cURL started sort of in the mid-90s, I work on another Open Source project with a friend, which was, it was an IRC bot. We call it a Dancer at the time. It doesn’t really matter, but in 1996 or so in the autumn there, I figured out that I wanted to provide a currency translation or exchange service for the bot for IRC users. So I started to look around on how to do that. And I realized that, of course, if you want to do a currency exchange, we need the currency rates sort of reasonably updated every now and then. I needed a little tool to download currency the rates using HTTP, because I found sites that hosted currency rates on HTTP.

Daniel Stenberg 00:02:28 So I looked around and I found a little tool called HTTP GET that would do the job for me. And so, I started working with that tool to do my currency exchange thing, and then I pretty immediately found some issues with that tool. So I corrected those and I sent patches back to the author who accepted it, and then it released for what releases for that tool HTTP GET. That first HTTP GET version I found and used was released in November, 1996. And I think Rafael, the author that too, got bored with me pretty quickly because I kept sending him patches for doing more things. So I became the maintainer of that tool within weeks, actually I think. I don’t remember exactly the timing there, but I was the maintainer of that tool within a few releases. I think I did my first release of that tool in the end of 1996.

Daniel Stenberg 00:03:27 So that is 25 years a little bit more than that. So, I worked on that tool HTTP GET for a while, until I realized I wanted to extend my currency exchange service with more rates. And I found another site that hosted currency rates on Gopher. So yay. I need more currency rates. I just need to make my tool support Gopher as well so that I could download Gopher too. HTTP and Gopher. So, I added Go for support to HTTP GET, and then HTTP GET became a pretty bad name because it didn’t only do HTTP. So it did HTTP and Gopher. I just changed the name to URL GET instead because it would get URLs. And then by that time, the tool would also work on URLs. That was one of the early changes I participated to make in the tool.

Daniel Stenberg 00:04:18 And then we kept it as the URL GET for a while; we released version 2 and version 3 and called it URL GET during 1997. And then I also found, I don’t remember exactly why, I think I found another site as well, that provided currency rates over FTP. So, I added FTP support. So now it could download data from FTP, Gopher and FTP. And in the early 1998, I started to add support for FTP uploads as well. And then again, I realized that while calling it URL GET, it doesn’t reflect the nature of the tools since now it doesn’t only do GETs anymore. It would do puts or uploads as well. I wanted to rename it again. So, I renamed it to cURL and we released the first cURL version in March 1998. And I kept the version numbering from the previous tool. So, URL GET version three became cURL version 4.0 there in March 1998. Then it could download from three protocols, upload to one protocol.

Gavin Henry 00:05:30 The does the C in cURL for C programming language?

Daniel Stenberg 00:05:34 No, I actually wanted to have a name and I’ve thought it would be fun with the name that has a URL in it because it works on URLs. So, I figured I wanted short maybe pronounceable name unique style. So maybe C, could be for client. I figured client for URLs and C could also work as a sort of if you pronounce it, SEE the URL as a sort of more of a pun-like thing. So, I figured why not? And I just want the primary goal was to have a short, short word so that you could type it easily in command lines. So, I went with cURL. I didn’t really spend a lot of time with the name. It was just, yeah, let’s go with cURL and I think it’s a pretty good name. So, at that time, then in 1998, by that time it started in 1996, it was slightly less than 300 lines of code in the first tool.

Daniel Stenberg 00:06:31 I don’t have the entire early history preserved. So I’ve sort of restored some of it, but the time I did the first cURL release, it was about 2,400 lines of code. And I think it had 25 command line options or so. And that was only a command line tool then. And we started working on that, or I kept on working on that. And then we got people training out, submitting patches, and extending it more and more. And the first major change from that point was in the summer of 2000 — summer here in my part of the world. I re-modeled the internals a little bit and provided a library. So, libcurl was born in 2000 so that we could provide an API and internet transfer capabilities, basically, to others — other applications or programming languages and so on — because I thought about it from the beginning and I thought it would be cool.

Daniel Stenberg 00:07:30 And at that time I’ve sort of made it happen. And when I did, one of the first that immediately adopted libcurl as a library was the PHP language, which I think was fortunate for us because they really had a lot of users. They still have a lot of users. They really tested it. They really got to submit a lot of bugs, and they had ideas how to do it. So, we got it tested and it took off really quickly from that point. Well, not like a rocket, but it sort of gradually increased poularity and people started to use it. And from that point we just kept on fixing bugs, adding things. We added more protocols support over time. We added TLS support already before we had the library so it supported HTTPS already back in, I believe 1999. And from that point on, we’ve just kept on adding support for things: features and a lot of different backends.

Daniel Stenberg 00:08:33 We pretty soon decided to support multiple implementations for different protocols. So, for example, we started with TLS support with the old SS — I don’t even know how they pronounce it, the precursor to open SSL: SSLE or whatever they pronounce it — and then we switched to open SSL, but pretty soon we also started to support other TLS libraries like GNU TLS, NSS, and a few of the others. And over time we’ve always worked on supporting a lot of different TLS libraries and over time, then we also have added support for multiple different libraries for other things like SSH or IDN and name resolving and stuff like that. So, we had that sort of infrastructure idea from early on to pretty much allow the user who’s building cURL to decide what kind of third-party libraries they want to use when they build cURL.

Gavin Henry 00:09:28 Thanks. Yeah. I’ve seen the options when you go to install libcurl or cURL; it gives you different versions of TLS libraries if you’re installing through Debian packet monitor or Ubuntu or something. Excellent. That’s a good bit of history. Best you can do in 10 minutes. Were you quite an accomplished C programmer before you started in ‘96, just before I finish off this section, move us on?

Daniel Stenberg 00:09:54 Yes, I had been working — I mean, I’m a software developer since, of course, since before that. I had been working professionally with C programming for several years before that, so I was pretty comfortable with writing programs in C, yes.

Gavin Henry 00:10:09 Excellent. So now we’ve had that good history lesson. Can you think of a couple of things for the next 10 minutes that you learned over that time that surprised you, or might surprise others, in those 25 years?

Daniel Stenberg 00:10:25 I’m not sure I have learned much sort of big surprises. I think I learned all those things that most people would learn doing something like this for a long time. For example, just learning how to write something that is actually maintainable over time. For example, clear code, comments in the code, explaining things to my future self, and stuff like that. And the value of doing test cases, and documenting things, and just having sensible hygiene in the project, nothing of that is surprising or strange in any way, but it’s when you work in something for a long time, I think more of those things actually become important because you get to sort of discover things about your own code and thinking down the road, because you have to, when you live with it for such a long time.

Gavin Henry 00:11:22 Maybe what about a protocol that you implemented that took much longer than you ever expected and that surprised you?

Daniel Stenberg 00:11:29 Oh yeah but I think in general, I mean, HTTP is my primary protocol. I think that’s the protocol cURL is most known for, most used for, and that one I spent most time on. And I think HTTP is one of those protocols that, yeah, it seems so simple. I remember when I started working on HTTP, it seemed so simple to implement you know: just text and just type GET and it’ll GET that. And over time you really realize that HTTP — yes, it seems so easy on the outside and on the surface, when you see that text. And of course, over the years, in the recent 10 years, we’ve switched away from the text-based as well, but it was never an easy protocol and it’s getting more and more complicated over time. Implementing something in HTTP today, it’s really complicated — in particular, if you want to support multiple versions. So yeah, I think basically all protocols that are well-used have turned out to be much more complicated in real life and in the real world than I, for sure, sort of foresee from the beginning. And I mean, none of them are ever done, right? Because we keep getting bug reports today on stuff we wrote and implemented decades ago. Things are never done. It’s doing things. Internet protocols, networking across the internet is challenging.

Gavin Henry 00:12:52 And have you been surprised on protocols that have come and gone or libraries that you use or things you’ve implemented that have outlasted, how long you think you’d need to support them or you’ve had to drop stuff over that time?

Daniel Stenberg 00:13:06 My primary view of things is that I don’t really foresee, I don’t make any projections or, or try to tell how the world will look in the future. I’m looking at where we are right now. And I’m trying to adapt to that and maybe where we’re going this year or this a few months ahead. I never tried to actually tell what we will do in the next 2, 5, 10 years at that, because I find it impossible to do that. But sure in general, things stick around much longer than you ever think when it shows up. So of course, for example, introducing new protocol version, something we know that the old protocol versions, they will stick around for a very, very long time, even when something new, better, shinier comes along. And in cURL we have this concept that we don’t modify within break API.

Daniel Stenberg 00:13:57 So API, we stick around, we support everything we provided in the past as well. I’m not sure I’m surprised that it’s more of how the world works. And of course, it’s really hard to say, especially when you use a lot of third party libraries, it’s hard to say, sure, we can add support for a new third party library today, but we can’t tell how that third-party library will be maintained, survive or act tomorrow, right? Or in two years or five years or 12 years, who knows where they are going. So over the years, of course, we realized that some, for example, TLS libraries that we added support for in the past, they mostly maybe died over the years and then we eventually rip out support for that particular library or subsystem or stuff like that.

Gavin Henry 00:14:44 And the infamous question, I’m sure people always ask us, are you happy of the choice of the C program in languages or language for cURL and libcurl?

Daniel Stenberg 00:14:55 In general I would say that I’m very happy and that is based on several things really. Because first of all we started, as I mentioned before, we started this in the nineties and in the nineties making a portable library or portable tool portable anything, there was no real choice other than C. I mean, C++ could possibly be in a choice, but not even C++ had a stable ABI back in the day. You couldn’t really do any portable libraries back then with C++, and I’ve never been a C++ fan. So I avoid C++. So yes, I’m happy with C and a C has made it possible to really make cURL and Libcurl deportable available everywhere library that it is, it is C that is the explanation why it is used and can be used in so many, many different places, operating systems, CPU, architectures, and everything. I would say it isn’t until very recent years that there actually have started to appear viable alternatives that could have been used, but they can be used now. They could not be used 20 years ago. One of the benefits, one of the things with cURL is that we have the age, we have the maturity we have been around for so long. So it has had the time to mature and stabilize and everything. And that’s very big thing too.

Gavin Henry 00:16:19 Yeah. It’s not something that you just want to start again and a new language that’s come up.

Daniel Stenberg 00:16:23 No, exactly. Because whatever you do, it takes a really long time to become a really stable and solid thing to do like this. And I think that’s one of the primary benefits you have when you go with cURL, you have to get all this battle proven time and have been shaped by nature for so long. And it’s, that’s hard to replicate or, I mean, you can replicate it. It’ll just take a long time.

Gavin Henry 00:16:48 Well, that brings us nicely onto the next section, which I’ve called key events in the timeline. So, I really like the history and timeline document that you have on GitHub and what I saw on the mail list, it’s very complete. Could you pick two or three of your favorite things from the timeline filed? You shared, I think it was December or last month, or maybe talk about things you wish you could delete on that list.

Daniel Stenberg 00:17:17 Well, there’s so much,

Gavin Henry 00:17:19 I’ve got a couple of my list then you can agree or disagree. So, I’m thinking when cURL was on Mars, when Apple included at macOS, your favorite protocol, when the user base reaches a certain amount, the number of bugs, when you got your first CVE security thing, any of those?

Daniel Stenberg 00:17:39 Yeah, there was a good event. So of course, in the beginning, when cURL started, of course as anything that is started as a small project, when people like it, start using and adopted in different surroundings, those are key events. And those were really fun to mark of course. When Apple included it in macOS in 2001, in September 2001, that was a really a key event for me because it’s so, that was one of the first non-Linux operating systems that actually adopted cURL as a standard tool in their operating systems. So that marked something sort of a notch, a sign of success. So, I have that marked, and I thought that was really great moment in time. And of course as you mentioned, it was confirmed to be used in the Mars helicopter mission in 2021. And that was a really fun moment.

Daniel Stenberg 00:18:31 Of course, it really excellent ego boost. And one of the things we’ve talked back and forth in the cURL project for a long time is to get any kind of confirmation that cURL has been used in space? Because we’ve had that people have mentioned that in the past that it might’ve been used on the ISS and stuff like that. But I’ve never had it confirmed from anyone or had any proof. And then finally, when we got the proof that they actually used it in the Mars mission, that was such a cool moment to say that, yes, finally, one more planet than just earth.

Gavin Henry 00:19:04 So, is that in something that was doing requests, but then an operating system on Mars or coming back to the base?

Daniel Stenberg 00:19:11 They won’t tell. So, it’s really impossible to say. I have no idea. They’ve just said that they’ve used it in the helicopter mission.

Gavin Henry 00:19:19 I wonder what the latency is.

Daniel Stenberg 00:19:22 Yeah, I imagine it can’t really be done from Mars to earth using cURL. I would imagine it must be something shorter distance, but I really couldn’t tell. And they won’t tell so we can just speculate on whatever it is. For me of course, one key moment in time is when I got the Polhem Award prize in Sweden in 2017. I actually got a gold medal from, which is an engineering award here in Sweden. It’s really an old one, its over a hundred years old, sort of handed out to engineers in Sweden who have sort of accomplished something, blah blah. But it was a good moment in time for me. And I got that award handed over to me by the Swedish king at the great gala dinner here in Stockholm. That was awesome.

Gavin Henry 00:20:07 Wow, congratulations. And the user base figures or bugs or security issues or was there a point on that timeline where you thought, what have I created?

Daniel Stenberg 00:20:19 There haven’t been some times when people have said something that have made me realize that, wow, the number of users is a really high number now. I remember counting at some point in time and I realized it might be several hundred million installations now. That’s crazy! And nowadays we count somewhere maybe more than 10 billion installations. So you get a little to the numbers because there’s so immensely big now it’s, it’s hard to even imagined. But of course, I remember stuff like when I realized that it was used in for example, wow, it’s installed in most Android installations. And when I also realized, and it’s used by default iOS, then I also realize that, wow, it’s used in quite a lot of places. And I have these fun email interactions when I got that email from, from a woman, I think this was in 2016 or so I got an email from the woman who, well, she was confused, but she wanted my help to fix her Instagram account because apparently I know the Instagram people because she found my name in Instagram. And that was one of the moments when I realized, wow, they’re using my code in the Instagram app on iOS as well. Those particular moments could be a little bit of eye opening that it is used in a lot of those big volume apps.

Gavin Henry 00:21:43 Yeah. It’s kind of mind blowing, isn’t it? When you just think about everything, not document. So yeah. I noticed that you keep track of the security releases as well. Are they different things or is that programming patterns that keep appearing, or how would you classify those types of things?

Daniel Stenberg 00:22:01 I tried to keep very close track of exactly all the security problems that we have had reported on cURL. And we have this bag bound where we reward the security researchers who file or submit issues that are confirmed to be security problems. And then I try to make it a really good effort. I pride myself to actually go into the details and then research it exactly when we insert the problem when we fix it and try to figure out exactly how it happened and how you can explore that and try to document all of that. And part of the reason for doing that, except for then later, being able to do fun graphs and when bugs were inserted or fixed, is that is also a good way to try to learn something from the problem. It was inserted at this point, this is the mistake, we fixed it like this, but what could we have done or what should we do now so that we don’t do this same kind of what even the, exactly the same mistake once more? That’s really hard because it’s like a normal bug, right?

Daniel Stenberg 00:23:05 Once you read it and once you have that report, you can oh, you realize that, yes, that’s a silly mistake. Why did we ever do it like that? Of course, it’s stupid, but it wasn’t stupid. Or at least we didn’t realize the stupidity at the time when we inserted it. So, what do you learn from that? So, it’s typically very hard to actually not view it as a one-off mistake and sort of everyone makes mistakes. We can’t fix that. But then we also try, I have tried to do certain things in the code, like avoiding certain styles of programming patterns. For example, one of the things I realized actually, that we had several security problems that were the result of silly integer overflows and reallocs or mallocs based on that potential into your overflow. And I’ve actually done two things in the project to reduce the likeliness of that ever happening again; one of the things is that we have nowadays a pretty much universal limit on string lengths of string data inputs you can send to libcurl.

Daniel Stenberg 00:24:13 Which limits string size to, I think it’s eight megabytes, which is a ridiculously high limit, but it avoids the chance that someone can put in the string that is next to two gigabytes on a 32-bit architecture, for example, or stuff like that. And we also have introduced a new sort of internal API and buffer system to try to make it reduce the number of reallocs done within the C code. Because I realized that we had several of those security problems in close association to reallocs and reallocs to growing buffers, growing memory buffers. I’m trying to avoid stuff like that. So hopefully avoid some of the mistakes we’ve done in the past. Other things we’re doing that we recently or we started in late 2020. I now worked with the ISRG who has sponsored a project to support replacing the built in HTTP back and the HTTP coding, but not all of it, but part of it with an HTTP library written in Rust called Hyper. That of course, another way to potentially address or avoid future mistakes, at least see mistakes by making sure that we use less C and more other languages than C.

Gavin Henry 00:25:40 That’s a good point to move on to the next section. So war stories, I’m calling this. I’d like you to now talk about some of the hard bugs you squashed or other memorable stories during the project life if that’s okay? What stands out for you and makes you think if I did that, I can do anything? Or we could drill into some of those security issues a bit more because I like the sound of what you just explained, what you’re doing with that HTTP library layer. So yeah, if I did that, I can do anything. Is there anything that comes up?

Daniel Stenberg 00:26:11 Not really or rather there’s so much of that I think. Bagging wise I think doing things, there are so many layers of code I think. In cURL itself there’s a lot of layers and people, applications, and there are languages. And I think in general, we have things like languages doing bindings, doing libcurl, who’s doing things. And then someone writes an application in that language using the binding using libcurl was doing TLS, doing a protocol where something is wrong when you’re using a third-party library. So, I figure sometimes it’s really, really difficult to understand, or just figure out where the problem might be, or there’s so many layers, so many different responsibilities, so many different angles it could be. So, I think sometimes we’d really dig around for a very long time and a lot of code to figure out where it is.

Daniel Stenberg 00:27:11 So I think it’s common pattern. One of my favorite ones. I think I have a quote somewhere when Facebook reported a problem with cURL Facebook, I think they still use, they have a PHP version. A lot of Facebook is written in, so they use libcurl from PHP. And while they experienced some kind of lag that took, I don’t remember exactly. I think it was some delay with some milliseconds in some kind of request. And I got it. I have saved their response quote because the person I worked with or communicated with then sent me an email and said, I tested your patch in production. And it works. And I figured that was fun just because testing my patch in production on Facebook that’s seemed like it was a few years ago, whatever was still, hundreds of hundreds of millions of users. And that was fun. Another fun little bug. I remember that sort of stands out among other bug fixes is that I was contacted by a company in Germany who was doing software for some car company and the person who contacted me said that, “we have 8 million cars waiting for a firmware upgrade here, but we can’t deliver that because cURL is crashing.”

Daniel Stenberg 00:28:36 And that was back in the day when I didn’t even work on cURL. So I was just, you know what okay, thank you for telling me that. But you know cURL is a spare time project here, so I don’t know what you expect me to do here. His next then follow-up question was, “Can you fly down here tomorrow and help us fix this?” I tried to explain to him, no, you know I have this full-time work. And I’m expected to deliver something this week and I can’t just take off in the middle of a week to go to Germany to fix your thing. I managed to find a friend who could fly down there and I could help them from remote. So, we fixed it within a day or two. So that was fun. But yeah, there’s been a few of those adventurous bug fixes over the years.

Gavin Henry 00:29:18 Yeah. What was the one the other day I saw, maybe it was on Twitter; I think it was to do with the Log4j exploit, wasn’t it?

Daniel Stenberg 00:29:27 That’s the biggest story. So, since cURL, I don’t know exactly why, but we modified the MIT license slightly when we adopted the MIT license back in 2001; I think we switched cURL to MIT license. So it’s slightly modified from the MIT language; it’s just a few words that are not the same. It’s basically MIT. But anyway, in that license file, this has copyright blah, blah, blah, Daniel Stenberg, blah, blah and my email address. And that particular license file is usually included in different operating systems, or products or devices, and about screens, on a lot of places, partly because it’s not an MIT straight off. So it’s usually recognized as the cURL license and other regular. So, when people bundle a lot of licenses, it’s still stands out because it’s not among the regular MIT ones, it’s separate, its current one.

Daniel Stenberg 00:30:22 And it also usually then ends up as one of the few licenses that actually has a personal email address in them. When people ship products or devices and stuff, and they put together a bunch of licenses, hundreds of licenses isn’t that uncommon, people eventually, or some people eventually find my name and email in there. And they email me about whatever problem they have that is associated that they have with their device or tool or car or printer or anything. Computer games is pretty common, too. So, people have problems with things they look around. And usually, I guess they are actually pretty upset with something and they are frantically searching for someone to contact. I guess in many cases, they already tried to contact 22 different people. And then finally they find my email somewhere in there. And then I’m going to email this guy and he’s going to help me with my issue, whatever the issue is.

Daniel Stenberg 00:31:22 So I get a lot of fun emails from people who want help with issues with their software, where I usually don’t even know what they’re talking about. And recently I got an email from a big company there. They actually called, I didn’t say that in the blog post, but they’re actually MetLife. MetLife is a really big insurance company and they’re, I think their own fortune 100. And they emailed me a lot of questions about how to make sure that their products aren’t vulnerable for the log4J vulnerability. And they called me a partner in the email, I guess they found my, my address in some kind of like that scanning a lot of licenses in their products or something. And of course, for me it just turned out really confusing because I don’t do any java anywhere and I’ve never participated in any Java products anywhere.

Daniel Stenberg 00:32:17 So, of course, nothing that I ever wrote has any log4J in it. So, the question was mostly confused, but then as I said, I’m kind of used to getting those kinds of questions because I think almost the same day I got that log4j question, I got another question from someone who he was upset about the player choices he got when playing some soccer game. I don’t even remember the name of it, but that guy asked me to help him get better players. And then he sent me also a screenshot that showed my name in the license window of the computer game.

Gavin Henry 00:32:54 You have to drill pretty deep to get the About page in most apps. There’s some user interface failure if I have to go to the About page and drill into licenses to find the contact.

Daniel Stenberg 00:33:09 Not only user interface failure, I think there’s also a general feedback customer relation problem, but it was also had a lot of car problems mailed to me and finding my name in a car infotainment system that is also it takes a lot of will, patience to find it. It says something about how hard it is for regular people to actually get in contact with someone who did the software for their devices.

Gavin Henry 00:33:39 Just before we move on to the next section, it sounded really interesting what you mentioned about bringing Rust in as a library. Will that mean that you’ve then got another library to maintain that’s part of the library, or how will that work?

Daniel Stenberg 00:33:56 Pretty much. Yes. Basically already, when you build cURL today or libcurly also, we use third-party libraries for certain things that we don’t do ourselves — like handling TLS, SSH, different compressions, and stuff like that. We are already leaning on other libraries for doing part of the functionality. So, when you build libcurl and ship it with your thing, you already use libcurl and a number of other libraries. When now we’re enabling or making it possible to build libcurl to use different Rust libraries, you’re only maybe adding libraries or replacing libraries so you go with the Rust ones instead of other ones, but yes, you’re certainly going to add the dependency and rely on other libraries as well as on top of libcurl then.

Gavin Henry 00:34:49 So that means the core HTTP functionality will be moved away from C and into Rust as a separate library that way?

Daniel Stenberg 00:34:57 Yes, but I’m doing it the same way as I do with all the different TLS libraries pretty much. I still have a native implementation in C that you can replace at build time. So, you can go, you go either with a C solution, the native one, or you go with the one in Hyper, the Rust one. So, at build time, you select which one to go, because I’m a firm believer that I need to keep and maintain the C version because as I sort of talked about half an hour ago, the C version is what makes cURL as portable and as popular in, in so many places. So, I think the C version is going to still stick around and be available and be used by – I’m not sure if “most” people but a lot of people going forward as well. And we’ll see how the Rust solutions go. I mean, if they will become popular and used and so on, I haven’t really no way of telling or no idea how they will fare going forward. Hopefully they will be popular and used, but I really can’t tell if they will be.

Gavin Henry 00:36:00 Yeah, going over the timeline of what the history of cURL that’s a long time. So, you’ll just have to see, I suppose? Just to close off this section, you mentioned the Rust bit to help potentially with some security issues, do a lot of the security issues. Are they particular to programming parts and C or nothing to do with C or a combination or something in how the protocols implemented that’s being missed?

Daniel Stenberg 00:36:27 I’ve tried to count the number of obvious mistakes that are due to the programming language C and I think it’s about half. I think we’re going slightly below half now, but somewhere in the neighborhood of 50% of the problems have been C mistakes. So, if we would imagine that entire cURL would have been written in a memory-safe language, maybe we could have avoided 50% of them. But that said, we also do things differently now. So, I’m not convinced that we’re going to see 50% of them being C mistakes in the future, but it’s hard to say.

Gavin Henry 00:37:04 Excellent. Thank you. So the next section I’d like to talk about release cycles and feature request process. Can you tell us about your release cycle or feature request process, for example, how do we request features? How would you assess their suitability? And what made me think of this as the other day, you Tweeted about release and the sense of relief that comes out of that. And then an hour later a bug report comes in and you’re like “Arrrgh!”

Daniel Stenberg 00:37:34 Yeah. That’s part of the regular release cycle. Yes. So, I’ve always been a, been a believer of the traditional Open Source mantra to release early and release often. And nowadays people do that even more since a lot of software these days already are server-based or cloud-based. But anyway, I’ve always tried to do a lot of releases so that people can get the opportunity to have the latest code often. So, if we fix anything, they don’t have to wait around for a long time until they, again, the next release. So pretty much we started out early on to do very frequent releases. And after a while, maybe a decade — I don’t remember exactly when we switched to it — I think it was like 15 years ago or maybe something like that. We switched to a completely time-based release cycles. So, we pretty much just set the clock and we stick to that cycle.

Daniel Stenberg 00:38:31 So, we do releases every eight weeks if nothing else happens. So, we stick to that and we have the first half of that release cycle open for merging features and doing changes as we call them things that are actually potentially adding features of changing things. And then the second half of that release cycle, we don’t accept new changes or features. We just fix bugs. Then we do a release and then we start over, pretty much. I think it has turned out to be pretty successful because it limits the speed in which we allow features. And it also, it makes us have a pretty long time where we only work on bug fixes, which has turned out to be, I think, pretty good because it makes people work a lot on bug fixes. And I think bug fixes are the most important things we can do.

Daniel Stenberg 00:39:30 And we stick to this whenever we find something really critical buggy within the release cycle, we can make an exception and make another release without eight weeks having sort of being using that as a cycle. And we do that every now and then when we find some terrible bugs that we inserted, but the ideal case is eight weeks then release. And usually we don’t even do emergency releases for security fixes either because they’re rarely that critical. So we usually bundle the security fixes too, and include them in the release at that particular release point in time. And having eight weeks like on the clock, it makes it also very easy to plan everything because we know ahead of time exactly the dates of all the future releases, as long as we just keep the release cycle. We know when we go to the feature freeze, we know when the release is going to happen, and so on.

Daniel Stenberg 00:40:25 So it’s also an easy scheduling thing for me, I think. And for the relief, I think it’s when we work on something for eight weeks and we package everything and we put it together and upload it to the site that we can blank out the change log and say, wow, we start out on a blank sheet. Now everything is released, everything is fine, this feels great. That’s an awesome feeling to just ship it then. Ahhh, that’s it. I so enjoy that moment when everything feels fresh and new and everyone can upgrade to the latest and greatest; that moment is awesome. And as you said, pretty much until someone reports a bug in the new version as well, or a new one or something bad, or anyway, it’s still a great feeling. And when we have done a new release, we always do releases on Wednesdays.

Daniel Stenberg 00:41:19 So we do releases on Wednesdays and then another one, eight weeks later. So when we have done a release on a Wednesday, we wait until the following Monday to open the feature window again, but pretty much to give it a few days for anyone to report alarming bugs, because if there’s an alarming bug, we don’t open the feature window and we work on emergency fixes that, and maybe we do another release the next week or so. But if we open the feature window again, after that release, we pretty much allow features to get merged. And then of course becomes the question, follows the question where, what features do we merge when we have the feature open? And it’s a bit of a random thing. It’s pretty much what people are providing in poll requests that are in good shape, mature and we agree that it’s good change.

Daniel Stenberg 00:42:14 And combined with someone who is actually also able to review it and accept it and work with them or third, to make sure that it gets up to snuff and being good enough to merge. I usually myself have a few things that I sort of keep working on that I want to have much myself. So, I try to make sure that I have pull requests ready or in time because I, of course also sort of submit to the same rules. I’m only emerging changes when the feature window is open. That’s the only time I can merge features as well. So I, and of course I have a slightly easier chance to get my stuff merged because I know better than most, exactly how to do it and how to do everything correctly and have it accepted by everyone. But otherwise it’s a discussion. I usually allow anyone to provide whatever. And as long as you can motivate it, then discuss or argue for your sake and for your features, we discuss it and we work with it and we make sure that we have some kind of rough consensus and then go forward with that.

Gavin Henry 00:43:26 Is it usually a case where it support requests through get help and they’ve done the feature and they just want to see it be part of the library or the cURL project, or did they request that you guys could put into your schedule to do?

Daniel Stenberg 00:43:39 I think we have everything from both ways where it was sort of where and everything there in between. Sometimes someone shows up with a huge pull request that says, I already done this. We’ve used it for two years. Here’s the pull request. And sometimes it’s just people nagging and say, why don’t you ever do this feature? We really need it. Or something like that. And we have everything there and in between. Of course, the best thing is when someone is actually working with us, the best thing is when people don’t come there and submit the really big one. The first thing we hear about it is when they submit a few thousand lines of diffs, because maybe they did it in a way we don’t quite agree with. Maybe they did it in a way we could have done better to take advantage of whatever.

Daniel Stenberg 00:44:28 So it’s better to get that communication started early and see if we want to do this. What’s the best way to do it and then work with the team to get it done. But I, wasn’t getting a lot of good ideas from people who anyone who is randomly using cURL that says, oh, I thought of a good idea. Maybe it should do this. And of course, good ideas need to be provided first before we can do anything like that. Right. So, a good idea. It’s a good idea. Even if sometimes good idea is also it’s a little bit too easy to just submit the good idea, because an idea is easy also, but maybe they’re actually implementing the idea is not always as easy. And in addition to that, I work on cURL full-time, I work for wolfSSL and this works because I sell cURL support. So, someone is paying me to help them with use cURL or help them do cURL correctly in their applications and devices. And part of that, they can also pay me to help them get features done in cURL that they want. And of course, that has to be featured that I accept and want into the project as well. So sometimes people are actually paying, or I do work as part of my paid contracts to land features as well.

Gavin Henry 00:45:49 And have you ever had to say no, that doesn’t work? We don’t want your money or?

Daniel Stenberg 00:45:54 Yes, but usually it doesn’t really work. It’s rarely they say something and I say, I blankly say no. If they say I want this, and I think it’s a bad match for cURL, maybe we don’t have to do exactly like you asked. Maybe we could do this part in cURL and you should do that part in your application and we could work it through. So, it’s rarely a yes or no situation. It’s more of a gray area where we can discuss exactly what should cURL do, what should your application do, what shouldn’t? So, it’s more of a matter of discussing and debating. Oftentimes when I talk to people actually pay for this and it goes with whoever submits a pull request too right? Sometimes people send me a lot of things that maybe you were asking cURL or libcurl to do a little bit too much. Maybe you should take away a little bit and do that yourself outside of cURL instead. Or maybe this is completely out of that course you shouldn’t do? But having worked with the project for so long, we have to make a really big effort to limit the number of features and limit the growth of just scaling everywhere. We can’t do that because we have to make sure that we stick to the concept here and not just branch off in every imaginable direction.

Gavin Henry 00:47:13 So if a listener, have to an idea or an improvement or something, how would you recommend they reach out?

Daniel Stenberg 00:47:20 Generally the best way to discuss anything is on the mailing lists. We are now Open Source project. We use mailing lists. That’s the best way to discuss ideas. If you’re just having an idea, if maybe you have an embryo or a start of some coach, actually you started to do something a change or reading a concept root than a concept, then maybe you could submit a pull request. Here’s my first shot, take a look at this. Would this be acceptable to you? And then work with us, maybe within that pull request, this is a good base. Maybe you should do it like this. Instead, maybe this contradicts what we’re doing here. We should remodel and do it like that. And so on and just be prepared to work with us and do a little bit and back and forth, and then go forward.

Daniel Stenberg 00:48:05 Usually I also tried to make sure that if you really want to see something happen, make sure that you also stick around for the follow-up discussion because don’t just sort of code at us and go away and come back in two weeks. Because if you do, you’ll find those questions or follow-up questions that were filed 30 minutes after your pull request was made, and then it’s been dead silent for two weeks. If you really want to make something happen, be there and make sure that you follow code style and you’ve made sure that everything works. You have test cases, you have your document and new features and stuff like that, and just make sure that everything is in shape. Then I would say it, isn’t hard to do anything, to do changes in cURL as long as you just do things correctly and you have some patience and stick around.

Gavin Henry 00:48:55 Thank you. Well, that brings us into the last section of the show. A strange one, but if you are starting cURL again today, would you, we did do all the same or hindsight’s a wonderful thing. And we should maybe indulge at some point in our lives.

Daniel Stenberg 00:49:14 Yeah, I would imagine that if I hadn’t started it, I feel like something I wouldn’t start now, but if I hadn’t done cURL or libcurl, someone else would have done it and then there would exist something else that would be similar to cURL. I mean, as you described me from the beginning, I like internet transfer, internet protocols. That’s sort of what I’m intrigued by that I’m fascinated. I think that’s fun. And I, I mean, I participate in a few different Open-Source projects and I do some other things. And so apart from cURL, for example, the biggest ones that I maintain as well is LibSSH2 and CA risk projects. They do SSH and DNS stuff. So that’s sort of the area I’m interested in internet protocols, internet transfers. So, if I hadn’t done cURL, if I didn’t work so much on cURL today, I would probably sort of still be nosing around and digging around in network related libraries, network related code. So maybe not cURL specifically, but I would have done something internet-ish at least.

Gavin Henry 00:50:19 And what advice can you give after your hard-earned experience for other Open Source project founders or those that wish to help with a project like cURL?

Daniel Stenberg 00:50:29 Well, for other maintainers, I don’t know. I don’t want to say the others should do what I haven’t done, or I don’t think I’ve done anything magically strange or wonderful in the current project. I try to lead by example. I try to listen in what other people say. I try to make sure that others can do as much as possible so that I don’t have to do things, make sure that we can widen the number of developers and everyone can do things individually and independently so that we don’t introduce unnecessary bottlenecks in the project. I’m not sure I’ve succeeded in that, very good. But that’s what I’m trying to do. And we were open for discussions and ideas and suggestions and stuff like that. But I think all of these is just how to, how any Open Source maintainer would think and consider Open Source.

Daniel Stenberg 00:51:22 Working on Open Source it’s a lot of working with people. Of course, you just have to realize that there’s a lot of different people and you need to understand that people are different, there’s many different cultures. You have to have a fixed game and manage people somehow. That’s really hard. And usually when I try to give advice to anyone who wants to participate in a project or join in and do something here with us, I try to get people to work with something that you think is fun or that concerns you. Maybe you have an itch to scratch. Maybe you have a use case. You, you haven’t found fulfilled, or you found a typo you want to fix or something that actually concerns you is because it’s much more fun to work on something that affects you personally. So maybe that little feature you’re missing or that little thing that doesn’t work the way you want it, get to that, to fix that, work on that. And that doesn’t really matter. I mean, that’s certainly not a cURL recommendation. That’s whatever you want to do in Open Source. It’s much better if you start with something that is near to your heart. Otherwise I’m not a guy to give advice. I feel more like a lottery winner. Do you have any advice on what lottery numbers to pick? I don’t, it was fortunate for me. I’m not sure I’m the one to tell anyone not to repeat it.

Gavin Henry 00:52:47 Well, I think we’ll accept that, but I feel you’re downplaying your role dramatically.

Daniel Stenberg 00:52:52 Well, maybe, but it’s really hard for me to say what works for me and what doesn’t work for me. I’m trying to run and be in the project the way I would have appreciated someone else to do it. If I was a participant in the project, sort of.

Gavin Henry 00:53:07 Yeah. That comes across. I mean, your personal applies to emails and things and how you can induct stuff is a really good example. What are some of the things that most users don’t know about maintaining like a project, like cURL we’ve talked about the support request again, or is there anything else that goes on behind the scenes that is not normal for Open Source projects?

Daniel Stenberg 00:53:31 If you’re in an Open Source maintainer for a smaller project, because I think cURL is still a smaller project, it might be well used and popular and known, but it’s still a smaller project in that. I’m the only one working on it full time. So I think what a lot of people may not, if you’re an Open Source maintainer this, but people from the outside if people are working with other things, don’t realize how much other things than working on code you have to do when you’re maintaining a project. Maintaining the servers, maintaining the mailing lists, doing releases, setting up your scripts to update things, to do it’s the CI jobs, the everything else around the project that needs to be maintained for it to run smoothly. I think a lot of people are sort of missing that huge amount of work that you have to keep up in a project to just keep everything afloat and going forward smoothly.

Daniel Stenberg 00:54:31 So I think it’s sometimes I spend a lot of time on stuff like that just polishing things around the project to make sure that it goes forward well, but that work isn’t visible at all because when everything works you don’t see what work that went on to make sure that nothing broke. The other day, for example, in a weak moment, I upgraded a little detail in my server and the server that runs all the mailing lists. And in that little moment of bad decisions, I accidentally upgraded my Python installation on the server to no longer feature Python two. And then in one blow, I just broke a lot of server infrastructure. So that mailing lists and I run a lot of mailing lists. They all broke in one moment, sort of, and that I had to spend several days restoring Python to installation so that the mailing list could work again. And of course, from the outside, it wasn’t real. Okay. The breakage was possibly visible for a select few who try to use the emailing list.

Gavin Henry 00:55:35 I did actually see that,

Daniel Stenberg 00:55:38 But it was still a lot of work just to bring up everything back to look exactly like it did before. And for me, I bet I spent 12 hours on that or maybe more, and it was a really annoying and tough time here, but yeah from the outside, I didn’t do anything on cURL. Everything was just looking the same way as it did before

Gavin Henry 00:55:59 It wasn’t on a Friday evening after a glass of wine, was it?

Daniel Stenberg 00:56:03 I think it was actually worse, but it wasn’t Friday evening, but it was still it wasn’t one of those decisions I did without even considering. And then afterwards, like, oh, what did I do? Oops, this was not good. Then I had to suffer through it.

Gavin Henry 00:56:20 Yeah. So that’s a very good example. Right? I’m going to close off in a sec and then start wrapping up. But I know one of the statistics that you like to talk about is a number of command line arguments that you can do. I think it’s 750 or something. What are some of the weird, bizarre, and new one’s that you would like to let somebody know about?

Daniel Stenberg 00:56:40 I actually added the 245th the other day.

Gavin Henry 00:56:44 Oh, wow.

Daniel Stenberg 00:56:44 So we have 245 and the most recent one is what is not in a release yet, but it’s dash dash Json.

Gavin Henry 00:56:52 Yeah. There’s been a bit of noise about that one.

Daniel Stenberg 00:56:54 Yeah, exactly. There there’s been a bit positive and negatives about it, but this is a very simple one. I’ve added it to make it simpler for people to send and receive Json. And I think a lot of people have been pretty positive about it. So that’ll be fun. Now. I think some of my favorite ones that might not be that well used always one of my absolute favorite ones is the dash dash libcurl, which is a command line option that converts the command line to a libcurl code or generates a template code in C for the command line, you wrote. Basically if you write a command line using cURL, did you do some kind of transfer, upload, download, whatever? And then you say, ah, I want to convert this into an application instead that use the libcurl. You run the same command line and you do dash dash Libcurl example.C, and then it’ll generate that example.C for you. With a skeleton code that uses slid code to do that exact same.

Gavin Henry 00:57:56 I really wish I knew about that one. I just did that the other day. It goes on the mailing list, but I’ll do that and check things out.

Daniel Stenberg 00:58:05 Yeah, I think it’s really cool. It’s not complete, of course, because it’s hard to convert all of that into C code, exactly? But you get a pretty good start to base your further work on whatever you want to do when you want to do a libcurl application. And what is good is that most bindings for libcurl are actually rather thin. Most bindings for libcurl have the same options and stuff like that. You can usually fairly easily even convert that C code into, for example, PHP code or Python code or other binding steroids, because they usually look fairly similar to libcurl itself. That’s one of my favorites. Another one that I like to point out to people is the dash dash resolve function, which is a way to pretty much populate the DNS cache from the command line. So, you can add an IP address for a host name on the command line, which is a way basically what you want to do is if you, for example, if you type, cURL example.com, but you’ve decided to host that example.com on your local machine, for example, on local host. And then you get a problem with the names because the certificate maybe won’t match and stuff like that. So, then you have an option for cURL that you can say that in this invocation example.com is going to use this particular IP address instead.

Gavin Henry 00:59:33 That’s brilliant. Because that’s normally somewhat difficult to do when you’re running CI jobs or editing, et cetera, host and everything like that?

Daniel Stenberg 00:59:41 Exactly. Or when you were experimenting or you want to send in a particular name on that particular IP address and stuff like that.

Gavin Henry 00:59:49 It was resolver?

Daniel Stenberg 00:59:50 Resolved.

Gavin Henry 00:59:51 Resolve, right. Okay, excellent. So I’m going to wrap up now. Obviously cURL’s a very powerful tool, with a strong history and global deployment base. If there was one thing that a software engineer should remember from our show, what would you like it to be?

Daniel Stenberg 01:00:08 I usually maintain that one of the primary qualities that made cURL or has made cURL and libcurl succeed is just persistence to just keep on working on it until it actually works. And that’ll actually succeed. I often get to hear from people who try out things to write the tool and to say that well nobody’s using it. It doesn’t work and nobody is succeeding. I usually then try to go back and see that it took me many, many, many years with cURL and libcurl till we had a number of users. So, I think if one particular criteria to actually succeed with something like this is to just give it enough time and effort. So if you just want to and just keep on working on it, you can succeed in the long run. It’s not necessarily an immediate hit just because it’s a good idea. You just sometimes have to keep at it.

Gavin Henry 01:01:07 And then was there anything that we missed that you’d like to talk about or mention?

Daniel Stenberg 01:01:12 I could mention that we just recently surpassed 1000 commit authors in the project. So we were more than 1000 people actually written code much into the project. Sometimes people think of me as sort of, yeah, I’m the lead developer, but we’re also huge amount of people actually contributed code to.

Gavin Henry 01:01:31 And what are their names?

Daniel Stenberg 01:01:34 Well, we have the thanks list in GIT and that the thanks also includes all contributors. Also people who have reported bugs and help out in other ways. And that’s, I think that’s approaching 2,600 names now. So quite a lot of people who are helping out all the time.

Gavin Henry 01:01:51 That’s brilliant. Where can people find out more or get in touch?

Daniel Stenberg 01:01:55 Everything currently is of course on cURL.SE if you want to read up on cURL, we have this book on everything.cURL.dev, which is my book efforts to document URL and everything about me is on daniel.haXX.se. And of course, I’m on Twitter as back there. And I tweet a lot of cURL stuff none stop, a lot of blabbing.

Gavin Henry 01:02:19 Daniel, thank you for coming on the show. It’s been a real pleasure. This is Gavin Henry for Software Engineering Radio. Thank you for listening.

[End of Audio]

SE Radio theme: “Broken Reality” by Kevin MacLeod (incompetech.com)
Licensed under Creative Commons: By Attribution 3.0