Lane Wagner of Boot.dev speaks with host Philip Winston about Go, the programming language that’s popular for web, cloud, devops, networking, and other types of development. In addition to discussing existing features such as structs, interfaces, concurrency, and error handling, Lane and Philip take a deep look at generics, a recent addition to the language. They also explore the developer experience with Go.
Show Notes
- SE Radio 202: Andrew Gerrand on Go
- The Go Programming Language
- Why Golang Is Widely Used in the DevOps and Cloud Native Space
- An introduction to Generics
- Chi HTTP Router
- SQLC
- Memory Arenas in Go
Lane Wagner
- Twitter – @wagslane
- Blog – www.wagslane.dev/
Learning Go
- Boot.dev
- Backend Banter Podcast
- Go Time Podcast
- Tour of Go
- Learning Go
- Bill Kennedy
- The Go Blog
- Russ Cox Blog
Transcript
Transcript brought to you by IEEE Software magazine and IEEE Computer Society. This transcript was automatically generated. To suggest improvements in the text, please contact [email protected] and include the episode number.
Philip Winston 00:00:18 Welcome to Software Engineering Radio, this is Philip Winston. My guest today is Lane Wagner. Lane has been a backend developer and data engineer for the last eight years in fields such as hardware sensors, data pipeline, social media analysis and educational software. Lane founded Boot.dev in 2020. Boot.dev is a learning platform that uses game design principles to teach courses in Go, Python, JavaScript, and various backend technologies. Lane has a bachelor’s degree in computer science. Lane, is there anything you’d like to add to your background?
Lane Wagner 00:00:53 No, that’s pretty good. I don’t know how much we’re going to go into past work history, but I spent a lot of time in Python and Go and yeah, struck out on Boot.dev just a few years ago. I only went full time on the project end of 2022, but like you mentioned it was kind of a side project since early 2020.
Philip Winston 00:01:12 Okay, great. Today we’re going to revisit the Go programming language. The last full episode we did on Go was Episode 202, Andrew Jaren. That was back in 2014, but I believe much of it is still true today. But let’s start at the top. What is the Go programming language?
Lane Wagner 00:01:30 Yeah, that’s a great question. So Rob Pike is one of the original authors of the Go programming language, Ken Thompson, these guys really came from a background in C, right? And the goal with Go as far as I understand it, of course I’m not one of the authors, but the goal with Go really was to build kind of a newer, more modern version of C and really stick to a lot of the kind of Unix philosophies that went into kind of vanilla C. And so the way I think about Go especially in 2023, what 12 years or so after the initial release, I think of Go as kind of modern C for the web.
Philip Winston 00:02:10 Okay, well Go is a compiled language like C++ and Rust. What are the implications of that in terms of the development process or deployment or just what is it to work in a compiled language?
Lane Wagner 00:02:23 Yeah. So if you’re coming from JavaScript or from Python, working with a compiled language can feel a lot different for several reasons. I mean first of all there’s a build step that you might not be used to, granted these days with a lot of JavaScript developers using TypeScript. There might be a little bit more of a familiarity with the whole idea of building something before you run it. But what I found to be the bigger difference, and to be fair this isn’t quite the same thing but static versus dynamic typing, right? Almost all compiled languages at least that I’m aware of have static typing Go. And that can be quite the mental shift when you’re moving into a language Go. Now you mentioned that Go is it’s compiled, it’s fast like C or C++, but there are a few unique properties and I don’t want to jump the gun on questions you’re going to ask, but one thing that’s pretty interesting about Go is that it is compiled, it is very fast, but I would not place it in terms of raw performance on the same level as C, C++ and Rust.
Philip Winston 00:03:25 So you’re talking about execution speed, but the compilation speed is also a factor. So it is a compiled language, but what I read was one of the developers of the language was tired of waiting on 45-minute compiled times. I’m guessing that was C++, but they didn’t mention what language. So how did they try to keep the compilation speed under control, and do you feel that was successful?
Lane Wagner 00:03:52 So definitely successful. I’ve worked on some pretty large Go projects. I don’t think I’ve ever had compile times more than just a few seconds, which is absolutely fantastic for a developer productivity, right? It’s kind of that old argument that was used a lot for dynamic languages, which is, we don’t have to wait 30 minutes for a compilation, but it also is a great money saving thing. if you’re using GitHub actions or some sort of CICD pipeline to deploy your code, fast compilation is a huge selling point. I guess part of it is cost but also it is, developer productivity if you’re shipping to production four or five times a day, it’s really nice when your build and deploy pipeline just takes a minute or two instead of upwards of an hour before you get feedback that the build failed, for example.
Lane Wagner 00:04:42 So definitely successful on that front and it is important to keep in mind the difference between, I want to be super clear about this, the difference between compile speed and execution speed. So Go does execute fast. When you write a program in Go, that program will be fast when you compare it to languages Java, JavaScript, Python. Like I said before, I wouldn’t place it into the same category of execution speed as C++, C and Rust. But as you mentioned, one of the trade-offs there is Go compiles extremely quickly. So, a similarly sized program in Rust or C is going to take much. I want to say honestly there’s at least an order of magnitude difference in compile speeds.
Philip Winston 00:05:26 This might be harder to quantify but I think of Go as a smaller language than some of the other ones you mentioned. Do you think that contributes to the compilation speed or how are they getting this speed? Because it’s definitely sounds significantly faster.
Lane Wagner 00:05:42 Yeah, absolutely. So Go is famously a small and simple language, and the Go team is very hesitant to add features. I’m sure we’ll talk about this later. But generics were pushed for many years before they actually added it as a language feature, partially because they wanted to get the API right? And it was something they were kind of retrofitting onto the language but also just any feature in general they’re hesitant to add because having a small and simple language really just has so many benefits. We don’t have 17 different ways to write for a 4Loop in Go for example, looking at you JavaScript, right? And because the language is small and simple that makes it easier to write a fast compiler, right? Just the natural complexity of the problem is a lot smaller. And I will also mention that this is something I recently learned on my podcast. Back in band I had the author of the Rock programming language on, and we were talking about LLVM and Go does not use LLVM. So LLVM is basically this kind of standardized way of compiling programs, and it produces very efficient, very fast binaries but compiling with LLVM takes a long time. So one of the ways that the Go Team was able to get faster compilation speeds was explicitly by basically building their own.
Philip Winston 00:06:59 Yeah, we’re going to get into generics in just a little bit and then we’re going to kind of walk-through other language features and get your kind of summarize them but more get your take on them and kind of where things stand today with Go Development. But I wanted to ask what communities or problem spaces is Go popular with? You mentioned for the web in the beginning I think there are also other communities that are kind of embracing Go. Can you kind of characterize where you see Go being used today?
Lane Wagner 00:07:27 Yeah, for sure. I think — so, I’m going to put words in the authors of the Go program’s mouths, so forgive me if I’m wrong — but my understanding is that originally the idea was kind of to make a new C, and I think that that is not at all what actually happened after we’ve kind of seen Go in the real world. Where I’ve seen it, most is in backend web applications and it’s really taken hold of kind of the cloud native community famously Kubernetes is written in Go, Helm, right? Kind of that world, Docker, Terraform, Prometheus. This is kind of the world of the cloud native Go community and there’s a bunch of reasons for why Go is great on the cloud, but I’d say that’s probably number one. And then I would say number two, if I had to pick one is command line tools.
Lane Wagner 00:08:16 So because Go produces standalone statically compiled binaries, Go programs are fantastic for doing local development on your machine. When you install a Go program there’s no dependencies right? When you use NPM or Python you’re always worried about is the script I’m installing, what’s the dependency graph look like? How long is it going to take to install? Am I going to have conflicts? And even in kind of the world of C with dynamic linking you can run into, dependency problems. The nice thing about Go is every binary is statically compiled so you download it, you run it, it’s going to work.
Philip Winston 00:08:55 You mentioned command line utilities. I think every other day I see a new command line thing on Hacker News that I wasn’t aware of. How do you do argument parsing and command line flag handling in Go because in Python and C++ I’ve gone through sort of a rotating number of libraries that handle it different ways and it’s nontrivial to get all the flag processing right?
Lane Wagner 00:09:20 Yeah. So one of the major selling points of Go is this extremely rich standard library and there’s an OS package, an operating system package that makes it pretty easy to do your basic positional arguments and flag arguments. I’m going to be honest, the flag one in particular, I think positional arguments is really great, really easy, whatever. The flag parsing in Go, it’s simple, you can do it. The functional programmer in me doesn’t love how mutable and stateful the whole process is, and I won’t spell out exactly what the syntax looks but I guess the way I’ll put it tersely is it’s simple and easy but it’s a little weird from a mutable state perspective.
Philip Winston 00:10:05 Sounds it gets the job done though. So let’s start in on some language features. We are going to talk about generics first because it’s the newest major language feature that I found and in fact it’s got kind of a history to it. We can go in. So from your point of view, when did you start hearing the discussions about generics? And I can jump ahead and say it sounds it was released in 1.18 in March 22. So I’m wondering, I know the discussions went back far, but when did you start being aware that they were working on these generics?
Lane Wagner 00:10:38 So I started programming in Go, it was either late 2015 or early 2016 and very quickly I started hearing about generics. So I think people have been talking about generics probably almost since the, maybe even since the language was first released, right? Because you come from another language that has generics and you go, oh this thing is missing, right? And frankly that’s the experience of a lot of developers coming into Go. It’s oh this thing is missing and that’s by design again, Go is a very, very small language. You can learn most of the keywords and language features just in a few weeks if you’re an experienced developer. So with generics I’d been hearing about them the whole time and I think a lot of the resistance, so the first layer of resistance to adding generics to language was just this idea of keeping the language small and simple.
Lane Wagner 00:11:28 The second layer of resistance, I think, and this is now just me speculating on the community but for me, when I was building backend web applications in Go, which is what I was doing at the time and what I still am doing, I don’t feel the need to use generics very often at the application layer, most CRUD applications on the backend, you’re taking data from some source right? From a RabbitMQ or from some database from an HTTP handler. You’re doing some sort of transformation on it and you’re shoving it somewhere else. And so the thing you’re most concerned with is the structural shape of the data, which in Go is easy to represent with struts. If you’re working with JSON data, it’s very common that you’re just taking a big old JSON object, parsing it into a struct in memory nested structs whatever and putting it somewhere else.
Lane Wagner 00:12:19 So generics just don’t seem to come up all that often. I think where the pain, the people who were more vocal about getting them added in my opinion were probably people writing libraries, where they’re trying to write code that’s very agnostic. Um, they don’t want to repeat themselves 72 different times when they want essentially the same logic for different types. And that makes sense to me and I’m glad they took their time that they got the API that they did because I think they did a really good job with it. But frankly at the application layer I still don’t use them all that often.
Philip Winston 00:12:51 So going back to compile time for a minute in C++ the template system, which is generics has these pretty complex features apparently it is to complete, you can write programs that run at compile time essentially and that was template meta programming, recursive instantiation, all of these things which I think Go did not adopt. So was do you think again compilation time was a factor in their implementation of generics that maybe led them to avoid some of these features that can kind of blow up on you?
Lane Wagner 00:13:27 Yeah and I mean so much so that I’ve even heard C++ developers accuse Go of still not having generics. The idea being of course that the generics that are in Go are very simple and if you look into it, the Go team actually doesn’t really call them generics all that often. They usually refer to them as parameterized types. And so it is a fairly simple system, although I will say I’m not an expert on the C++ template system. So I have a hard time doing deep comparisons other to say I’m familiar with the fact that C++ can do some crazy stuff at compile time.
Philip Winston 00:14:01 One thing I did read about Java in contrast to C++ is they use something called type erasure. So the byte code doesn’t have the actual types in them, and this prevents certain optimizations, I think I understand the Go, although it’s simpler system than C++, it does optimize down to the types. Is that true?
Lane Wagner 00:14:21 Yeah, that’s my understanding. To be honest I’m not an expert on the compiler itself but I don’t see why they wouldn’t. In Go you’re using parameterized types, you’re adding type parameters to a function. So to give you an example of what you used to do in Go, so before type parameters or generics were released, in Go you basically had two options. You would, well let me give an example of a function that you might use generics for. So let’s say you have a binary tree, very simple data structure that you’re coding up in Go and this binary tree can store arbitrary data types, right? so say the logic for the binary tree is all going to be based on the keys and so all your sorting logic will be based on say like integers if we’re using integers for keys but you could store anything in it, you could put a struct inside, right?
Lane Wagner 00:15:08 You could put a string, you could put a map. It doesn’t really matter what the values are for the logic of your binary tree. So in the past with Go, what you’d have to do is if you want concrete types like a binary tree of let’s say a car, so you’ve got this car struck. Well you’d code up the binary tree and you’d be using all these concrete types and everywhere you’d be referring to this car struct and now you want to make a new binary tree and in this one you have, let’s just say another object of humans, right? And you want to sort the humans in the binary tree you actually have to duplicate all of that code. This is what you were forced to do, kind of in the early days of Go. And so you were basically stuck with either writing out all the code by hand twice or what a lot of people would do is use the Go generate command to essentially generate the code twice with multiple concrete types.
Lane Wagner 00:16:04 Now with generics you can have a type of parameter and the Go compiler will essentially build out those two different functions with concrete types for you. So I guess to answer your question in a roundabout way, yes it does keep the concrete types intact and in fact that’s one of the best parts because I guess you did have a third option in the early days you could use an empty interface which in Go is essentially like an any, if you’re familiar with TypeScript you’ve got the any type? The problem with that is now at runtime you have to write code to cast your types into and out of that empty interface so that any type. So to answer the question, yes generics made it a lot easier and yes, they do compile down to concrete types at least in most scenarios that I’m familiar with.
Philip Winston 00:16:52 Yeah, I think that’s what I read. So then one last point about generics, why square brackets, that kind of threw me off when I was reading the code. I think, to point out, I think C++ and Java use angle brackets, I’m not sure what other languages is used but.
Lane Wagner 00:17:10 Yeah that also threw me off. I think there were two primary reasons that I read about. The first was that in Go the square brackets are already kind of a standard for types in the sense that the Go programming language, even before generics, kind of had this one magical data structure — or I guess you could say two magical data structures — that were essentially generics but baked into the language which are slices or essentially Go’s version of lists and Python and maps and when you’d initialize a slice or a map you could pass in essentially a type parameter in the square brackets. So you know a slice of strings, a map of strings to ints — we kind of had this standard in the language of using square brackets. So, I think part of it was just to stay consistent with that standard. I also read something about the angle brackets throwing something off with the Go parser due to angle brackets being used for other things greater than less than science. I don’t know how true that is though. I just remember reading about it on some forum.
Philip Winston 00:18:15 There was a situation like that with C++ at least back in the day, where you wanted to do two angle brackets in a row and you had to put a space in between them or else it would think that was an insert operator or something you actually had to kind of fudge it or you get a compiler.
Lane Wagner 00:18:33 Yeah.
Philip Winston 00:18:35 So let’s move on from generics and we’ll kind of go back to the top and go and try to talk about for people that are less familiar with it. First maybe a trick question here is Go an object-oriented language or not.
Lane Wagner 00:18:48 So to put it simply I’d say no but now it’s time for the huge caveat. The way I look at object oriented with OO languages, the primary defining feature is inheritance and inheritance does not exist in Go. So if that’s the criteria then we immediately throw it out and say Go is not an OO language but there’s some other useful ideas from OO that Go does make good use of things like encapsulation polymorphism. There is really good support. In fact I’d argue that Go has one of the better systems for encapsulation with their package system and how packages in Go exist at the directory level rather than at the file level. So if you’re coming from JavaScript you’re probably used to having to encapsulate modules at the file level. In Go that same idea is done at the directory level, which I think is at least personally I think it’s an easier thing to work with. Makes organizing files, it just makes it more convenient.
Philip Winston 00:19:45 Sorry, so do you mean that all of the files in one directory are sort of glommed together as part of a module or what do you mean by directory versus file?
Lane Wagner 00:19:53 Yeah, essentially. So in Go at the top of every file you have a package name, and that package name is the same throughout a single directory on your file system. So for example, you typically have a main package for your Go program and every Go file at that level in your directory hierarchy is going to be in the main package. And so to access functions within that package, you don’t need to export them, they’re just available within the package. And the nice thing about this in my opinion is that you can organize your functions into different files however you like and you don’t have to think about the private versus public, right? The whole idea of encapsulation and trying to protect certain data members or functions from being called outside of the package. Whereas in a language like JavaScript, where you have modules at the file level, I think the disadvantage is you might have wanted to split a file up into multiple files just for organizational purposes, right? But if you do that now you have to do the import export thing, you have to deal with encapsulation at that level.
Philip Winston 00:21:04 I want to maybe come back to that issue of public private, I didn’t even notice that not appearing in the Go that I saw but first Go has interfaces but not inheritance. So how does, I guess a struct in Go terms, how does it convey that it can implement an interface?
Lane Wagner 00:21:24 Yeah, so struct and Go are kind of your corollary to objects or classes in a lot of other languages they’re a lot simpler, right? They’re kind of structs and C, they’re just collections of data for the most part. The weird thing is you can put methods on structs and in fact you can put methods on other types as well. And basically the way an interface works is an interface defines a set of methods that are required in order to satisfy the interface. It’s the really classic example in Go is the stringer interface. It’s an interface that basically says if any type has a dot string method on it that returns a string, then that type satisfies the stringer interface. And so anywhere in my code that accepts a stringer, I can now use that concrete type as an instance of that interface. And one of the unique things about Go that I think can throw developers that are coming from, a language where interfaces are satisfied explicitly is that in Go they’re satisfied implicitly. So in other words, if you have a struct and you add that method, that string method that returns a string, a method named string that returns a string, then congratulations your type is a stringer now. Even though you didn’t add any other call out that you want to implement that interface.
Philip Winston 00:22:56 Yeah that’s definitely curious to me because in that example that seems such a common interface that it’s not a big deal. But suppose I have in my application three, four interfaces and I have some object that or struck that implements two of them. How do I kind of keep track of that? Do you just comment it. Is that an issue I guess to keep track of what is implemented?
Lane Wagner 00:23:22 I’ve never had it be an issue, to be honest. I will say one philosophy that Go developers’ kind of have and this has been handed to us from on high by the Go authors is that interfaces should be very, very small. So I think good Go code idiomatic Go code, right? You’ll typically see interfaces with one, two, maybe three methods on them but they tend to be very small. It’s a big mental shift coming from like NOO language where you’ve got an object with just, boatloads of methods that get inherited from level to level. In Go, what we would typically do is use a struct and that struct that concrete type could maybe have lots of methods on it or, or maybe you even just use regular old functions, right? That accept instances of that struct and you’d kind of save for the interface truly things that need to be used across multiple concrete types. And there’s nothing wrong with having an interface that is a very small subset of what the structs that implement it can actually do, if that makes sense.
Philip Winston 00:24:27 This might be related, so I know there’s something called struct embedding. I’d like you to explain how that works but I think that, well let’s hear how that works first and then I can ask my next question.
Lane Wagner 00:24:41 Yeah, struct embedding is a useful little quirk of the language. So typically if you have to like nest some data, right? So let’s say for example, this is a real example from the Boot.dev backend. We’ve got a user struct and the user struck told all the data for a user in our backend system, right? We’ve got email, password, username handle created at all that kind of stuff and there is a subset of the fields on the user struct that we want to keep private. In other words, when someone looks at your public profile page for example, we don’t want those fields making their way to the front end at all, right? We don’t want to expose a password on that struct for example. So one option would be to nest these private fields. So in other words we’d create a new struct maybe called user private fields and we’d in the user struct create a private fields key and that private fields key would map to an instance of this nested struct. That would be a nested situation, which is something I’m sure everyone listening to this is familiar with, right?
Lane Wagner 00:25:48 You can think of a nested JSON object or a nested dictionary in Python. Embedded structs in Go or a little different, it’s where you literally take the struct declaration, it’s private user fields and you just place that inside the struct’s definition of the parent struct. So in this case user, without a key. So now instead of, private fields colon, user private fields, the name of the struct, you’ve just got private user fields. And what that does is it elevates all those fields in the embedded struct into the parent struct. So you can access them with the dot operator at the top level. So you could just do like for example, user.password instead of user dot private fields dot password. It is a little wonky because you actually can still access them by doing user dot name of embedded struck, so user, private fields, password, you can actually access them both ways. But the nice thing is when you embed in this way, it’s a good way to share a common set of fields amongst different types. So if you had a user and an admin and you needed them to share 10 different fields, instead of typing that set of 10 fields into both structs and copying that code, you could just embed it.
Philip Winston 00:27:14 So a short question first you mentioned this was private fields but that’s nothing to do with like a private public keyword, correct?
Lane Wagner 00:27:21 Yeah, yeah this would be more of a private public in security across a network.
Philip Winston 00:27:27 Yeah, it’s the applications delineation, I guess.
Lane Wagner 00:27:31 Yeah.
Philip Winston 00:27:32 But my question before that was going to be, so back to the implementing interfaces, is it possible if I need to implement three interfaces that I could do that with three structs that are embedded or not?
Lane Wagner 00:27:47 So the embedding of a struct doesn’t really in any way correlate directly to the satisfying of an interface. So if we had a parent struct with a nested, or I should say with an embedded struct inside of it, whatever data is in that embedded struct could be used in the method that the parent struct would need to satisfy the interface. But it’s very agnostic about whether that data just existed at the top level or whether it was embedded, if that makes sense.
Philip Winston 00:28:20 So you mentioned struct methods, is that where the receiver comes in? What is the receiver in Go?
Lane Wagner 00:28:28 Yeah, exactly. So methods are funny things, especially when you come from kind of more of a functional background in a sense that you can really think of a method as just a function with one special parameter, right? Where there’s one special parameter that is the instance of the thing itself. So in Python they’ve kind of acknowledged this and they just always name that first variable in a method itself, right? So methods look just functions where the first parameter is an instance of the object. In Go it’s very similar but that special parameter is not just the first parameter in the function, it’s actually off to the left-hand side of the function name. So you’ll get the receiver parameter when you define a function and then the name of the function and then the actual parameters to the function that are not just the instance of the method.
Lane Wagner 00:29:22 And the reason it kind of needed to be this way and Go is because we don’t define methods on structs within the struck definition itself. So for example, to use the analogy of Python, in Python you would define a class and within the class definition you’re defining all of your data members and your methods. In Go, that’s not the case, you would just define a struct, which is just data, right? It’s just a struct with key value pairs essentially. And then outside of that struct definition is where you’re defining your methods on the struct. So that’s why you have this kind of special parameter off to the left-hand side so that the language knows that a method is associated with a specific type.
Philip Winston 00:30:09 So that was sort of objects interfaces and structs. Now we’re going to move on to one of the, I think sort of marquee features of Go when it, at least when it came out, which is concurrency and then just peeking ahead we’ll talk about air handling and then higher order functions and then some other topics. But starting in with concurrency, I think I remember when Go was released how significant it was to have I think Go calls them Go routines and channels and they’re really part of the language, it’s not a library. So I wanted to maybe even going back, why do you think they included that directly in the language I guess in 2012 when it came out sort of what was going on that concurrency was such a big deal and then moving on to how do you use it today, how often is it used? Questions that.
Lane Wagner 00:31:07 Yeah, so this is a very unique thing about Go I think still today, although not that I keep up with every programming language that’s been released in the last few years. But Go routines and channels are very unique to Go and when you compare Go’s kind of native support for concurrency to other languages as you pointed out with other languages you often have to go to outside libraries to get these kinds of features. So first, I guess we could just start with what’s a Go routine. So a Go routine is basically a lightweight thread. So in a language like C++, if you want to do multi-threading, you’ll very often just spawn a new operating system thread. It’s the operating system will, you could give you a new thread of execution to work with. In Go, the Go runtime manages those operating system threads and what it does is multiplex Go routines on top of threads.
Lane Wagner 00:32:09 So for example, I could spin up let’s say two Go routines and underneath the hood the Go run time will likely spin up two operating system threads. The difference is I could spin up a million Go routines and I’m not going to spin up a million operating system threads. Because operating systems are actually quite resource intensive and expensive, to get access to. So what will likely happen is in Go I might spin up four threads, I don’t know whatever algorithm it uses to figure out how many threads it’s going to spin up under the hood, but then we’ll multiplex all the kind of computational operations across those threads and handle that for you. And the nice thing about this, at least from my perspective is you get this really lightweight way of doing parallelization and the overhead of the operating system threads is drastically reduced but it’s abstracted away from you, so you don’t have to think too much about how many threads are actually under the hood. Usually what I find myself doing as a Go programmer is just thinking how many CPU cores do I have available to me, right? So if I’m on a single core machine, I know that even if I spin up 20 Go routines, everything’s still happening in essentially a single threaded environment. But if I have 16 cores available to me, I know that my code will be kind of spread out evenly across those cores as I need it to be.
Philip Winston 00:33:43 I’m not sure where CPUs were in 2012 but I know the number of cores is just increasing rapidly. I think on a server now you can get 96 cores, maybe even 192 cores or something and it makes sense to me to limit the number of hardware threads. I think in systems I’ve worked on often you want sort of N where N is the number of cores or maybe you want two N or four N, but yeah you don’t want a million certainly, you wouldn’t even be able to allocate that. So how about channels? I think you explained Go routines pretty well. Are channels intrinsic to Goines or is that another feature you can use?
Lane Wagner 00:34:25 Yeah, channels are, I would probably define them as a separate thing, and they just happen to be really useful in tandem with Go routine. So the most basic way I can think to describe a channel is it’s just a Go routine safe queue or a thread safe queue. So a channel is just this little data structure you can create a new channel, you can give it a type, so it’s kind of a slice or a map in that sense. You can make a channel of integers or a channel of booleans and you send stuff through a channel. So it’s a queue stuff goes in one side and out another side and its thread safe. So you can pass a channel into two separate Go routines, which in Go the way you spin up a Go routine is just by calling a function and putting the keyword Go in front of it.
Lane Wagner 00:35:12 So instead of create user, you go create user and that will call that function in a new Go routine. So with channels you can share channels among different Go routines and the way you do concurrent programming and Go, due to the use of channels feels a lot different than if you’re used to using for example primarily mutex as mutual exclusion, in a language like C++. And the reason for that is we’re trying to optimize here for communication, right? So we’re trying to communicate data from one Go routine to another rather than try to kind of safely access the same memory, which is what you do with a mutex, right? You’d have some variable, and you’d lock access to that variable with a mutex so that only one Go routine can access it at a time.
Lane Wagner 00:36:09 This is how you’d avoid race conditions. But in Go what you’re typically doing is sending data across a channel and channels are interesting because there’s a bunch of different operations you can do with them, but very often what you’ll do is you’ll have one Go routine sending a value across a channel and another Go routine is sitting around waiting for that value to come across the channel. So you essentially have this way to await or block and wait for something to come through a channel. So it’s very often used in data processing, right? We’ll take something a RabbitMQ or a Kafka or a pub subsystem and kind of map the incoming data into a channel and then we can have many Go routines all pulling data out of the channel and processing them in parallel. And because it’s thread safe, none of them are going to get the same copy of the data, they’ll all kind of pull it off one at a time, round robin. This kind of model is useful especially again, kind of in backend web systems at least that’s where I’ve used it the most.
Philip Winston 00:37:11 You mentioned mutex though and I think there’s also atomic operations. Do those still get used or do everything map onto Go routines?
Lane Wagner 00:37:19 Those do still get used. There’s a sync package in the Go standard library. There’s also an atomic package and so I use the sync dot mutex type from time to time. The thing to get right is to figure out when you should be using a mutex and when you should be using a channel and I don’t know if I have a really good heuristic for that but I guess one heuristic is if you have a big chunk of memory that you need to share access to, so imagine like a big map in memory, a big key value store, a mutex probably makes sense. But if you have workers’ items in a queue that need to be processed, that’s a very natural fit for something a Go routine and now your program doesn’t have to have any locking involved. And there’s this great video I was watching on YouTube a couple years ago where someone was writing a service in Rust and then they went and rewrote it in Go and what you’d expect is for the Rust version of the service to be faster and I’m sure and the author of the video was, I’m sure if I had written this better the Rust version would be faster. But Go just makes it so easy to write efficient concurrent code that his version in Go actually ended up being much faster and I’m assuming because there was just less blocking going on.
Lane Wagner 00:38:36 So getting that right could be tricky but for certain use cases Go routines and channels are great to work with.
Philip Winston 00:38:43 Again, I’m not sure we can do the direct comparison, but I feel in Python things have moved from explicit threads to async that has some similarities. So let’s move on from concurrency and talk about air handling. This is one of those cases where it was surprising to me. There are no exceptions in Go. Is that true?
Lane Wagner 00:39:08 That is true. I mean you can panic but as a general rule you shouldn’t do that and Go developers that are writing idiomatic code very rarely panic. So yeah, we handle errors as values. The only other language I’m super familiar with that does a similar idea is Rust.
Philip Winston 00:39:25 So error as value there is an error type and that gets returned as a second parameter from all functions or some functions?
Lane Wagner 00:39:36 Yeah, so the error type is built into the language and it’s just a little interface. It’s a little interface with one method called Error that returns a string and that’s all it is. And by convention when you have a function that can go wrong for lack of a better term, then the last return value because in Go a function can return multiple values, the last return value will be an instance of an error and famously in Go you have this line after what’s seemingly every function call. That’s if error does not equal nil, right? And if it doesn’t equal nil then you need to handle the error. And usually that just means passing the error up the call stack. So it’s usually if error does not equal nil return the error, right? So Go is very much kind of a guard clause-based programming language. That’s kind of a pattern that you’ll see in Go a lot, which basically just means a Go function will very often just have a bunch of if statements inside of it that’s saying if something goes wrong, return early, if something goes wrong, return early and then getting all the way down through the function would kind of be considered the success case.
Philip Winston 00:40:46 Yeah, with exceptions, although I them in some ways one of the potential mysteries is I’m throwing an exception and I want to figure out who is going to handle that. It might be several layers up with no indication in those intermediate layers that anything is going on. And so I’m guessing this is meant to be more explicit and eliminate that case where you’re just throwing an error and you have no idea who’s going to handle it.
Lane Wagner 00:41:15 Yeah so for me the biggest thing is just when I call a function in Go, if it doesn’t return an error, I’m reasonably certain that nothing can go wrong. So the only functions that don’t return errors are ones without side effects, right? Like pure functions they’re doing some sort of pure computation usually. Any function that accesses a database or does a network call or something that almost certainly is going to return an error and correctly as you pointed out this is my big frustration when I’m working in a language Python or JavaScript is that I can very easily call a function and I have no indication in the function signature that something can go wrong here that I need to be wrapping this in a try catch or whatever.
Philip Winston 00:42:01 Yeah that’s actually a different point than I was talking about but I think it’s even more fundamental that when you’re calling something in a language with exceptions I guess some in C++ I think you are supposed to declare that your rate could possibly raise an exception but it’s at least back in the day it was optional. I don’t know the details. How about the defer keyword? That sounds to me kind of finally in exceptions, which is basically I want to run this no matter what happens is defer similar or I guess it can’t be similar because there’s no exceptions, but how does defer work?
Lane Wagner 00:42:38 It’s very tangentially similar in this just in the sense that you’d kind of use it in the same way. The defer keyword is one of my favorite features of the language and I frequently find myself in, I do a lot of programming in JavaScript or Typescripts wishing that it existed. So basically the way the defer keyword works is you use the defer keyword and then you pass it some function call. Some evaluable expression and it runs that or it will run that before the function exits. So the most common reason to use it is just like cleanup code. So you’ll have a function say that opens a database connection and immediately after opening the database connection you’ll defer closing the database connection, because maybe later in that function there’s five different places where you could possibly return from the function, right? If there’s a network error, you’re going to return the network error. If you can’t parse the data that you get back from the database, you’re going to return some custom error value, right? And without the defer keyword what you’d essentially have to do is close the connection at every single one of those return, every single one of those exits have options. So mostly used for cleanup code.
Philip Winston 00:43:57 So the last few topics we were hearing about kind of ghost spin on those features. This one I want to talk about first class and higher order functions. In this case I was kind of surprised that Go did have it because it doesn’t feel very C like. It feels maybe JavaScript like, can you talk a little bit about first class and higher order functions?
Lane Wagner 00:44:21 Yeah, so first class and higher order functions, to kind of get hand wavy about it, it’s really this idea of functions as values, right? You can kind of create functions as values and pass them around your program just as if they were normal pieces of data and Go does support this. I think most languages these days tend to support it. To be honest, I don’t think I’ve personally worked with a language in recent years that hasn’t had some idea of functions as values. It’s really critical that Go does support this because let’s take the example of an HTTP server and Go. So very often in Go you’re building some sort of JSON API, right? Some backend service. And there’s kind of two interesting things about how you do this in Go. The first is that you’re using Go routines as your handlers, which is like a little different from the JavaScript world, right?
Lane Wagner 00:45:12 In JavaScript you’re kind of using the async weight, single threaded but non-blocking asynchronous operations to handle many requests at the same time. In Go you’re literally just putting every request on its own Go routine. So every handler can not only do asynchronous tasks efficiently, like IO bound tasks going to the network or going to a database, but it can also do computational tasks efficiently across handlers. And then the reason that first class and higher order functions are useful is because that’s generally how we define handlers in Go. So we’re kind of passing the reference to a function into some sort of routing library and very often it’s the Go standard library, it’ll take a path and a function as inputs to the router. So you’ll say I want slash users, I want that to map to this function and you’ll just pass in the name of the function. This is my http handler function. So without first class and higher order functions, I think that that whole process would look a lot different.
Philip Winston 00:46:20 Okay, maybe two more features and then we’ll move on to sort of developer experience and ecosystem, stuff like that. So maybe they’re related pointers and garbage collection. So in C++ you don’t have garbage collection, but you do have pointers and in some languages, you have garbage collection and not pointers. How is it that Go has both? And do you use pointers differently in Go than in some other languages? And how often do you use pointers?
Lane Wagner 00:46:51 Good question. I use pointers all the time in Go. Garbage collection, well okay, so there’s a couple things here. So first Go is interesting in that Go allocates a lot of stuff on the stack. So if you’re familiar with Stack versus heap, Go will actually embed collection types like structs specifically on the stack and that’s really fast, the stack is much faster than the heap. The minute you use a pointer in Go, you move to the heap. So this is actually the only reason I bring this up is this is a common gotcha in Go is as a speed hack. Sometimes people will use pointers in Go and with the idea being or the theory being, well I’m using a pointer therefore I’m not copying memory. As it turns out in most situations, and of course you you’d have to benchmark your specific situation to see if this is applicable to you, but in most situations it’s actually faster not to use a pointer and go and to just let the compiler allocate to the stack. Because when you do move into pointer land you do tend to move to the heap.
Lane Wagner 00:47:57 So that’s one thing. Pointers in Go are interesting, it’s one of these hotly debated topics people are really frustrated with nil. There’s this idea that null and nil and none in Python are like the billion-dollar mistake because we spend so much time around nil de referencing or something being undefined. But I do think that for the simplicity of the language pointers generally speaking we’re the right call. The heuristic I’d use in Go for whether I should use a pointer or not generally has nothing to do with performance optimization and it has a lot more to do with mutability. So when I have a function that accepts a pointer, I’m expecting that one of the quote unquote outputs or effects of my function is that I’m going to mutate whatever value that pointer points to, as part of my function execution. So if you don’t want a value to be mutated, you should not be passing it as a pointer in the general case. Pointer basically means let’s change this thing.
Philip Winston 00:49:00 So continuing on the theme I guess of simplicity and speed; I know garbage collection in Java has been optimized for decades. I think it’s a big focus of the developer community to improve garbage collection. How has Go’s garbage collection been since it came out? Is it considered a weak point? Is it a strength?
Lane Wagner 00:49:24 So as a general rule, in my experience a Go program will use less memory than an equivalent Java program. There’s a couple reasons for that. So first, as we talked about earlier, Go is a compiled language, and it compiles directly to machine codes. You get an executable as your output. You can contrast that with most Java programs where you’re compiling to byte code that’s going to run on the JVM. So it’s much more similar to C, C++ and Rust. The difference is, in Go when you compile to machine code, there’s a bit of extra code that’s included in every Go application and that is the Go runtime. So you basically have your code and then you have the runtime. So that’s why hello world and Go can actually be a fairly large static execute, got the whole runtime bundled in there and the runtime is responsible for the garbage collection.
Lane Wagner 00:50:18 My understanding is that the reason Go’s garbage collection tends to be, I shouldn’t say garbage collection, but memory management tends to be a little more efficient than some other counterparts like Java, C#, Python, Ruby, take your pick, is due to the simplicity of stack allocation. So, in Java you’re taking full objects and throwing them on the heap. In Go you’re very often just allocating primitive types or collections of primitive types directly onto the stack because their size is known at compile time. So that has a big impact on how much memory you’re allocating. Now of course you can go do crazy things like make a giant map in memory and then of course your memory will blow up and then you have all the same problems, stop the world garbage collection, that sort of thing. But I guess the way I’d say it is if you’re really concerned with memory then you need to look to another language. But for things like high performance web servers Go is still, it’s in that performance tier that is acceptable for the vast majority of cases.
Philip Winston 00:51:22 The stack allocation is interesting in C there’s a function that will allocate memory on the stack, but that’s considered kind of a bit of stunt programming. I mean you don’t see it that, you don’t see it used that often. So it’s interesting that they built in the stack allocation into the language. The runtime is a good segue into what I would call the developer experience. So this is beyond just the programming part. If the runtime is embedded in the executable, one trade-off there is you can’t upgrade the runtime of a program that’s already been deployed. So, with Python, if they come out with a new Python version, I can potentially run that on my old Python program. I don’t have to recompile it or anything and it gets the benefit of the new. In this case, if I have sort of a directory full of deployed Go programs, they might all have different runtimes depending on when they were last built. Is that true?
Lane Wagner 00:52:24 That’s true, but I have a hard time imagining a world where that’s not the easier thing to manage. In the sense that a deployed Go program, there’s no Go code in a deployed Go program, right? It’s just a binary, it’s when you want to redeploy you just build a new binary which takes a couple seconds and slap that up on your server. And so generally speaking I’d say it’s actually quite a bit easier because you’re not worried about if it compiles in the new version it’s going to work rather than having to test everything with the new runtime or the new interpreter.
Philip Winston 00:52:58 Yeah, I guess the same thing holds for dynamic libraries, which I think have fallen out of favor. The original idea was you could upgrade the library and all the applications that were using it would sort of get magically upgraded and I think that proved to be complicated to manage and embedding libraries statically, I feel it’s more common today.
Lane Wagner 00:53:25 Yeah when you are really constrained for resources then you can save, disc space and whatever by doing dynamic linking, like installing a program that has dynamic dependencies is going to be a much smaller and faster installation process than one where you’ve bundled all the dependencies together. But yeah, I think in web development in particular, what we’ve really migrated to over the years is like, we just want dependable builds. We want to know that when we deploy something all the dependencies are there that they haven’t changed. So, in the Python world, you’re using things requirements dot TXT, lock all your dependencies. In JavaScript you have this package JSON dependency file and in Go we just, bundle all dependencies that would be dynamically linked in another compiled language just directly into the binary. It means the binary is a bit bigger, that’s for sure the case, but it’s not on the order of gigabytes. So at least on the web where you have fairly powerful servers that you’re working with, the trade-off tends to be worth. I think this is why Go will continue to struggle in the embedded space, right? Where C and Rust and maybe Zig now I think are going to do really well because Go does make this trade off that you just, when you’re really constrained for resources, sometimes you can’t make.
Philip Winston 00:54:49 That’s interesting. I don’t often think about embedded programming but it’s definitely a domain. So going back to the package and module system and directory, how do I structure my Go application from a high level? Is there a tool that kind of spams out the directory structure or do I just start from scratch?
Lane Wagner 00:55:10 You usually just start from scratch. There’s not like an official directory structure. There’s a bunch of conventions that have cropped up over the years, but the simplest Go program is just a maine dot Go file with a function called maine and FMT dot print line, HelloWorld, right? So you can start really that simply and in Go it’s pretty rare. Well I guess what I should say is there’s really no Django or Rails equivalent in Go. There isn’t this giant fully fledged batteries included web framework in Go. Most services in Go make use of primarily the standard library and you’ll import certain third-party packages to do specific things. So for example, on Boot.dev what we use is the Chi router for HTTP routing, which is very, is just a thin wrapper on top of the built-in HTTP package in Go. So you can build a very simple HTTP server in Go that’s production ready just with the standard library, which is something that’s hard to do in a lot of other programming languages.
Lane Wagner 00:56:18 The Chi router just gives you some convenience wrappers around the standard library so that you kind of get a more declarative way to define all your HTTP handlers. So we use the Chi router and then we’re using right now a library called SQL C that allows us to compile raw SQL files into Go code that makes use of again the Go standard libraries SQL package. So this is kind of the philosophy of the Go community which is basically just we don’t have these all-inclusive batteries, these batteries included framework that exists outside of the language. Instead we have a rich standard library that everyone’s kind of using to the extent that they can and then you kind of pick a few third-party packages that makes your life easier for database access, HTTP routing or authentication for example.
Philip Winston 00:57:08 Related to external packages, I think I saw that Go encourages you to use GitHub or GitLab URLs in your list of dependencies. I’m wondering what the implications of that versus having a package manager or like in Python there’s PiPi as kind of the centralized manager.
Lane Wagner 00:57:30 We keep touching on all the things I love about Go maybe at the very end I need to give some gripes, so people don’t think I’m just all rose colored glasses about the language. But this is another thing I love. Yeah. In Go when you add a dependency to a project you’re doing, you’re directly adding the URL to the remote Git repository of the code you want to include. And that actual command is GoGet. So I’m in the root directory of my project, I do GoGet and then the URL of some GitHub repository and there’s a few advantages of that. First is there’s not a central source of truth baked into the tool chain. So with Node you’ve got the NPM registry, right? And the tool chain is aware of the registry and in order to publish a package that can conveniently be used by other JavaScript developers, you’ve really got to create an account on NPM, push up some code to NPM and then use it.
Lane Wagner 00:58:28 Or in the case of other systems, sometimes you’re not even pushing up the code per se, you’re more pushing up built object files or something. In Go because of the fast compile times, it’s really the compile times I think that make this convenient. Because of the fast compile times we just check the source in as a dependency and then we build from source which is convenient for a bunch of reasons but one of those reasons is I now have the source locally in my project directory and I can inspect it and look at it and see how it works without having to only rely on the docks for example. I guess inspect ability is a little bit more transparent. The other nice thing is, as I mentioned, there’s no central source of truth. So even if you’re using a private Git repository or even if you’re using GitLab or Bitbucket or some alternative provider, it doesn’t matter, the GoTo chain just cares that it’s a remote Git repository. Although there is that dependency on Git which is kind of interesting.
Philip Winston 00:59:28 So near the end let’s definitely get your gripes on Go. We’ll do that when we’re talking about what’s next for Go so you can register your complaints. But finishing up on developer experience, what can you say about the Go community? Where do they hang out? How do you feel it’s been interacting with them? What are they like?
Lane Wagner 00:59:47 Love the Go community. One of my favorite podcasts. So if you’re interested in Go specific content, first of all Go is a very small community that’s grown a lot especially in the last couple of years. I think the release of generics and just honestly just a bunch of podcasts, YouTubers, people talking about Go recently has really grown the language in the last couple of years it’s really started to pick up some steam? But I know the people that run the Go Time podcast, that’s fantastic. Obviously on my podcast backend banter we talk about Go a lot and every interaction I’ve had with the Go community has been pretty phenomenal. It is a very centrally managed community. So if you take a language like ZigZig as an example, Zig is not really run by a company, right? There’s not a central main sponsor, right? React has Facebook, Go has Google, Zig is like one guy. So that is one aspect to Go that I’ve heard criticism of which is, well Google can just do whatever it wants to do with Go and that’s true. I think that’s a fair criticism although it hasn’t been a problem yet though I guess you could make the argument with any sort of benevolent dictator for life situation that it’s never a problem until it’s a problem. But I’ve been happy with how the Go team does everything so far.
Philip Winston 01:01:07 How about learning Go? What would you point people to as a starting point? I think for any big language there are many many different starting points but if you had to list one?
Lane Wagner 01:01:19 Yeah, so I mean obviously I’m going to plug my own thing. Boot dev is a place to interactively learn Go and more specifically backend development. So if you’re interested you can go check it out Boot.dev. I’d also definitely plug the tour of Go. So it’s kind of the official kind of step-by-step through the language. If you just Google tour of Go, you’ll find it. I would point to Jon Bodner’s book if you’re more of a book person. I think its Learning Go by Jon Bodner. That’s a great one. I had him on the backend Vance or Pod recently. And then I’d also point you to Bill Kennedy, which is another friend of mine. He’s one of the co-founders of Arden Labs and he has a bunch of kind of more in-depth advanced tutorials for specific Go use cases. So building a web service in Go with his preferred methodologies for example. Those are a few that hopefully help.
Philip Winston 01:02:10 I’ll put those in the show notes. Now what is the future of Go? Can you describe anything that you know they’re working on and then what you would personally address in the language or the ecosystem? What are the sore points if any?
Lane Wagner 01:02:27 Yeah, so I don’t know what’s coming up in the near term. Russ Cox is the guy to look up and the person who writes the blog post that talks about what’s coming and his, well he publishes on his own blog as well as on the Go blog. So you definitely should check out both of those places for the real source of truth. One thing I’ve heard rumors of that sounds exciting is memory arenas. I don’t know if this is something that’s coming but it sounds interesting and basically the selling point as I understand it would be a drastic reduction in the performance hit of garbage collection. So a kind of different way to look at automatic memory management that in theory could really improve some performance issues. So that would be cool. Most of the gripes that I have I don’t think are easily solved. I think we’re kind of stuck with them. There are things that I think you kind of need to design for out of the box, but I don’t know, I could be wrong. I could be pleasantly surprised.
Philip Winston 01:03:37 Well how about name one of them then?
Lane Wagner 01:03:39 Yeah, my big one is definitely the type system. So in Go, we are really missing some fundamental typing ideas. The big one is Enums in my mind or union types, right? Or algebraic data types, whatever you want to call them. Basically the idea that a value could be of this type or this other type, right? So for example in Rust when you return an error in Rust you’re not returning two separate variables, you’re not returning the variable that represents the success case. So like a user struct and the variable that represents the error case, an error interface. You’re returning one thing and then you check whether or not it is this or it is that which basically results in safer code because you can’t forget to check the error. So in Go for example, sure I returned an error variable and sure it’s pretty obvious that you need to check the error variable because it’s sitting right there in the line of code that you’re looking at. So it’s a lot more explicit than we talked about before JavaScript or a Python, but you’re not forced to unwrap that value, right? You’re not forced to check it by the compiler. And to me that is a weakness. I’m a big fan of static typing. I’m a big fan of essentially my tooling stopping me from making these sorts of mistakes and better support for Enums. So I wouldn’t have to do as much at runtime would be nice. Yeah, that’s the big one.
Philip Winston 01:05:20 How about in the larger ecosystem? Are there any libraries or frameworks or projects that you’re looking forward to?
Lane Wagner 01:05:29 I’m really excited about SQL C, I don’t know what that roadmap looks like, but I definitely want to plug it. For a long time I was using Gorm or Go ORM as the way I interfaced with Postgres databases from my Go applications. And I don’t want to badmouth a project. It’s a great project and I have a lot of respect for it, but it’s a very simple ORM. If you have a very small, very simple application, I think you can get far on Gorm but there’s a lot of disadvantages. you’re not getting type safe SQL for example. You kind of are writing your SQL queries in raw strings and then you have to actually run them to make sure that they’re valid SQL. SQL C is so interesting to me, and I haven’t seen an analog in another language either which I’m sure they exist.
Lane Wagner 01:06:15 I just haven’t seen them. The way SQL C works is fascinating. You write a raw SQL query, which I’m a big fan of because SQL and I like having that full expressive power and being able to use all the features of my database, but it basically reads that query and compiles it into Go code. So then in my application, after I generate the Go code, I can just call my functions to access the database. That was really exciting. We have run into a couple little edge cases, little tiny bugs with it as you can imagine, just thinking about that project, there’s an incredible amount of engineering work that needs to go into compiling something as large as all of SQL into Go. But that’s been my favorite project so far.
Philip Winston 01:06:59 Okay, Lane, thanks so much for taking the time today. Where can people follow you online?
Lane Wagner 01:07:05 Yeah, so I’m at Wags Lane pretty much anywhere. I’m most active on Twitter if you want to just see what I’m talking about. So at Wags Lane on Twitter, if you podcasts, I mean you want to learn more about Go and in in specific backend development. Then the Backend Banter podcast is definitely wanted to check out. I just had you Philip Winston on, couple episodes back, so that’ll be something to check out. And then of course Boot.dev if you’re interested in learning backend development in Python and Go. That’s my current project.
Philip Winston 01:07:34 Okay. Lane, I hope we nudge some people into learning more about Go programming. This is Philip Winston for Software Engineering Radio. Thanks for listening.
[End of Audio]
