Zach Lloyd, CEO of Warp.dev, discusses how to implement and effectively use command-line terminals. Host Gregory M. Kapfhammer speaks with Lloyd about how command-line terminals work and how the Warp terminal uses the GPU and AI to enhance a software developer’s productivity. They also discuss the trade-offs associated with using the Rust programming language to implement a command-line terminal.
- Zach Lloyd’s web site: https://thezbook.com/
- Web site for Warp.dev: https://www.warp.dev/
- Documentation for Warp terminal emulator: https://docs.warp.dev/getting-started/readme
Additional Show Notes
The Terminal emulator (i.e., “the terminal”) is a software tool used by many software engineers. This episode will overview the features provided by terminal emulators such as alacritty, kitty, and iTerm2. Next, the episode will investigate the limitations of existing terminal emulators and explore how Warp, a new terminal emulator implemented in Rust addresses those concerns.
Nat Friedman wrote the following about the Warp terminal emulator:
“Finally, innovation in terminals!”
- Why did the Warp team pick Rust for the implantation of Warp? What are the trade-offs?
- How did you implement the Rust primitives and protocols for the Warp terminal?
- How does Warp integrate with existing shells such as bash, zsh, and fish?
- Overview of features provided by Warp: command blocks, cursor positions, and completion menus
- How does the Warp terminal integrate generative AI to enhance developer productivity?
- How does Warp support integrated documentation, notes, and programmer collaboration?
Transcript brought to you by IEEE Software magazine and IEEE Computer Society. This transcript was automatically generated. To suggest improvements in the text, please contact [email protected] and include the episode number and URL.
Gregory Kapfhammer 00:00:43 Welcome to Software Engineering Radio. I’m your host, Gregory Kapfhammer. Today’s guest is Zach Lloyd, the founder and CEO of Warp, a startup dedicated to helping software engineers quickly create great software. Prior to Warp, Zach was a principal engineer at Google working on Google docs, sheets and slides. He was also the interim CTO of Time and the CTO and co-founder of a startup called Self-Made. Zach, welcome to Software Engineering Radio.
Zach Lloyd 00:01:15 Thank you for having me here, Gregory. Excited to chat today.
Gregory Kapfhammer 00:01:19 So today we’re going to be talking about the Warp command line terminal. I’m really excited about a lot of the cool features that are in the Warp terminal and I really look forward to diving into them. With that said, I hope that you can share a few of the key features that the Warp terminal provides and then explain why you decided to build it.
Zach Lloyd 00:01:41 Sure. So the short answer is we decided to build it because the terminal is a, I think a very important tool to a lot of developers. Most developers, it’s something that they use every day, but it’s also something that hasn’t really been rethought or explored from first principles. I think almost ever? The version of the terminal developers use today is basically identical to what they’ve used for the last 40 years. And from a usability perspective, we thought that there were ways we could improve upon it. So some of the big improvements that users will see right out of the gate is when you use Warp, we actually put structure around the output in the terminal. So we have this concept called blocks. So when you run a terminal command, we group the command and its output together into a block and you can navigate your terminal on a sort of command-by-command basis and you can take easy actions on those like copy, paste.
Zach Lloyd 00:02:39 You can search just within one command output, any command output you can easily share. So that’s one big thing that people will notice. And then another big thing is the way that input works in Warp is very different from a normal terminal. So in a normal terminal, the mouse doesn’t work for example. And so in Warp the terminal input works a lot more like how a code editor works like VS code or something that. So you get things like syntax highlighting, you get command completions out of the box, it’s fully mouse accessible. And so those were two of the really key user experience things that we launched with. And then there’s also a whole bunch more that I hope we get to talk about. But yeah, those are some of the basics.
Gregory Kapfhammer 00:03:18 Okay. So it sounds like Warp has a lot of cool features that standard terminal Windows don’t have. Before we start unpacking the many technical innovations that are in Warp, I’m hoping we can define some key concepts. So let’s start with terms like console, terminal, command line and shell. Zach, can you dive in and give us a brief definition of those types of terms?
Zach Lloyd 00:03:44 Sure. So they’re all related. So console typically means the actual sort of hardware terminal I guess, which is maybe the starting spot. Typically when people are using a terminal, what they’re actually using is a terminal emulator, which is an actual Gooey app that you run. Could be running it on your Mac, your Linux Windows. And what that app does is actually emulating the behavior of the physical piece of hardware that people use to regularly use. And so the terminal has the Window and it has, it’s basically a character rendering device and a character input output device. So it’s the program that takes input from your keyboard one character at a time and then sends it actually to the shell, which is another one of the terms you mentioned. The shell is basically the actual command interpreter. So the input to the shell is text, it parses that text, it figures out what command that text is launching and then the shell launches applications.
Zach Lloyd 00:04:46 It then takes the output of those applications, which again if you’re in this command line environment are character based and sends them back to the terminal and the terminal is responsible for rendering the characters on the screen. So at really simple level, it’s the terminal is where the characters are rendered and the characters go into the shell. The shell is what understands those executes programs and returns them back. And then the final term you mentioned is the command line, which I think is kind of more colloquially the whole interface around doing these text-based apps or at least that’s how I think of it. So sometimes you’ll just talk about the command line. I’m developing a command line app and what that means is it’s an app that runs with a text-based interface so it runs through the terminal and shell, does that make sense?
Gregory Kapfhammer 00:05:31 Yeah, that makes a lot of sense. I want to pick up on two things that you’ve said so far. First you mentioned that terminals have been around for decades and second I heard you frequently talking about how terminals are text-based in nature. Can you comment quickly? Why are text-based terminals so exciting and powerful for developers to use?
Zach Lloyd 00:05:54 Yeah, it’s a great question because there aren’t many other 40-year-old apps that people still regularly use. What I would say is that terminals and text-based apps have a bunch of inherent advantages compared to other types of apps. So, for one thing they’re the easiest type of app to write in a lot of ways. Most developerís first app that they write, their hello world is something that outputs text to a terminal. After that it’s just very easy to process text as an input. So there’s the standard format of, I forget what it’s Arc V and Arc C if you’re writing a C app, it’s it just passes in the parameters via text. It’s just a much easier way to take data input than to write something that handles mouse events and renders a rich UI.
Zach Lloyd 00:06:41 So there’s this advantage of these apps being easy to write. I think a second advantage is they’re very fast to run and they’re very lightweight. So if you’re developer who likes to use the keyboard a lot sort of a lightweight execution environment for running these apps. There’s even more so from a UI surface standpoint, I think it’s very easy to write a command line app that is sort of infinitely configurable just in terms of flags, whereas you start to run into the limits of what you can do with a Gooey app. You have to make a lot of choices around are you going to have this button or that button, this form or that form. In a command line app, it’s effectively infinite, you can just keep adding flags. So there’s that value. There’s the value in terms of these apps like text being a sort of universal interface where you can take the output of one of these command line apps and very easily pipe it as the input to another one.
Zach Lloyd 00:07:32 And so these apps are sort of naturally composable and you see this with Unix pipelines, it’s really powerful concept for how do you define very minimal apps that work well together. Just having text as the interface is really good for that. And then I would say one final thing is like, a lot of the way the internet works, a lot of what’s running internet servers when you think about it are things that are like docker containers that are essentially running a bunch of command line apps and those servers, they don’t really have, they’re not actually not running on Gooey and so if you want to interact with them you sort of need to be using a command line interface. I think I know that was a lot, but I do think there’s a lot of these inherent advantages to using text as the interface that make this interface still super relevant for developers today. And it’s an ubiquitous tool every developer uses.
Gregory Kapfhammer 00:08:20 Yeah, I really resonated with what you said. For me the command line terminal is also super relevant and I use it in a wide variety of different workflows. Could you share with us what are some common workflows that you adopt when you’re using a command line terminal?
Zach Lloyd 00:08:37 It’s a great question. So it’s a really, really flexible tool. It’s almost like I sometimes use the analogy of a spreadsheet for knowledge workers. You can do so many different things with it. I would say just to name a few very common ones. So people will do a lot of source code management is very common. Some developers, it’s still popular as an editor. So people will still actually run Vim and Emax and use the terminal for that. Attaching to remote machines is a very popular thing. So it’s like terminal like I said, if you want to access a server and production or do some sort of debugging on a Raspberry Pi, typically those machines will not have Gooeys and so you’ll use the terminal for that, internal tooling. So for developers, I would say command line apps are still the most popular by far way of writing internal tools and scripts.
Zach Lloyd 00:09:25 And so that could be whatever a company wants to do. It could be like pulling information for data analysis, it could be orchestrating GitHub actions and different workflows that always starts with work in the terminal. And so I guess one of the things a lot of companies that are cloud providers will, will build and ship CLI is one of the primary ways for doing interactive work with whatever their cloud thing is. So GCloud or AWS, a lot of the, if you’re going to do interactive work are done through the terminal.
Gregory Kapfhammer 00:09:58 So you mentioned many of the ways that I use a terminal on a regular basis. In fact I even use a terminal to edit all of my text because I use a command line tool to edit text like Neovim or Emax.
Zach Lloyd 00:10:11 Yep, exactly.
Gregory Kapfhammer 00:10:12 So with those things in mind, before we start talking about the specifics of Warp, there may be some developers who are listening who don’t normally use a command line terminal. So can you tell us a little bit, what steps would a developer normally take if they want to download and install and set up and configure a command line terminal?
Zach Lloyd 00:10:34 Yep. So depends on what platform you’re on. So typically the platforms will actually ship with a default terminal. If you’re on Mac, there’s the terminal app you’re on. I think most Linux machines will ship with a default console, but that’s a very, very bare bones experience I would say. So if you want to get it to a point where you have a more ergonomic or efficient experience, you’re typically going to download and install a third-party terminal. There’s different ones that are popular on different platforms. On Mac, there’s not just Warp but there’s iTerm, there’s some other terminals I think we’ll talk about later Alacrity and we term(?), so there’s a variety. So the first step is to choose what terminal you want to use and then depending on the terminal, there’s typically a bunch of setups that you then go and do in your shell and actually and maybe in the terminal too.
Zach Lloyd 00:11:24 So you might want to choose a theme. There’s a lot of open-source stuff out there that’ll make your shell experience better. This is particularly relevant if you’re not using Warp, like in Warp, one of the things that we try to do is actually really minimize the amount of config that you need to make the terminal usable. But if you’re using a standard terminal, you might want to install something that helps with auto completions or helps with themes or you might want to change your shell. Often the default shell might be Bash but you might want Zsh or Fish(?). And so there’s a bunch of configuration to do to figure out how do I make this thing usable for me? You might want to change your prompt, that’s a very common thing people do so they can see what their current context is before they run a command.
Zach Lloyd 00:12:07 So, and then there’s tons of resources on the internet and honestly some of the most popular open-source projects on GitHub in terms of stars are things about making your terminal better. So there is a ton of information out there if you want to get a terminal going and then set it up. I guess one other thing I didn’t mention is IDEs will often ship with a default terminal, so that’s another option. So if you’re using VS code or one of the JetBrains things, there’s an inbuilt terminal that comes with it. Although if you want to get my personal preference and people will vary on this is to have a dedicated terminal app but you can also use it within the IDE.
Gregory Kapfhammer 00:12:44 So thanks for your responses. I think you’ve done an awesome job at setting the stage. Let’s dive into some of the basic features that are provided by the Warp command line terminal. A moment ago you talked about something like Bash, which is a shell. Does Warp provide its own shell or does Warp integrate with other shells like Bash? Can you give us a few more details?
Zach Lloyd 00:13:09 Yes, so as of today, Warp does not have its own shell. Warp is compatible with Bash, Zsh and Fish(?), which I think are the most probably the three most prevalent shells in use these days. There are other shells that are a little bit less frequently used that we could support but don’t currently support. And the general experience with Warp is that if you download it, open it up, it will use your default shell config and hopefully basically just work without you having to do any extra work to get it set up to work with your shell.
Gregory Kapfhammer 00:13:42 So the features that you’ve described in Warp sound really cool. What steps would a developer take if they want to install the Warp terminal? Do they go to warp.dev, do they use Homebrew? Do they take some other approach? Can you give us some more details Zach?
Zach Lloyd 00:13:58 So those two options are exactly right. So you can go to warp.dev and you can download the DMG and install it you would any other Mac app or if you want to use Brew it’s simple too. It’s brew install dash dash cask space Warp and that will also install Warp just any other brew formula.
Gregory Kapfhammer 00:14:17 Okay. So I know some developers may already be using their own terminal window and they may have even adopted the use of something called a terminal multiplexer like SCREEN or tmux. As we do more of a deep dive into Warp, does it provide its own terminal multiplexer, or does it integrate with existing multiplexers? How does that work Zach?
Zach Lloyd 00:14:40 As a Warp user, you have a couple choices and just to sort of describe what a terminal multiplexer is, maybe for a second. So terminal multiplexer like tmux is, it basically lets you do things like session management and pane management within a terminal window, which is useful if you want to sort of multitask across different terminal sessions. And so for Warp, the short answer is you can use tmux so you can use SCREEN, if you use them though you currently lose a bunch of the stuff that makes Warp different as a terminal. So you sort of fall back to the behavior of a traditional terminal where you don’t get blocks or input editor. So it’s not an awesome experience just to be honest. We’re trying to figure out how to make it better. The other option if you want that type of functionality is you don’t have to use a multiplexer like Warp and natively supports having split panes and tabs and multiple sessions that you can view at once within the terminal, but it doesn’t yet support some of the more advanced functionality that tmux has around session management. And it’s actually, I think it’s one of our top 10 issues on GitHub to offer better support for that.
Gregory Kapfhammer 00:15:45 Okay. So that sounds really interesting. You’ve mentioned on more than one occasion the idea of the command block feature that Warp provides and I’m understanding that that feature would be turned off if I used tmux, but let’s say I wasn’t using tmux. Can you tell me a little bit about what the command block feature is and how it works?
Zach Lloyd 00:16:08 Sure. So the basic idea is that most, not all but most of the way that people use the terminal is as a sort of Rappel, so a redevelop print loop. And what that means is you’re either in the state of entering a command or actually after you enter command it tends to produce some output and in most terminals the terminal just has no sense or understanding of what the structure of the underlying session is. So the terminal will typically have, when you run a command it’s just filling up a buffer of text and there’s no demarcation between what command is attached to what text, which is one big buffer. And so for instance, if you want to do something copy the output of one of your prior commands in a normal terminal, you have to sort of take your mouse and select and scroll to just the right region and capture just the right characters and then you copy it.
Zach Lloyd 00:17:03 And same thing if you want to do anything that is oriented around a single commands output that is very hard in a normal terminal. In Warp what we do is, we automatically understand a little bit more of the structure of what’s happening. And so essentially if you were to look in the UI, we group, command with the output that it generates and that gives you some really nice sort of UX benefits around things I can very easily navigate my command, my scroll history, a command at a time for instance. If I want to scroll right to the start of a command or right to the out end of a commands output, I can do that. If I want to search just within one command’s output, I can trivially do that and that’s really useful if you’re running the same command over and over again.
Zach Lloyd 00:17:46 But just want to search for a log message and the last version of it, if I want to share something I did, we have a native sharing feature that’s based on blocks. So you can easily get essentially a gist or a perma link to anything you did in the terminal and you can send it to people on your team. And so the way I view this is as a new UX primitive almost similar to the block style thing that you would get in a Jupyter Notebook or another structured document format that makes working with terminal outputs, I think just a lot more effective and productive.
Gregory Kapfhammer 00:18:19 Yeah, I’m glad you said Jupyter Notebook because in fact that’s something that I thought of when I was using the command block feature in Warp. Can you give us a concrete example? How would a command block help me if for example, I was testing or debugging my program, how would I use blocks?
Zach Lloyd 00:18:36 A good example would be, let’s say you’re doing successive runs of the same command that’s producing different log output each time and you want to just search in the most recent invocation of that command’s output to see if a certain debugging string is there. So what you do is you would, every block we have has a little header and you would just sort of right click and be okay I want to search within the contents of this block and then your search is automatically scoped to the last command notification. And then going further, let’s say okay, I can’t figure out what’s going on here so let me see if someone else can look at this debug output who I’m working with. With one click you can grab a link to the block, send it to your teammate who can look at the output in their browser. They don’t even have to install Warp and maybe they can tell you, oh I see what’s going on, this is what you need to do to debug it further. And so that’s an example of how having that extra structure can be really helpful when working with terminal outputs.
Gregory Kapfhammer 00:19:34 Yeah, I can see that being super useful to be able to share the output of a specific block with a member of my team and say–hey can you figure out why this test case is failing? That would be something I would really want to be able to do as a developer.
Zach Lloyd 00:19:48 Yeah, it’s cool.
Gregory Kapfhammer 00:19:50 So I know that you mentioned before the idea of a session management and you talked about various window management features that Warp has. Can we do a deep dive into session and window management? What kinds of features does Warp have in that area and then how does it compare to the similar terminals that might maybe for example, come on macOS
Zach Lloyd 00:20:13 We offer, I guess two big features in Warp. So one is multi tabs and so, if you’re to look back at terminals, I don’t know, 10, 20 years ago it was just you had a single terminal window that you worked in. Within Warp you have the ability to have multiple sessions and switch between them. And the ways that you can switch between or create new sessions in Warp is at the window level. So you can always do command end and get a whole new terminal window and some users like to work like that. Within a window you could have tabs and so, think of like Chrome tabs essentially where each tab is a single terminal session or within a tab you can have split panes and so you can actually have side-by-side sessions. I wouldn’t say there’s any like particularly groundbreaking about Warp, at least right now in this respect.
Zach Lloyd 00:20:56 Itís like sort of on a par with some of the other terminals on Mac that ship with this type of session management, it is all keyboard accessible so itís if you want to quickly navigate between ones or others, you can do that. We have another feature that’s actually pretty cool and popular, which is something that we call the official name is like a hotkey window, but the colloquial name is a Quake mode Window. And what that does is it lets you assign a sort of global keyboard shortcut, which no matter what application you’re using, you have the ability to get a Warp terminal to sort of modally pop up in front of you. And, it’s called this because I think in the video game Quake there was this sort of same terminal that you could pull up anywhere. And so we have that too. So yeah, those are some of the cool window and tab management functions that we have in Warp right now.
Gregory Kapfhammer 00:21:45 Yeah, I’m familiar with the feature that you had talked about and I have to say it is cool to be able to have a Warp terminal zoom in regardless of the program that I’m currently using. So in summary we said that like for example you could have maybe one pane in Warp that is related to testing and I could have another pane that’s related to running a compiler and I could have another one of the panes that was related to for example running a server or something of that nature. Are there any limits in terms of the way that I lay out my panes or lay out my tabs or is Warp configurable in that fashion?
Zach Lloyd 00:22:23 It’s totally configurable. And then we have actually one other thing which I should have mentioned, which is this concept of a launch configuration. And so you can actually create a YAML file which says, I think your example is a good one. It’s–hey I know that when I’m working on this project I always want one window for the client, I want one window for the server, I want one window for connecting to production. So you can define that in a file and then you can launch that configuration as sort of one atomic thing so that you can quickly get the setup that you want.
Gregory Kapfhammer 00:22:54 Okay. So is there a way for example, that I can have one setup when I’m building one program and another setup when I’m building another program?
Zach Lloyd 00:23:03 A hundred percent, yeah. And you can define those as two separate, launch configuration files and then, in Warp we have this feature called the command palette, which you can push command P wherever you are. And one of the options from there is launching, one of these launch configurations and it will just bring up a window that has the exact pane sizes programs running that you want, so that you can waste less time getting set up and you can context switch much more easily.
Gregory Kapfhammer 00:23:31 That sounds really cool. So we’ve talked about a number of the key features that Warp provides. I remember a while ago you talked about the idea of Warp offering a completion menu. What is a completion menu and how does a developer use it?
Zach Lloyd 00:23:46 Yeah, so I’d say the command line has a bunch of awesome attributes around its UX being very fast and extensible, but one of the challenges around the command line UX is you have to remember how these infinitely flexible commands are actually used. And so documentation and discoverability around how to use commands is a challenge for this kind of interface. So one of the things that we offer is we help you complete commands and command completion is a standard shell feature that you will see in all sorts of shells, but in Warp it’s actually quite different because we’ve implemented it at the level of the terminal. And what that means is you get essentially a visual completion menu which, if you’re an IDE user, if you use VS code or JetBrains we’ll be familiar with you. And so we’re able to, as you like start typing a command, if you push tab and you can configure this, but as you push tab, we bring up these visual completion menus which give you an ordered set of our best guess at what the next thing that you might want to enter is.
Zach Lloyd 00:24:53 So if you do like Git space and then hit tab, you get options of all the Git commands. If you start typing CH then it filters down to get checkout. So it filters as you type to the mostly suggestion. And then another sort of unique thing about Warp that we offer that other terminals don’t offer is we give you actual documentation as you type around what these commands mean. And so if you have something with obscure flags and you’re, remind me what dash dash force does or something that, then we can actually in line show you, okay, this is what this flag means and you can bring that documentation back up whenever you want it to. So if you can position the cursor and hit command-I and you’ll get the documentation, you can hover and get that same documentation.
Zach Lloyd 00:25:40 So I mean that’s to my knowledge, a totally unique, feature at the terminal level in Warp and so that’s I think a real user advantage. And then you don’t have to configure this in Warp either. So, typically if you want completions and I think gets a good example, you have to put something in your RC file in one of your startup files that will source those completions and make the shell aware of them so they’re available and Warp all this stuff to ships out of the box, which is very nice.
Gregory Kapfhammer 00:26:10 So the features you’re describing sound really useful. If I am already using, for example, Fish and Fish provides completion, does that completion work alongside of the completion that Warp provides? How does that work?
Zach Lloyd 00:26:25 So Warp typically takes it over and this is a, I would say it’s a trade-off. So we try to do the best that we can shipping out of the box with completion and we ship with 400 or 500 tools, I forget the exact number, but if you’ve configured things at the shell level because in Warp we sort of take over the entire shell input editor, we aren’t able to use those completions. It’s a thing that we’ve considered, and this is maybe getting technically in the weeds of is there a way to use the shell’s completions with acceptable sort of performance characteristics and mix them in with our own? The Warp ones tend to be super, super low latency because it’s all natively bundled into our app, whereas when you’ve kind of like go out to the shell for questions, it becomes a little bit slower and then the shell ones don’t have the same spec format as ours. They don’t give you documentation. And so as of right now this is a real trade off and it’s definitely something that users of Warp give us feedback on and we’re trying to figure out what is the best possible solution. But it’s a great question.
Gregory Kapfhammer 00:27:34 Okay, so right now it seems Warp basically takes over your command line completion. If I start running a text editor inside of my Warp terminal window, does it also take over the completion of Neovim or Emax? What happens in that case?
Zach Lloyd 00:27:52 In that case everything works just as it normally would. So in general, and again this will be a little bit technical, but I think that’s okay, there’s a couple modes of running the terminal. There’s the Rappel mode where Warp is providing the command line editor and doing things like completions and then there’s the mode that you go into when you run an app like Vim or Emax, which is actually using a different character buffer, which we call it the alt grid or the alt screen. And in those types of apps, good examples like Vim, Emax, talk (?), those fully interactive like TUI Text User Interface based apps. Warps behavior is exactly the same as a normal terminal. And so a lot of people use Vim plus Warp Emax plus Warp and that all just works exactly as you would expect it to.
Gregory Kapfhammer 00:28:40 Okay, that makes a lot of sense. I often do development of my own command line applications. So if I’m using the Go programming language or Python and I’m building my own command line interface, does Warp integrate with it in terms of providing completion of its command line arguments.
Zach Lloyd 00:29:00 So sort of, the way that that works is there’s an open-source repo that we use that has the specs for all of these different completions. It’s actually run by another startup, which is called Fig, where they’ve defined a spec that’s in, it’s in Json and it gives you the ability to say, okay, here’s what this command looks like, here are all the flags, here’s what they mean. We then take that repo and translate it into something that we can use, which is in Rust if you want to support new tools, at least for now until we have a sort of more fleshed-out solution on the Warp side what you would do is you would make a submission to that repo and it would eventually make it into Warp. The longer-term solution for us is to actually create a sort of spec where you don’t have to publicly publish your command line. A lot of use cases for this type of thing are people’s internal tools or private tools and I think the right solution is something there is a spec that you can publish and just limit the scope of without putting it out fully open source into the world and maybe just use it yourself even if you want. And that’s something that’s on our roadmap. Actually this coming quarter we’re going to be working on the start of this.
Gregory Kapfhammer 00:30:15 Yeah, what you’re describing would be really helpful to me as a terminal window because I’ve implemented a bunch of my own programs and they’re not in that repo and I may not even want to publish the details about my own programs. So if there was a way that I could make Warp aware of my own command line applications, that would really be awesome.
Zach Lloyd 00:30:36 Thatís a hundred percent how it should work. And that’s on our roadmap. We’re just not quite there yet.
Gregory Kapfhammer 00:30:41 Okay. So we’ve talked about a lot of the features that Warp provides and I want to dive into some of the more advanced features of the Warp command line terminal. So there’s a whole bunch that I’d like to talk about, but let’s start with the ways in which Warp uses the GPU or the graphics processing unit. Can you talk a little bit about why your terminal uses the GPU and what are the benefits associated with using a GPU?
Zach Lloyd 00:31:09 The primary reason that we use the GPU is performance. When you’re a developer using the terminal, there’s a lot of reasons why you might want to use the terminal, but one of the things I think that’s absolutely essential is that the terminal performs really, really well and you can think of different dimensions of performance. There’s a dimension of just pure input output operations. So that’s how quickly do characters go from the terminal into the shell and then more importantly typically from a shell program back into the terminal. But there’s also a big performance that I would think of more in terms of graphics performance, almost things that if you’re a video game developer you might care about. So it’s what’s the frame rate at which we can render terminal changes and how fast can you scroll around your terminal?
Zach Lloyd 00:31:56 And these are operations that become really important if you’re, if you’re doing things that work with a lot of text, the sort of non-GPU way of doing it is using a more CPU or software based rendering framework. The performance of the GPU based way of doing it is, is certainly better. What it means for a terminal like Warp is, that we are interfacing directly with the graphics APIs on Mac and what that looks is that you’re actually, you’re writing code in a shader language on Mac. It’s like the shader language is metal, which is somewhat similar to OpenGL and it’s a very different style of programming. You’re basically writing graphics operations that have to be able to run in parallel and you’re running shader code that gets shipped to a GPU. But yeah thatís the basic, sorry, long-winded answer here is like it’s faster .
Gregory Kapfhammer 00:32:50 Okay, so it’s faster and I’m assuming it’s faster from the perspective of user perceived latency. When you use the GPU for doing the rendering of text in the Warp terminal, does it also reduce things like the system load? How does it work in terms of things beyond user perceived latency?
Zach Lloyd 00:33:14 So I think user perceived latency is the biggest one. I haven’t written a text rendering thing that uses the CPU so it’s a little bit hard for me to sort of compare what it would be like, but generally I think we’re doing it in the most efficient way. Because it is not pegging the CPU in pegging system load as you do this. It’s doing it on a totally different piece of hardware. And yeah, I mean these are generally I think this is the most efficient way that you can do the graphic stuff is by going directly to the GPU and not forcing your CPU to do a bunch of math to figure out how to draw paths and render glyphs.
Gregory Kapfhammer 00:33:53 So you mentioned metal and we’re going to talk about metal later in the show because I think it’s important to dive into how you implemented Warp in Rust and how you use Metal on macOS. Before we do that, can you give an example of what a terminal would be like if it didn’t use the GPU? What would we forego if Warp didn’t use the GPU?
Zach Lloyd 00:34:17 So I’m winging it here slightly because I haven’t built a non GPU based terminal, but what I think you would do instead is you would use a set of higher level UI libraries where you would do things, it could be a Vector style API you could be drawing lines and curves, you could be telling it to print out texts directly, but ultimately the, there wouldn’t be such a clean set of graphics programs that are, that there would be a lot of processing on the actual CPU to figure out where the pixel should be painted on the screen. I’m trying to think of a good example here. if you were to do this using just the straight up Apple APIs, I don’t think that everything would be graphics except GPU accelerated, meaning if you were to use an NSV and an S-label and those sort of higher level UI components, I don’t think you end up with a GPU accelerated thing all of the time because there’s a lot of intermediate processing on the CPU in order to figure out how to, render everything.
Gregory Kapfhammer 00:35:19 Okay, so that makes a lot of sense. It sounds like overall when Warp uses the GPU I’m going to see that my terminal window has a better frames per second. Okay. It’s going to reduce the load on my CPU, which can be used for other things, so overall it sounds like using the GPU gives me a better experience as a user of the Warp terminal window
Zach Lloyd 00:35:44 For sure.
Gregory Kapfhammer 00:35:45 Okay,
Zach Lloyd 00:35:45 Those are the exact two benefits that you mentioned. Yeah.
Gregory Kapfhammer 00:35:48 So what I’d like to do now is to turn our attention to some of the innovative ways in which Warp uses Artificial Intelligence. I learned that Warp integrates AI directly into the terminal. My big question is what in the world does it mean to integrate AI into a terminal window?
Zach Lloyd 00:36:08 It’s a great question and it’s, I think it’s an important question because of the amount of hype there is around AI right now, but in the terminal it’s actually super, super useful. So for us with AI, there’s sort of two main ways of interacting with it. The first way is it allows you to go from natural language to a command. And so this is super useful. So it’s, say I want to write a command that allows me to find all Rust files greater than a hundred lines long and then extract a certain term from them or something like that. I don’t know how to write that command. I could Google, I could leave the context to the terminal, I could ask a friend maybe who knows the way it works better, but the easiest thing by far is to just ask the terminal, hey, this is what I want to do, can you help write the command for me?
Zach Lloyd 00:36:58 And more often than not with these large language models, the command that comes back works, which is incredible. So you go from a very obscure way of expressing your intent, which is these the commands instead of having to actually write them out that, you can simply just say in natural language. So that’s one entry point I think is very cool way of using it. The second one is like the more conversational mode. And so this is more similar to how you might use chat GPT or another one of these LLM chatbots. The thing that makes it super useful in the terminal is the integration with command inputs and outputs. So for instance, if you run some GCloud command and it produces some permissions error and you’re like, oh no, what do I do to fix that?
Zach Lloyd 00:37:48 it’s very, it is trivial and Warp to be, okay, let me ask Warp AI how do I fix this thing that went wrong? So you can go right from an error to a conversation about how to fix it and that conversation will often produce new commands and those new commands you can directly run. And so you have a very tight feedback loop when you’re trying to figure out how to solve problems in the command line. And so it’s kind of an awesome use case for it and it’s made even better by the fact that these LLMs are all, they speak text and the terminal speaks text. So it’s very simple to go right from terminal input output into the LLM and back. And then these LLMs, they’re often trained on a lot of sort of command line data. If you look at Stack overflow and look at the internet, there’s a ton of training data around API documentation on how to actually use these tools better. So it’s a really good application of technology for making the command line better.
Gregory Kapfhammer 00:38:43 So what you’re saying sounds really compelling and I see what you’re getting at, because of the fact the terminal window is text centered and these AI large language models are also text centered. It’s really cool to pull those two things together inside of the Warp command line terminal.
Zach Lloyd 00:39:01 Absolutely. It works really, really well. We survey users about it, this is pretty game changing and it just saves a whole context switch and lets you get to, it just lets you get to the answer that you want faster and with less of a headache and the conversational stuff is extra cool because it remembers the context as you build up a conversation. So if you can be like, it knows that I’m trying to achieve some task in GCloud, you can use prior questions to inform the next answer that it gives. It’s very powerful.
Gregory Kapfhammer 00:39:35 Can you tell us a little bit about how you actually implemented Warp AI? What are you using to create this powerful feature?
Zach Lloyd 00:39:44 Yeah, so right now Warp AI is built on top of Open AI, their chat GPT model. And so the main challenges on our side were, how do you engineer the prompt correctly so that you get back meaningful results for the command line? How do you find the right integration points into the UI for the AI? And so, you want to balance making it discoverable and making it useful with, you don’t want to annoy people. You don’t want the Microsoft Clippy thing, I don’t think at least maybe that was before its time and the right way to do it, but it’s I don’t think you want AI things popping up everywhere in the terminal, especially since it’s a tool people do a lot of focus work in and so you have to balance those things.
Zach Lloyd 00:40:29 And then a third thing for us is you really have to balance privacy and security. So it’s our AI at the moment, you have to opt in to use it every time you want to use it just because a lot of private information flows through the terminal and currently the model is not a local model, which it would be interesting to see if we could get to a local model, but the model is a cloud-based model and so we want to be very careful people being clear if–hey this information is being sent off the terminal, people should know that and choose to be doing it.
Gregory Kapfhammer 00:40:57 So you mentioned that right now you’re using a cloud-based model. Can you give us a concrete example? If I have a compiler error or if my program crashes or if it produces output that I think would be wrong, how can Warp AI help me in those types of circumstances?
Zach Lloyd 00:41:15 Yeah, so compiler error is easy. This is a great example of how some of our features tie together. So say you try to compile and you get a compiler error and so let’s say that error then would be contained in one block. And so we know that the block has an error code in it and so that you can go right from the block to something called Ask Warp AI. And so you just, with a keyboard shortcut or a click you’re–hey explain this error to me. And then Warp AI comes back with a–this is what the error means and here’s a suggested fix. So that’s one concrete use case. Another concrete use case is less around writing code and more doing something in production. And so a real thing that I tried to do was I wanted to figure out how to access logs and tail logs in the terminal from one of our production servers that’s running on GCloud.
Zach Lloyd 00:42:02 That’s not a one command thing, that’s a first I need to list all the instances, then I need to find the name of them, then I need to make sure I’m like correctly set up to access things through the firewall. And so it will do a guided sequence of commands with you to get you to the right point, including answering if you run a command and that instance doesn’t exist, it will try and give you ideas on how to debug getting to the right thing. So super powerful for those types of production use cases.
Gregory Kapfhammer 00:42:34 Yeah, I can imagine that that would be incredibly helpful for me as a developer. Can you comment quickly, if my program crashes and there’s a stack trace in the terminal window, can Warp AI help me to understand that stack trace?
Zach Lloyd 00:42:49 A hundred percent. Yeah, so you just go ask Warp AI and if you want to ask a particular, you can either just be, tell me what’s going on, or you can ask a particular question like why is this having a null point or exception or why did this SEC fault or, and it will do its best. Now I will say these, the AI models have limits, and they are prone to hallucination sometimes, so use it with care. I’ve had situations where it will refer me to documentation websites that don’t exist or confidently tell me to do something that isn’t quite right. So I would certainly use your judgment but as an aid to solving things quickly there it’s, it’s a phenomenal technology.
Gregory Kapfhammer 00:43:34 So a moment ago you mentioned the idea of using a local first model. Before we talk about the local first approach, I wanted to clarify one quick issue. Is Warp AI actually sending to the cloud the source code of my application or is it exclusively using the text that it sees in the terminal window?
Zach Lloyd 00:43:55 It’s exclusively using what you directly give it on a per invocation basis. So there is absolutely no background sending of someone’s context or data in our current implementation. Now that said, I actually think the AI would be significantly more powerful if we were able to provide more context, but we’ve taken a very conservative approach, again because of privacy and security concerns. So right now to answer your question itís you basically you can see in our sort of input area exactly what’s being sent and there’s no more than that being sent.
Gregory Kapfhammer 00:44:34 Okay, thanks for clarifying that. Let’s dive into the idea of a local first model. What is a local first model and what are the trade-offs associated with using it either as a substitute for a cloud-based model or perhaps in conjunction with a cloud-based model?
Zach Lloyd 00:44:51 Yeah, so local first just means we wouldn’t be using the network at all to sort of generate suggestions or AI responses. Thereís different ways that you can do this. The sort of trade-offs I think are in terms of like model size, sophistication, disk space resources to run the sort of inference step. And so then there’s also on the plus side there’s increased privacy security like you have confidence stuff’s not leaving your computer, it works even if you don’t happen to be connected to the internet, which is something that Warp, we generally strive for is Warp should always work. You shouldn’t have to have an interconnect internet connection to use your terminal. We want graceful fallback with that. But today it’s you, you need an internet connection to use Warp AI. And so I think there could be benefits around latency as well.
Zach Lloyd 00:45:44 Right now we have network latency, although I would say most of the actual latency is in the inference, the model, the actual performance of the prediction on the model side, not the network, but having something that’s local could be really, really good, especially for super highly latency sensitive operation. So for instance, if we wanted to do something like predict your next command as you’re typing, that’s a very latency sensitive thing where you don’t want three seconds to pass as you’re typing to get the prediction you want sub 100 milliseconds worth of latency so that it seems it’s just happening in real time. Those are some of the trade-offs.
Gregory Kapfhammer 00:46:24 Yeah, what you’re saying makes a lot of sense. Ultimately, are you considering some type of hybrid approach where sometimes you use a cloud-based model, other times you use a local model? What’s on your roadmap Zach?
Zach Lloyd 00:46:37 Yeah so, I mean we want to work backwards from what’s going to be best for users. Yeah, it could be some sort of hybrid, it could be that even there’s a third option which is a self-hosted model for organizations that’s cloud-based but not on using open AIs, public APIs. And so there’s a lot of different ways that we could do this and we’re still, I’d say we’re still very early on, we launched our first iteration of this about two months ago and it’s something new is happening in AI every single day. We are actively trying to figure out what’s going to be the best user experience and that’s ultimately what’s going to drive it. Yeah, it could be some hybrid.
Gregory Kapfhammer 00:47:14 Thanks for the insights that you’ve shared about how you use AI in Warp. I think you mentioned a really good point, which is you want a terminal window that’s best for developers and obviously you made a decision to implement the Warp terminal window in Rust because it was going to be best for developers. Can you share a little bit about the key features of the Rust programming language and why you decided to use Rust for your implementation?
Zach Lloyd 00:47:44 Sure. So Rust is a great language. It’s, I think you typically call it a systems language. It compiles down to, to binary it is, I don’t know, I’m probably going to not describe this in a way that Rust enthusiasts, but it’s almost a, the next generation safer, more ergonomic version of C++ in a lot of ways. It’s not really object oriented in the way that C++ is. Some of the benefits are produces very fast code. So that was again we’re hyper performance focused produces very fast code. It has a, it does this by sort of a unique memory management model where in Rust there is a concept of ownership, of references and sort of moving references that is very different than in most languages. And, Rust consequently is not a garbage collective language, so there’s no overhead of a garbage collector.
Zach Lloyd 00:48:38 It’s also not a sort of manual memory allocation and freeing language. So you don’t have a lot of the issues that can cause, sort of memory safety errors or seg faults or that type of stuff that you get in C++. It has its own very unique model of who owns what value. And as a programmer it actually, I would say it takes a fair amount of getting used to, something that you expect to be an assignment operation is often more of a move operation in Rust where you’re moving ownership and so you do spend more time, I’d say fighting the compiler than you do in most languages. But once you learn it, I think it’s actually a very ergonomic language to work in. It has great community support and it has great cross platform support, which is another really key thing for us.
Zach Lloyd 00:49:30 So you can compile Warp apps for most platforms including the web platform, which is, I think maybe we’ll talk about this in a bit, but it is a platform that we really care about supporting at Warp. And so I would say of all the languages maybe besides TypeScript but maybe even better than TypeScript, Warp has a compilation to web assembly, which is a path that we want to use to bring Warp to the web. And so we’ve had an awesome experience using it. It’s a great programming language.
Gregory Kapfhammer 00:50:01 So we’ll talk about using Warp in the web browser in a moment. It sounds like one of the things that you’re saying is that there’s a learning curve associated with Rust, but if you want to make a high-performance terminal window, Rust is a really good language to pick.
Gregory Kapfhammer 00:51:09 Yeah, what you’re saying makes a lot of sense. I remember earlier you mentioned that you used Metal, which is the macOS GPU API. Can you talk a little bit about how Rust and Metal worked together in the Warp command line terminal?
Zach Lloyd 00:51:23 Yeah, so the way that we use Metal, so Metal is a set of graphics APIs that let you access the GPU on Macs. And the sort of language that Metal works with is a shader language for Metal. So, if you’re writing Metal shaders, it kind of looks like you’re writing shaders in OpenGL or WebGL, you’re writing fragment shaders and you’re writing pixel shaders and you’re writing things that go into the graphics pipeline. And so that’s not written in Rust, that’s written in actual Metal shader language. What we do with Rust is we have a sort of platform abstraction where we’ve actually built our own sort of lightweight UI framework where if you want to be able to say, okay give me a rectangle that has text inside of it with rounded corners, you don’t have to go all the way down to the shader code to do that.
Zach Lloyd 00:52:16 That would be insane to write. You write it. And what we do is, we construct a scene, and this all happens in Rust. The scene goes down to a set of primitives that map one-to-one into the shader language, and it’s actually very few primitives. It’s super cool. So all you really need to render a terminal is the ability to render a rectangle and that rectangle can have borders, backgrounds, round corners, that type of stuff. You need to be able to render glyphs. And so the text rendering is one of the more complex parts of our rendering pipeline, and you need to be able to render images and icons and that almost honestly gets you everything. Those are the primitives. And so what our Rust based UI framework does is essentially translate higher level things into those, what ends up being a sequence of graphics calls to draw Rex glyphs and images.
Zach Lloyd 00:53:07 And so you go from Rusts into those graphicsí calls, the graphics calls have to actually happen like objective C because, so we, there’s Rust allows you to sort of do something, I think it’s the foreign function interface. It lets you bind into native platform code selectively if you need to. And so 95% of our app is Rust, but the pieces that are–hey, we need to call Metal, we need to actually do that from objective C. And so you bind in the native code to do that.
Gregory Kapfhammer 00:53:38 So thanks a lot for giving us the overview of how Warp was implemented using Rust and using Metal. One of the things that I can tell clearly is that performance is central to your work. So my question is how do you make sure that when you ship a new version of Warp that you’ve maintained your performance goals?
Zach Lloyd 00:53:59 It’s a great question and it’s really hard. I’d say there’s a few general approaches. So one approach is benchmarking. And so, we have a sort of suite of benchmarks that we can run against Warp and make sure that we are performing well, at least against other terminals. We are in terms of where we’re at with that, we’re not necessarily the fastest terminal in the market for every single operation, but we are a very fast terminal, for most operations. And so that’s the way we compare to the competition. The other thing is, harder is like regression, preventing regressions in performance. And so there’s different approaches I’ve used in my career for this, what we use at Warp right now. We don’t have an environment in which we can consistently get sandbox performance data.
Zach Lloyd 00:54:44 So we tend to use more, looking at actual stats of work performance in the wild, which is gives you okay view in the aggregate, but it has several problems. It doesn’t give you a way of preventing regression until it’s live. The ideal thing that I’ve actually did at Google, which is not a thing that’s easy to do, or at least I haven’t found a great solution, was having a suite of performance tests that could run and you could run them as part of your continuous integration to make sure that you’re not resting things very, very hard because there’s a lot of noise and you need a very controlled environment to get sort of accurate regression tests. But yeah, those are the different ways that I can think of.
Gregory Kapfhammer 00:55:30 Thanks for sharing all of the ways in which you’re using Rust to implement the Warp terminal window. I know that ultimately your goal is to ensure that Warp can run on macOS, Windows, Linux, and ultimately the web. Can you tell us a little bit about the steps you’re going to take to make Warp work in all of those locations?
Zach Lloyd 00:55:50 Yeah, we’re working on this right now actually. So Linux is our number one requested user feature. We’re fortunate in using Rust that all of the non-platform specific parts of our code will basically just compile for Linux and Windows. For the web, it’s trickier because there’s some sort of threading and async perimeters that we use in Rust that don’t translate that cleanly to the web execution model. So it’s taking a little bit more work to sort of do a platform compilation for web, but for the most part the Rust code will compile the pieces that are platform specific tend to be harder. And so that is text rendering, getting the actual glyphs is a platform specific thing and getting the text shaping and all the text metrics that you need to actually lay out text.
Gregory Kapfhammer 00:57:48 So it sounds you have a lot of work ahead of you. Can you give us one concrete example of how someone would use Warp on the web and how would it be similar to or different from the way they use Warp on their macOS desktop?
Zach Lloyd 00:58:03 The big sort of product innovations that we’re trying to bring to the terminal is to make it work not just for individuals, but for teams. And so one of the features I talked about earlier, the ability to take a block, get a, get a link to it and share it to people on your team is the start of that. But there’s more to it. We would to be able to actually not just share a block but share an entire session. And you can do that we could build that within the native app so that if both people have the Mac app installed, they can share the session in the Mac app. But I think it’s actually a more powerful primitive to be able to let people to open up a terminal session in their browser, especially if that terminal session is fast and has all of the work features, which is what we’re building towards.
Zach Lloyd 00:58:46 And so terminal session sharing is one example. There’s other examples around knowledge sharing where itís you want to be able to link to terminal documentation and have other people on your team use it. We are, I think by the time this podcast comes out, we’ll have launched an actual feature in Warp that is a sort of multiplayer feature where people can store and share terminal commands and soon notebooks. And so all that stuff, all these sharing features I think are much more powerful in the web. The second thing is, I do feel the world of development will eventually move from being just development on your local machine to primarily development on remote machines. I think it’s going to take a long time to get there because people’s setups are so complicated and there’s actual advantages to being local. But in a world where you’re not developing on your local machine, it’s actually, there’s a lot of advantages to having a web-based rendering. It’s you can look at the thing from your phone or from your laptop, so the web does give you a lot of benefits and we’re trying to position ourselves to take advantage of those.
Gregory Kapfhammer 00:59:51 Yeah, I could imagine it being really compelling to be able to use Warp locally. And then when I want to loop in some of my colleagues on my team, I can have them work with me on Warp running in my web browser.
Zach Lloyd 01:00:04 Exactly. we feel that’s a potentially such a cool killer use case, whether it’s for teams or for classes or for people doing firefighting to just have a one common view like you have in something like Google Docs. I think it could be transformative. And so the web, it’s not strictly necessary for that, but it makes it so much easier and better.
Gregory Kapfhammer 01:00:25 So at this point, we’ve covered many of the basic and advanced features of Warp and I think you’ve done a really great job of sharing those exciting ways in which we, as listeners of Software Engineering Radio can dive in and use Warp. Are there any features of Warp that we didn’t talk about that you would like to highlight?
Zach Lloyd 01:00:45 I mean, the one that I was going to call out was this thing called Warp Drive, which will be launched by the time this is out. I would love for anyone who is using Warp in a team setting to try it out and give us feedback. It’s a totally novel thing where it’s–hey, I have these five commands that I use to talk to GCloud, which could then be shared with your team, and you could all use them. So I’m super interested in any feedback on that. Other stuff, I don’t know. We have, we have some really cool stuff around theming that’s always a lot of fun. People can go in and play with our themes and, we have a community oriented way of people submitting themes. So if you want to make a theme for us and share it, that’s always cool. But no, I think we covered most of the cool stuff.
Gregory Kapfhammer 01:01:28 So that’s awesome. As we’re drawing our episode of Software Engineering Radio to a conclusion, I want to turn to you as an expert, both in using a terminal and also in developing and shipping a terminal. Can you give a call to action for our listeners when it comes to terminals or developer tools or the Warp command line terminal in specific?
Zach Lloyd 01:01:52 So I’d love for folks to try Warp and give us feedback. Obviously, I would say if you’re a developer who’s not much of a terminal user or who has been intimidated or turned off or just not wanted to bother with the setup and headache of getting a good terminal environment, I would give terminals a try. Because I do think there’s huge productivity gains for people who really know how to use them well. I would say Warp is probably the easiest way to get in, or at least that’s our goal, is to make it the easiest and most powerful experience. But I don’t know, I’m a really big fan of a command line as a way of increasing your productivity and increasing your skill as an engineer. Most of the really best engineers I know, they do stuff in the command line that feels magic to me and it saves them a bunch of time. So my pitch would be, give it a shot and spend some time learning the command lines I think could really up your game as a developer in general.
Gregory Kapfhammer 01:02:53 All right. Thanks for that call to action. Is there anything that we didn’t talk about in this episode that you’d like to share before we draw things to a conclusion?
Zach Lloyd 01:03:02 I don’t think so. I would just say there’s, I know there’s probably a lot of great engineers listening to this show. We work in Rust. If you’re interested in how do you help developers ship better software more quickly, if you’re interested in terminals, we’d love to hear from you. And all of that information is on our website, which is warp.dev, which is also where you can download and try out Warp.
Gregory Kapfhammer 01:03:22 All right, so this has been an awesome conversation. I’ve really been keen to learn more about Warp. I hope that people will try Warp out and get in contact with you. If anyone wants to learn more about Warp or the way in which Warp was implemented, they’re welcome to check the show notes for this episode to catch many of the details that Zach and I will pass along to you. Zach, thanks for being here on Software Engineering Radio.
Zach Lloyd 01:03:48 Gregory, thank you so much for having me. This was fantastic.
Gregory Kapfhammer 01:03:52 Alright, thanks for being here, Zach. This is Gregory Kapfhammer, signing off for Software Engineering Radio. [End of Audio]