Elizabeth Figura, a Wine Developer at CodeWeavers, speaks with host Jeremy Jung about the Wine compatibility layer and the Proton distribution. They discuss a wide range of details including system calls, what people run with Wine, how games are built differently, conformance and regression testing, native performance, emulating a CPU vs emulating system calls, the role of the Proton downstream distribution, improving Wine compatibility by patching the Linux kernel and other related projects, Wine’s history and sustainment, the Crossover commercial distribution, porting games without source code, loading executables and linked libraries, the difference between user space and kernel space, poor Windows API documentation and use of private APIs, debugging compatibility issues, and contributing to the project.
This episode is sponsored by Monday Dev
Show Notes
Related links
Transcript
Transcript brought to you by IEEE Software magazine.
This transcript was automatically generated. To suggest improvements in the text, please contact [email protected] and include the episode number and URL.
Jeremy Jung 00:00:19 Hey, this is Jeremy Jung for Software Engineering Radio, and today I am talking to Elizabeth Figura. She’s a Wine developer at Code Weavers, and today we’re going to talk about what that is and all the work that goes into it. Elizabeth, welcome to Software Engineering Radio.
Elizabeth Figura 00:00:34 Thank you, Jeremy. I’m glad to be here.
Jeremy Jung 00:00:36 I think the first thing we should talk about is maybe saying what Wine is, because I think a lot of people aren’t familiar with the project.
Elizabeth Figura 00:00:44 So Wine is a translation layer. In fact, I would say Wine is a Windows emulator that is what the name originally stood for. It re-implement the entire Windows, or you say Win32 API. So that programs that make calls into the API will then transfer that code to Wine and we allow that Windows programs to run on things that are not Windows. So Linux, Mac OS, other operating systems such as Soliris and BSD. It works not by emulating the CPU, but by re-implementing every API basically from scratch and translating them to their equivalent or writing new code in case there is no equivalent.
Jeremy Jung 00:01:24 I believe what you’re doing is you’re emulating system calls. Could you explain what those are and how that relates to the project?
Elizabeth Figura 00:01:32 So system call in general can be used to refer to a call into the operating system to execute some functionality that’s built into the operating system. Often, it’s used in the context of talking to the kernel. Windows applications actually tend to talk at a much higher level because there’s so much high-level functionality built into Windows as opposed to other operating systems that basically we end up implementing much higher-level behavior than you would on Linux.
Jeremy Jung 00:02:00 And can you give some examples of what some of those system calls would be, and I suppose how they may be higher level than some of the Linux ones?
Elizabeth Figura 00:02:08 Sure. So of course you have low level calls like interacting with a file system, create a file and read and write and such. You also have high level APIs who interact with a sound driver. There’s one I was working on earlier today called XAudio where you actually build this bank of sounds. It’s meant to be played in the game and then you can position them in various 3D space and the operating system in a sense will take care of all of the math that goes into making that work. That’s all running on your computer and then it’ll send that audio data to the sound card once it’s transformed it. So it sounds like it’s coming from a certain space. A lot of other things like parsing XML is another big one that there’s a lot of things. This space is honestly huge.
Jeremy Jung 00:02:55 Yeah, I can sort of see how those might be things you might not expect to be done by the operating system. Like you gave the example of 3D audio and XML parsing and I think XML parsing in particular, you would’ve thought that that would be something that would be handled by the standard library of whatever language the person was writing their application as. So that’s interesting that it’s built into the OS.
Elizabeth Figura 00:03:17 Yeah, well, and language is like C, it isn’t even part of the standard library. It’s higher level than that. It’s you have specific libraries that are widespread but not codified in a standard, but in Windows they are part of the operating system. And in fact, there’s several different XML parsers in the operating system. Microsoft likes to deprecate old APIs and make new ones that do the same thing very often.
Jeremy Jung 00:03:40 And something I’ve heard about Windows is that they’re typically very reluctant to break backwards compatibility. So you say they’re deprecated, but do they typically keep all of them still in there?
Elizabeth Figura 00:03:51 Oh yeah. It all still works.
Jeremy Jung 00:03:51 And that’s all things that Wine has to implement as well to make sure that yeah, the software works as well.
Elizabeth Figura 00:03:59 Yeah. And we also need to implement those things to make old programs work because there is a lot of demand, at least from people using Wine for getting some really old programs working from the early nineties even.
Jeremy Jung 00:04:12 And that’s probably a good thing to talk about in terms of what are the types of software that people are trying to run with Wine and what operating system are they typically using?
Elizabeth Figura 00:04:22 In terms of software? Literally all kinds, any software you can imagine that runs on Windows, people will try to run it on Wine. So we’re talking games office software, productivity software, accounting. People will run build systems online, just run, build their programs using on Visual Studio, running on Wine. People will run Wine on servers for example, like software as a service kind of things where you don’t even know this is running on Wine. Really super domain specific stuff. Like I’ve run astronomy software in Wine design, computer assisted design, even hardware drivers can sometimes work on Wine. There’s a bit of a gray area.
Jeremy Jung 00:05:00 I think from maybe the general public, or at least from what I’ve seen. I think a lot of people’s exposure to it is for playing games. Is there something different about games versus all those other types of productivity software and office software that makes supporting those different?
Elizabeth Figura 00:05:20 There’s some things about it that are different. Games of course have gotten a lot of publicity lately because there’s been a huge push largely from Valve, but also some other companies to get a lot of huge wide range of games working well under Wine. And that’s really panned out in a way. I think we’ve largely succeeded. We’ve made huge strides in the past several years, five, 10 years, I think. So when you talk about what makes games different, I think one thing games tend to do is they have a very limited set of things they’re working with and they often want to make things run fast. So they’re working very close to the metal. They’re not going to use an XML parser for example. They’re just going to talk as directly to the graphics driver as they can, right? And probably going to do all their own sound design.
Elizabeth Figura 00:06:08 I did talk about that XAudio library, but a lot of games will just talk as directly to the sound driver is Windows Let some, so this is often a blessing honestly, because it means there’s less we have to implement to make them work. When you look at a lot of productivity applications, and especially the other thing that makes some productivity applications harder is Microsoft makes them and they like to make a library for use in this one program like Microsoft Office and then say, well other programs might use this as well, let’s put it in the operating system and expose it and write an API for it and everything. And maybe some other programs use it. Mostly it’s just office, but it means that office relies on a lot of things from the operating system that we all have to reimplement.
Jeremy Jung 00:06:52 Yeah, that’s somewhat counterintuitive because when you think of games, you think of these really high-performance things that seem really complicated. But it sounds like from what you’re saying, because they use the lower-level primitives, they’re actually easier in some ways to support.
Elizabeth Figura 00:07:08 Yeah, certainly in some ways. They’ll do things like re-implement the heap allocator because the built-in heap allocator isn’t fast enough for them. That’s another good example.
Jeremy Jung 00:07:17 You mentioned Microsoft’s more modern office suites. I’ve noticed there are certain applications that aren’t supported like for example, I think the modern Adobe Creative Suite. What’s the difference with software like that and does that also apply to the modern office suite or is that actually supported?
Elizabeth Figura 00:07:36 Well, in one case you have things like Microsoft using their own APIs that I mentioned. With Adobe, that applies less, I suppose. But I think to some degree the answer is that some applications are just hard and there’s no way around it. And we can only spend so much time on a hard application. Debugging of things can get very hard with Wine. Let me explain that for a minute. Because normally when you think about debugging an application, you say, oh, I’m going to open up my debugger. Pop it then break at this point, see what like all the variables are, or they’re not what I expect, or maybe wait for it to crash and then get a back trace and see where it crashed and why. You can’t do that with Wine because you don’t have the application, you don’t have the symbols, you don’t have your debugging symbols, you don’t know anything about the code you’re running unless you take the time to disassemble and decompile and read through it.
Elizabeth Figura 00:08:26 And that’s difficult. It’s only difficult every time I’ve looked at a program and been like, I’m going to just try and figure out what the program is doing. It takes so much time and it is never worth it. And sometimes you have to, sometimes you have no other choice, but usually you actually rely on seeing what calls it makes into the operating system and trying to guess which one of those is going wrong. Now, sometimes you’ll get lucky and it’ll crash in Wine code or sometimes it’ll make a call into a function that we don’t implement yet and we know, oh, we need to implement that function. But sometimes it does something more obscure and we have to figure out, well, like all of these millions of calls it made, which one of them are we implementing incorrectly? So it’s returning the wrong result or not doing something that it should and then you add onto that all this sort of harder to debug things like memory errors that we could make. And it can be very difficult. And so sometimes some applications just suffer from those hard bugs and sometimes it’s also just a matter of not enough demand for something for us to spend a lot of time on it, right?
Jeremy Jung 00:09:32 Yeah, I can see how that would be really challenging because like you were saying, you don’t have the symbols, so you don’t have the source code, so you don’t know what any of this software you’re supporting, how it was actually written. And you were saying that a lot of times there may be some behavior that’s wrong or a crash, but it’s not because Wine crashed or there was an error in Wine. So you just know the system calls it made, but you don’t know which of the system calls didn’t behave the way that the application expected.
Elizabeth Figura 00:10:04 Exactly.
Jeremy Jung 00:10:05 I can see how that would be really challenging. And Wine runs so many different applications. I’m kind of curious how do you even track what’s working and what’s not as you change Wine? Because if you support thousands or tens of thousands of applications, how do when you’ve got a regression or not?
Elizabeth Figura 00:10:26 Great question. Probably over half of Wine by like source code volume. I actually check what it is, but I think it’s probably over half is what we call is tests. And these tests serve two purposes. The one purpose is a regression tests, and the other purpose is a conformance tests. That tests how an API behaves on Windows and validates that we are behaving the same way. So we write all these tests, we run them on Windows and write the tests to check what the Windows returns, and then we run them on Wine and make sure that that matches. And we have just such a huge body of tests to make sure that we’re not breaking anything and that all the code that we get into Wine that looks like, wow, it’s doing that really? Well, nope, that’s what Windows does. The test says so. So pretty much any new code that we get, it has to have tests to demonstrate that it’s doing the right thing.
Jeremy Jung 00:11:23 And so rather than testing against a specific application, seeing if it works, you’re making a call to a Windows system call, seeing how it responds, and then making the same call within Wine and just making sure they match.
Elizabeth Figura 00:11:40 Yes, exactly. And that is a lot more automatable, right? Because otherwise you have to manually these are all graphical applications. You’d have to manually do the things and make sure they work, but if you write automatable tests, you can just run them all and the machine will complain at you if it fails it continuous integration.
Jeremy Jung 00:12:00 And because there’s all these potential compatibility issues where maybe a certain call doesn’t behave the way an application expects. What are the types of ways that shows when someone’s using software? I mean, I think you mentioned crashes, but I imagine there could be all sorts of other types of behavior.
Elizabeth Figura 00:12:21 Yes, very much so. Basically anything you can imagine again is what will happen. You can have crashes of the easy ones because when and where it crashed and you can work backwards from there. But you can also, it could hang, it could not render, right? Like maybe render a black screen for games. You could very frequently have graphical glitches where maybe some objects won’t render right. Or the entire screen will be red, who knows? In a very bad case, you could even bring down your system and we usually say that’s not Wine’s fault, that’s the graphics library’s fault. Because they’re not supposed to do that no matter what we do. But sometimes we have to work around that anyway. But yeah, there’s been some very strange and idiosyncratic bugs out there too.
Jeremy Jung 00:13:05 Yeah, and like you mentioned that there’s so many different things that could have gone wrong. I imagine very difficult to find, yeah. And when software runs through Wine, I think a lot of our listeners will probably be familiar with running things in a virtual machine and they know that there’s a big performance impact from doing that. How does the performance of applications compare to running natively on the original Windows OS versus virtual machines?
Elizabeth Figura 00:13:36 So in theory, and I haven’t actually done this recently, so I can’t speak too much to that, but in theory the idea is it’s a lot faster. So there is a bit of a joke acronym to Wine. Wine is not an emulator, even though I started out by saying Wine is an emulator and it was originally called a Windows emulator. But what this basically means is Wine is not a CPU emulator. When you think about emulators in a general sense are often emulators for specific CPUs, often older ones like Commodore emulator or an Omega emulator. But in this case you have software that’s written for an X86 CPU and it’s running on an X86 CPU by giving it the same instructions that it’s giving on Windows. It’s just that when it says now call this Windows function, it calls us instead. So that all should perform exactly the same.
Elizabeth Figura 00:14:28 The only performance difference at that point is that all should perform exactly the same as opposed to a virtual machine where you have to interpret the instructions and maybe translate them to a different instruction set. The only performance difference is going to be in the functions that we are implementing themselves and we try to implement them to perform as well or almost as well as Windows. There’s always going to be a bit of a theoretical gap because we have to translate from say, one API to another, but we try to make that as little as possible. And in some cases the operating system we’re running on is just better than Windows and the libraries we’re using are better than Windows. So our games will run faster for example. Sometimes we can do a better job than Windows at implementing something that’s under our purview. There are some games that do actually run a little bit faster in Wine than they do on Windows.
Jeremy Jung 00:15:24 That reminds me of how there’s these gaming handhelds out now and some of the same ones, they either let you install Linux or install Windows, or they just come with a pre-installed. And I believe what I’ve read is that oftentimes running the same game on both operating systems, running the same game on Linux, the battery life is better and sometimes even the performance is better with these handhelds. So it’s really interesting that that can even be the case.
Elizabeth Figura 00:15:53 Yeah, it’s really a testament to the huge amount of work that’s gone into that, both on the Wine side and on the side of the graphics team and the kernel team, and of course the years of work that’s gone into Linux even before these gaming handhelds were even under consideration.
Jeremy Jung 00:16:10 So for people who are familiar with handhelds like the Steam deck, they may have heard of Proton. I wonder if you can explain what Proton is and how it relates to Wine?
Elizabeth Figura 00:16:22 Yeah, so Proton is sort of a fork, although we try to avoid the term fork. We say it’s a downstream distribution because we contribute back up to Wine. So it is an alternate distribution fork of Wine, and it’s also some code that basically glues Wine into an embedding application. Originally intended for Steam and developed four Valve, but it has also been used in other software. So where Proton differs from Wine besides the glue part is it has some extra hacks in it for bugs that are hard to fix and easy to hack around as some quick hacks for making games work now that are like in the process of going upstream to Wine and getting their code quality improved and going through review. But we want the game to work now when we distribute it. So that’ll go into Proton immediately. And then once a patch makes it upstream, we replace it with the version of the patch from upstream. There’s other things to make it interact nicely with Steam and so on.
Jeremy Jung 00:17:22 Yeah, and I think for people who aren’t familiar, Steam is like this, I don’t even know what you call it, like a gaming store.
Elizabeth Figura 00:17:30 Call it a game store game distribution service. It’s got a huge variety of games on it and it’s a great way for publishers to interact with a wider gaming community just after paying a cut to Valve of their profits. They can reach a lot of people that way. And because all these games are on Steam and Valve wants them to work well on their handheld, they contracted us to basically take their entire catalog, which is huge, enormous, and try and just step by step, fix every game and make them all work.
Jeremy Jung 00:18:01 And I guess for people who aren’t familiar, Valve software is the company that runs Steam. And so it sounds like they’ve asked your company to help improve the compatibility of their catalog.
Elizabeth Figura 00:18:12 Yes, Valve contracted us, and again when you’re talking about Wine using lower-level libraries, they’ve also contracted a lot of other people outside of Wine. Basically the entire stack has had a tremendous, tremendous investment by Valve software to make gaming on Linux work well.
Jeremy Jung 00:18:30 And when you refer to the entire stack, like what are some of those pieces, at least at a high level?
Elizabeth Figura 00:18:36 Let me think. There is the Wine project, the Mesa Graphics Libraries, that’s another open-source software project that has existed for a long time. But Valve has put a lot of funding and effort into it. The Linux kernel in various different ways. The desktop environment and Window Manager are also things they’ve invested in. Everything that the game needs on any level and that the operating system of the handheld device needs.
Jeremy Jung 00:19:04 And Wine’s been going on for quite a while. I think it’s over a decade, right?
Elizabeth Figura 00:19:10 Oh, far more than a decade. I believe it started in, I want to say about 1995, mid-nineties. I probably have that date wrong. I believe Wine started about the mid-nineties. So it’s been going on for about three decades at this rate.
Jeremy Jung 00:19:23 Wow. And so all this time, how has the project sort of sustained itself? Like who’s been involved and how has it been able to keep going this long?
Elizabeth Figura 00:19:34 I think as is the case with a lot of free software, it just keeps trudging along. There’s been times where there’s a lot of interest in Wine, there’s been times where there’s less and we are fortunate to be in a time where there’s a lot of interest in it. We’ve had the same maintainer for almost his entire existence, Alexander Julliard. There was one person who started maintained it before him and left it maintainer ship to him after a year or two, Bob Amstat. And there’s been a few developers who have been around for a very long time, a lot of developers who have been around for a decent amount of time, but not for the entire duration. And then a very, very large number of people who come and submit a one-off fix for their individual application that they want to make work.
Jeremy Jung 00:20:20 How does Crossover relate to the Wine project? It sounds like you had mentioned Valve software hired you for subcontract work, but Crossover itself has been around for quite a while. So how has that been connected to the Wine project?
Elizabeth Figura 00:20:35 So the company I work for is Code Weavers and Crossover is our flagship software. So Code Weavers does a couple different things. We have a sort of a porting service where companies will come to us and say, can we port my application usually to Mac? And then we also have a retail service where we basically have our own similar to Proton but older, but the same idea where we will add some hacks into it for very difficult to solve bugs and we have a nice graphical interface. And then the other thing that we’re selling with Crossover is support. So if you try to run a certain application and you buy Crossover, you can submit a ticket saying this doesn’t work and we now have a financial incentive to fix it. We’ll spend company resources to fix your bug. Right? So Code Reverse has been around since 1996 and Crossover has been around for probably about two decades if I’m not mistaken.
Jeremy Jung 00:21:32 And when you mention helping companies port their software to, for example Mac Os, is the approach that you would port it natively to Mac OSíS APIs or is it that you would help them get it running using Wine on Mac OS?
Elizabeth Figura 00:21:50 So that’s basically what makes us so unique among porting companies is that instead of rewriting their software, we just basically stick it inside of Crossover and make it run. And the idea has always been the more we implement, the more we get correct, the more applications will work. And sometimes it works out that way, sometimes not really so much and there’s always work we have to do to get any given application to work. But it’s very unusual because we don’t ask companies for any of their code, we don’t need it. We just fix the Windows API.
Jeremy Jung 00:22:23 And so in that case, the ports would be where let’s say someone sells a Mac OS version of their software, they would bundle Crossover with their software?
Elizabeth Figura 00:22:34 Right? And usually when you do this, it doesn’t look like there’s Crossover there. Like it just looks like this software is native, but there is Crossover under the hood.
Jeremy Jung 00:22:43 And so earlier we were talking about how you’re basically intercepting the system calls that these binaries are making, whether that’s the executable or the DLLs from Windows, but I think probably a lot of our listeners are not really sure how that’s done. Like they may have built software but they don’t know how do I basically hijack the system calls that this application is making. So maybe you could talk a little bit about how that works.
Elizabeth Figura 00:23:12 So there’s a couple steps to go into it. You think about a program that’s say that’s a big file that’s got all the machine code in it, and then it’s got stuff at the beginning saying, here’s how the program works and here’s where in the file the processor should start running. Yes, your EXE file and then in your DLL files are libraries that contain shared code. And you have I guess similar sort of file that says here’s the entry point that runs this function, this parse XML function or whatever have you. And here’s this entry point that has the generate XML function and so on and so forth. And then the operating system will basically take the EXE file and see all the bits in it. Say I want to call the parse XML function, it’ll load that DLL and hook it up. So the processor ends up just seeing jump directly to this parse XML function and then run that and then return and so on. And so what Wine does is part of Wine is a library, is that the implementing that parse XML and read XML function. But part of it is the loader, which is the part of the operating system that hooks everything together. And when we load, we redirect to our libraries. We don’t have Windows libraries, we redirect to ours and we run our code. And then when you jump back to the program.
Jeremy Jung 00:24:30 So it’s the loader that’s a part of Wine that’s actually not sure if running the executable is the right term?
Elizabeth Figura 00:24:39 OSOS think that’s a good term. It starts in a loader and then we say, okay, now run the machine code and it’s executable and then it runs and it jumps between our libraries and back and so on.
Jeremy Jung 00:24:49 And like you were saying before, often times when it’s trying to make a system call, it ends up being handled by a function that you’ve written in Wine and then that in turn will call the Linux system calls or the Mac OS system calls to try and accomplish the same result.
Elizabeth Figura 00:25:08 Right, exactly.
Jeremy Jung 00:25:09 And something that I think maybe not everyone is familiar with is there’s this concept of user space versus kernel space. Could you explain what the difference is?
Elizabeth Figura 00:25:19 So the way I would describe a kernel is it’s the part of the operating system that can do anything, right? So any code that runs on your computer is talking to the processor and the processor has to be able to do anything the computer can do. It has to be able to talk to the hardware, it has to set up the memory space. So actually a very complicated task has to be able to switch to another task and basically talk to another program. And you have to have something there that can do everything, but you don’t want any program to be able to do everything. Not since the nineties. It’s about when we realized that we can’t do that. So the kernel is a part that can do everything. And when you need to do something that requires those permissions that you can’t give everyone, you have to talk to the kernel and ask it, hey, can you do this for me please?
Elizabeth Figura 00:26:08 In a very restricted way where it’s only the safe things you can do. And a degree it’s also like a library, right? It’s kernels have always existed. And since they’ve always just been the core standard library of the computer that does the things like read and write files, which are very, very complicated tasks under the hood, but look very simple because all you say is write this file and talk to the hardware and abstract away all the difference between different drivers. So the kernel is doing all of these things. So because the kernel is a part that can do everything, when you think about the kernel, it is basically one program that is always running on your computer, but it’s only one program. So when a user calls the kernel, you are switching from one program to another and you’re doing a lot of complicated things as part of this, you’re switching to the higher privilege level where you can do anything and you’re switching the state from one program to another. And so this is what we mean when we talk about user space where you’re running like a normal program and kernel space where you’ve suddenly switched into the kernel and now you’re executing with increased privileges and a different idea of the process space and increased responsibility and so on.
Jeremy Jung 00:27:55 And so do most applications, when you were talking about the system calls for handling 3D audio or parsing XML, are those system calls considered part of user space and then those things call the kernel space on your behalf or how would you describe that?
Elizabeth Figura 00:28:13 So when you look at Windows, most of the Windows library, the vast, vast majority of it is all user space. Most of these libraries that we implement never leave user space. They never need to call into the kernel, only the core low level stuff. Things like we need to read a file, that’s a kernel call when you need to sleep and wait for some seconds. That’s a kernel call. Need to talk to a different process. Things that interact with different processes in general, not just allocate memory, but allocate a page of memory like a from the memory manager and then that gets sub allocated by the heap allocator. So things like that.
Jeremy Jung 00:28:46 So if I was writing an application and I needed to open a file for example, does that mean that I would have to communicate with the kernel to read that file?
Elizabeth Figura 00:28:56 Right, exactly.
Jeremy Jung 00:28:58 And so most applications, it sounds like it’s going to be a mixture. You’re going to have a lot of things that call user space calls, and then a few, you mentioned more low-level ones that are going to require you to communicate with the kernel.
Elizabeth Figura 00:29:12 Yeah, basically. And it’s worth noting that in all operating systems you’re almost always going to be calling a user space library. That might just be a thin wrapper over the kernel call. It might, it’s going to do like just a little bit of work and then call the kernel. In fact, in Windows, that’s the only way to do it. In many other operating systems, you can actually say, you can actually tell the processor to make the kernel call there a special instruction that does this and it defined interface for this. But in Windows, that interface is not defined. It’s not stable or backwards compatible like the rest of Windows is. So even if you wanted to use it, you couldn’t. And you basically have to call into the high-level libraries or low-level libraries as it were, that create a file. And those don’t do a lot, they just kind of tweak their parameters a little and then pass them right down to the kernel.
Jeremy Jung 00:30:06 And so Wine, it sounds like it needs to implement both the user space calls of Windows, but then also the kernel calls as well. But Wine itself, is that only in Linux user space or Mac OS user space?
Elizabeth Figura 00:30:24 Yes. This is a very tricky thing, but basically all of what is Wine runs in user space and we use kernel calls that are already there to talk to the host kernel. You get to sort of second nature of thinking about the Windows user space and kernel and then there’s a host user space and kernel. And Wine is running all in the host user space, but it’s emulating the Windows kernel. In fact, one of the weirdest, trickiest parts is I mentioned that you can run some drivers in Wine and those drivers, they actually are, they think they’re running in the Windows kernel, which in a sense works the same way. It has libraries that it can load, and those drivers are basically libraries and they’re making kernel calls and they’re making calls into the kernel library that does some very, very low-level tasks that you’re normally only supposed to be able to do in a kernel. And because the kernel requires some privileges, we kind of pretend we have them. And in many cases, even the drivers are using abstractions. We can just implement those abstractions kind of over the slightly higher-level abstractions that exist in user space.
Jeremy Jung 00:31:33 Yeah, I hadn’t even considered being able to use hardware devices, but I suppose if in the end, if you’re reproducing the kernel, then whether you’re running software or you’re talking to a hardware device, as long as you implement the calls correctly, then I suppose it works because you’re talking about device, like maybe it’s some kind of USB device, right? That has drivers for Windows, but it doesn’t for Linux.
Elizabeth Figura 00:31:59 Yeah, that’s kind of the example I’ve used. I think one of my best success stories was drivers for a graphing calculator.
Jeremy Jung 00:32:07 Oh wow.
Elizabeth Figura 00:32:07 That connected via USB and I basically just plugged the Windows drivers into Wine and ran it. And I had to implement a lot of things, but it worked. But for example, something like a graphics driver is not something you could implement in Wine because you need the graphics driver on the host. We can’t talk to the graphics driver while the host is already doing so.
Jeremy Jung 00:32:27 I see. Yeah. And in that case, it probably doesn’t make sense to do so anyways. Right?
Elizabeth Figura 00:32:34 Right. It doesn’t because the transition from user into kernel is complicated. You need the graphics driver to be in the kernel and the real kernel having it in Wine would be a bad idea.
Jeremy Jung 00:32:44 Yeah, I think there’s enough APIs you have to try and reproduce that I think doing something.
Elizabeth Figura 00:32:51 Yes, and it’d be very very difficulat.
Jeremy Jung 00:32:51 Right. There’s so many different calls both in user space and in kernel space. I imagine the, the user space ones Microsoft must document to some extent?
Elizabeth Figura 00:33:05 Well, sometimes.
Jeremy Jung 00:33:07 Sometimes. Okay.
Elizabeth Figura 00:33:08 I think it’s actually better now than it used to be, but here’s where things get fun because sometimes there will be regular documented calls. Sometimes those calls are documented, but the documentation isn’t very good. Sometimes programs will just sort of look inside Microsoft’s DLLs and use calls that they aren’t supposed to be using. Sometimes they use calls that they are supposed to be using, but the documentation has disappeared just because it’s that old of an API. And Microsoft hasn’t kept it around. Sometimes Microsoft on software uses APIs that were never documented because they never wanted anyone else using them, but they still ship them with the operating system. There was actually a kind of a lawsuit about this because it is an antitrust lawsuit because by shipping things that only they could use, they were kind of creating a trust and that got some things documented. At least in theory, they kind of haven’t stopped doing it though.
Jeremy Jung 00:34:07 Oh. So even today they’re, I guess they would call those private APIs, I suppose.
Elizabeth Figura 00:34:12 Yeah, you could say private APIs, but if we want to get newer versions of Microsoft Office running, we still have to figure out what they’re doing and implement them.
Jeremy Jung 00:34:21 And given that they’re either, like you were saying, the documentation is kind of all over the place. If you don’t know how it’s supposed to be behave, how do you even approach implementing them?
Elizabeth Figura 00:34:33 And that’s what the conformance tests are for. I mentioned earlier we have this huge body of conformance tests that double as regression tests. If we see an API, we don’t know what to do with or an API, we do know we, we think we know what to do with because the documentation can just be wrong and often has been. Then we write tests to figure out what it’s supposed to behave. We kind of guess until we, and we write tests and we pass some things in and see what comes out and see what the operating system does until we figure out, oh, so this is what it’s supposed to do and these are the exact parameters in and, and then we implement it according to those tests.
Jeremy Jung 00:35:10 Is there any distinction in approach for when you’re trying to implement something that’s at the user level versus the kernel level?
Elizabeth Figura 00:35:18 No, not really. And like I mentioned earlier, I mean a kernel call is just like a library call. It’s just done in a slightly different way, but it still got parameters in, it’s still got a set of parameters. They’re just encoded differently. And again, like the way ker calls are done, done is on a level just above the kernel where you have a library that just passes things through almost verbatim to the kernel and we implement that library instead.
Jeremy Jung 00:35:48 And you’ve been working on, I think, Wine for over six years now.
Elizabeth Figura 00:35:54 That sounds about right.
Jeremy Jung 00:35:56 What does your day to day look like? What parts of the project do you work on?
Elizabeth Figura 00:36:01 It really varies from day to day. And some people will work on the same parts of Wine for years. Some people will switch around and work on all sorts of different things. And I definitely belong to that second group. Like if you name an area of Wine, I have almost certainly contributed a patch or two to it. There’s some areas I work on more than others, like 3D graphics, multimedia, I worked on a compiler that exists, uh, sockets. So networking communication is another thing I work a lot on. I kind of just get a bug for some program or another and I take it and I debug it and figure out why the program’s broken and then I fix it. And there’s so much variety in that because the bug can take so many different forms like I described, and, and the fix can be simple or complicated or, and it can be in really anywhere to a degree. Being able to work on any part of Wine is sometimes almost a necessity because if a program is just broken, you don’t know why. It could be anything. It could be any sort of API and sometimes you can hand the API to somebody who’s got a lot of experience in that, but sometimes you just do whatever. You just fix whatever’s broken and you get an experience that way.
Jeremy Jung 00:37:16 Yeah, I mean, I was going to ask about the specialized skills to work on Wine, but it sounds like maybe in your case it’s all of them.
Elizabeth Figura 00:37:24 There’s a bit of that. The skills to work on Wine are, it’s a very unique set of skills because, and it largely comes down to debugging because you can’t use the tools you normally use to debug. You have to be creative and think about it different ways. Sometimes you have to be very creative and programs will try their hardest to avoid being debugged because they don’t want anyone breaking their copy protection for example. Or hacking in sheets. They don’t want anyone hacking them like that. And we have to do it anyway for legitimate purposes. We would argue to make them work better on more operating systems. And so we have to fight that every step of the way.
Jeremy Jung 00:38:01 Yeah, it seems like it’s a combination of, like you were saying, being able to debug and you’re debugging not necessarily your own code, but you’re debugging this behavior. And then based on that behavior you have to figure out, okay, where in all these different systems within Wine could this part be not working? Exactly. And I suppose you probably build up some kind of mental map in your head of when you get a type of bug or a type of crash, you think like, oh, maybe it’s this, maybe it’s here, or something like that.
Elizabeth Figura 00:38:36 Yeah, there is a lot of that. You notice some patterns experience helps, but because any bug could be new, sometimes experience doesn’t help and you just kind of have to start from scratch
Jeremy Jung 00:38:50 At sort of a high level. Can you give an example of where you got a specific bug report and then where you had to look to eventually find which parts of the system were the issue?
Elizabeth Figura 00:39:02 One, I think good example that I’ve done recently. So I mentioned this XAudio library that does 3D audio and say you come across a bug, I’m going to be a little bit generic here and say you come across a bug where some audio isn’t playing right, maybe there’s silence where there should be the audio. So you kind of, you look in and see, well where’s that getting lost? So you can basically look in the input calls and say, here’s the buffer it’s submitting that’s got all the audio data. You look at where you think the output should be. Like that library will internally call a different library, which programs can interact with directly. And this our high-level library interacts with that is the, give this sound to the audio driver, right? So you’ve got XAudio on top of MM Dev, API, which is the other library that gives audio to the driver and you see the buffers that XAudio is passing into MM Dave API, they’re empty, there’s nothing in them.
Elizabeth Figura 00:39:53 So you have to kind of work through the XAudio library to see where’s that sound getting lost. Or maybe that’s not getting lost, maybe it’s coming through all garbled. And I’ve had to look at the buffer and see why is it garbled. I’ll open up it up in Audacity and look at the weight shape of the wave and say, the shape of the wave looks like we’re putting silence every 10 nanoseconds or something or reversing something or interpreting it wrong. Things like that. You’ll do a lot of putting in print Fs basically all throughout Wine to see where does the state change, where does it right, and then where do things start going wrong?
Jeremy Jung 00:40:30 Yeah. And in the audio example, because they’re making a call to your XAudio implementation, you can see that okay, the buffer or the audio that’s coming in that part is good, right? It’s just that later on when it sends it to what’s going to actually have it be played by the, the hardware, that’s when it’s missing. So
Elizabeth Figura 00:40:51 We did something wrong in the library that destroyed the buffer. And I think on a very, very high level, a lot of debugging Wine is about finding where things are good and finding where things are bad. And in narrowing that down until we find the one spot where things go wrong, there’s a lot of processes that go like that.
Jeremy Jung 00:41:09 Like you were saying, the more you see these problems, hopefully the, the easier it gets to narrow down where
Elizabeth Figura 00:41:15 Often, yeah, especially if you keep debugging things in the same area
Jeremy Jung 00:41:20 And Wine supports more than one operating system. I, I saw there was Linux, Mac, OS, I think free BSD. How much of the code is operating system specific versus how much can just be shared across all of them?
Elizabeth Figura 00:41:36 Not that much is operating system specific actually. So when you think about the volume of Wine, the vast majority of it is high level code that doesn’t need to interact with the operating system on a low level, right? Because Microsoft keeps putting lots and lots of different libraries in their operating system. And a lot of these are high-level libraries. And even when we do interact with the operating system, we’re using cross-platform libraries or we’re using POSIX. So all these operating systems that we are implementing are basically conformed to the POSIX standard, which is basically like Unix, they’re all Unix-based. POSIX is a Unix-based standard. Microsoft is the big exception that never did implement that. And so we have to translate its APIs to Unix APIs. Now that said, there is a lot of very operating system-specific code. Apple makes things difficult by diverging almost wherever they can. And so we have a lot of Apple-specific code in there.
Jeremy Jung 00:42:36 Another example I can think of is, I believe MAC OS doesn’t support VULKAN.
Elizabeth Figura 00:42:42 Yes. That’s a great example of Mac not wanting to use, uh, generic libraries that work on every other operating system. And in some cases we look at it and are like, alright, we’ll implement a wrapper for that too, on top of your operating system. We’ve done it for Windows, we can do it for VULKAN. And then you get to Molten BK project. And to be clear, we didn’t invent molten bk. It was around before us, but we have contributed a lot to it.
Jeremy Jung 00:43:06 Yeah, I think maybe just at a high level, it might be good to explain the relationship between Direct3D or Direct X and VULKAN and yeah, maybe if you could go into that.
Elizabeth Figura 00:43:19 So Direct3D is Microsoft’s 3D API. The 3D APIs are basically a way to firstly abstract out the differences between different graphics cards, which look very different on a hardware level, especially they, they used to look very different. They still do look very different. And it’s secondly a way to deal with them at a high level because actually talking to the graphics card on a low level is very, very complicated. Even talking them to it on a high level is complicated, but it gets, it can get a lot worse if you’ve ever done any graphics driver development. So you have a number of different APIs that achieve these two goals of building a common abstraction and of building a high-level abstraction. So OpenGL is the broadly the free operating system world, the non-Microsoftís world’s choice back in the day.
Elizabeth Figura 00:44:14 And then Direct3D was Microsoft’s API. And both of these have evolved over time and come up with new versions and such. And when any API exists for too long, it gains a lot of cruft and needs to be replaced. And eventually the people who developed Open GL decided we need to start over, get rid of the cruft to make it cleaner and make it lower level. Because to get in a maximum performance games really want low level access. And so they made VULKAN, Microsoft kind of did the same thing, but they still call it Direct3D, it’s the newest version of Direct3D is lower level, it’s called Direct3D 12. And Mac looked at this and they decided we’re going to do the same thing too, but we’re not going to use VULKAN. We’re going to define our own. And they call it Metal. And so when we want to translate D3D 12 into something that another operating system understands that’s probably VULKAN and on Mac, we need to translate it to Metal somehow. And we decided instead of having a separate layer from D3 12 to metal, we’re just going to translate it to VULKAN and then translate the VULKAN to Metal. And it also lets things written for VULKAN on Windows, which is also a thing that exists that lets some work on Metal
Jeremy Jung 00:45:31 And having to do that translation. Does that have a performance impact or is that not really felt?
Elizabeth Figura 00:45:39 Yes. It’s kind of like anything, when you talk about performance, like I mentioned this earlier, there’s always going to be overhead from translating from one API to another, but we put in heroic efforts to try to make sure that doesn’t matter, to make sure that stuff that needs to be fast is really as fast as it can possibly be. And some very clever things have been done along those lines. And sometimes the graphics drivers underneath are so good that it actually does run better, even despite the translation overhead. And then sometimes to make it run fast, we need to say, well, we’re going to implement a new API that behaves more like Windows, so we can do less work translating it. And sometimes that goes into the graphics library and sometimes that goes into other places.
Jeremy Jung 00:46:26 Yeah. Something I’ve found a little bit interesting about the last few years is developers in the past, they would generally target Windows and you might be lucky to get a Mac port or a Linux port. And I wonder like in your opinion now that a lot of developers are just targeting Windows and relying on Wine or Proton to run their software. Is there any, I suppose, downside to doing that? Or is it all just upside, like everyone should target Windows as this common platform?
Elizabeth Figura 00:47:02 Yeah, it’s an interesting question. There’s some people who seem to think it’s a bad thing that we’re not getting native ports in the same sense. And then there’s some people who see, no, that’s a perfectly valid way to do ports just right for this de facto common API. It was never intended as a cross platform common API, but we’ve made it one, right? And so why is that any worse than if it runs on a different API on Linux or Mac? And that argument tends to make sense to me. I don’t personally see a lot of reason to say that one library is more pure than another. Right now I do think Windows APIs are generally pretty bad. This might be just some sort of, this might just be an effect of having to work with them for a very long time and see all their flaws and have to deal with the nonsense that they do. But I think that a lot of the Native Linux APIs are better. But if you like your Windows API better, and if you want to target Windows and that’s the only way to do it, then sure why not, what’s wrong with that?
Jeremy Jung 00:48:05 Yeah. And I think doing it this way, targeting Windows, I mean, if you look in the past, even though you had some software that would be ported to other operating systems without this compatibility layer, without people just targeting Windows, all this software that people can now run on these portable gaming handhelds or on Linux, most of that software was never going to be ported.
Elizabeth Figura 00:48:29 Oh yeah, absolutely. That’s kind of why Wine existed, because people wanted to run their software that was never going to be ported. They just wanted and community, just spent a lot of effort making all these individual programs run. Yeah.
Jeremy Jung 00:48:42 And I think it’s pretty amazing too that now that’s become this official way, I suppose, of distributing your software where, yeah, you say like, hey, I made a Windows version, but you’re on your Linux machine. It’s officially supported because we have this much belief in this compatibility layer.
Elizabeth Figura 00:49:03 It’s kind of incredible to see Wine having got this far. I mean, I started working on it six, seven years ago, and even then I could never have imagined it would be like this.
Jeremy Jung 00:49:14 So as we wrap up, for the developers that are listening or, or people who are just users of Wine, is there anything you think they should know about the project that we haven’t talked about?
Elizabeth Figura 00:49:27 I don’t think there’s anything I can think of.
Jeremy Jung 00:49:29 And if people want to learn more about the Wine project or see what you’re up to or should they head.
Elizabeth Figura 00:49:36 We don’t really have any things like news, unfortunately. Read the release notes. There’s some people from Code Weavers who do blogs. So if you go to code weavers.com/blog, there’s some Code Weavers stuff, some marketing stuff. But there’s also some developers who will talk about bugs that they are solving and how it’s easy and the experience of working on Wine.
Jeremy Jung 00:49:59 And I suppose if someone’s interested in like, like let’s say they have a piece of software, it’s not working through Wine, what’s the best place for them to either get help or maybe even get involved with trying to fix it?
Elizabeth Figura 00:50:14 So you can file a bug on WineHQ.org or find, there’s a lot of developer resources there and you can get involved with contributing to the software. And there’s links to our mailing list and IRC channels and the GitLab, where all places you can find developers. We love to help you debug things. We love to help you fix things. We try our very best to be a welcoming community, and we have a lot of experience working with people who want to get their application working. So we’d love to have another.
Jeremy Jung 00:50:47 Very cool. Yeah, I think Wine is a really interesting project because I think for, I guess it would’ve been for decades, it seemed like very niche, like not many people. Oh yeah, were aware of it. And now I think maybe in particular because of the Linux gaming handhelds, like the Steam deck Wine is now something that a bunch of people who would’ve never heard about it before, now they’re aware of it.
Elizabeth Figura 00:51:15 Absolutely. I’ve watched that transformation happen in real time, and it’s been surreal.
Jeremy Jung 00:51:20 Very cool. Well, Elizabeth, thank you so much for joining me today.
Elizabeth Figura 00:51:24 Thank you, Jeremy. I’ve been glad to be here.
Jeremy Jung 00:51:26 All right. This was Jeremy Jung for Software Engineering Radio. Thanks for listening.
[End of Audio]
