SE Radio 175: Game Development with Andrew Brownsword

Recording Venue: OOP 2011
Guest: Andrew Brownsword
Host: Markus

In this episode I talk with Andrew Brownsword about software development for (modern) games. At the time, Andrew worked for Electronic Arts, so our discussion is mainly based on the Need for Speed franchise. We discuss characteristics and performance properties of modern games and outline the challenges for software development. We then discuss various patterns Andrew and his team used to address these.

Show Notes

Links:

• Electronic Arts
• Need for Speed

Transcript

Transcript brought to you by IEEE Software

Markus Völter 00:00:35 Welcome listeners to another episode of software engineering radio. We are recording this one at the O O P or Uber conference in Munich, Germany, January, 2011. And, um, we were sitting in, in a room that’s intended to be used as a quiet area. So there might be other people coming in and being as quiet as we are. Um, I hope that won’t disturb too much. Anyway, our guest today is Andrew Brownsword from, uh, EA. And I guess you want to introduce yourself first?

Andrew Brownsword 00:01:05 Hello. Um, I work for electronic arts and I have worked for that company for approximately 21 years now. Um, we’re in the video game industry and we do titles such as FIFA soccer need for speed battlefield, um, and a variety of other titles in a variety of genres where we’ll roll a worldwide company. And, uh, I’m a senior software engineer in the Vancouver Canada studio.

Markus Völter 00:01:46 So I guess, uh, it’s obvious now what the topic is going to be for this episode. We are going to talk about, uh, game architecture and the kinds of games. I’m not sure if everybody knows what these title actually represent, but these are, let’s say actual games, racing games. So can you characterize these kinds of games a little bit? So differentiating them from let’s say SIM city or something, you know, is there a way of characterizing the kind of games you build?

Andrew Brownsword 00:02:20 Well, EA actually owns Max’s as well. So we do SIM city and the Sims and, uh, uh, our sports label games are, um, stadium based, baseball, football, American football, and a variety of other titles. Uh, and that typically the player is in control of a whole team. And he can switch between individuals on the team and get a, um, a three D view and control the player as the player runs around and actually plays the game and scores goals, hopefully, um, or whatever else you do in the sport. And then we also have titles like a battlefield or battlefield 1942 or battlefield bad company. And these are more in the, what we call the first person shooters, genre. And, um, they’re usually set in wartime or some science fiction genre, and you find weapons and try to achieve mission objectives. Um, we also have the need for speed line, which I’ve been personally involved in, uh, for the past several years, um, and that’s racing games and sometimes they are strength, sanctioned, street racing, uh, and sometimes they are sort of more elicit and there’s a storyline involved generally. And, you know, you’re an undercover police officer or something trying to, to break into the mob or something of that L um, yea has other games like the Lord of the rings series, um, which is sort of a fantasy role playing or a what’s called, um, RTS real time simulation. And it’s sort of multiple sides fighting Epic battles. Uh, we have also done the Harry Potter series and games like dead space, which is sort of a dark horror adventure kind of genre. So quite a variety.

Markus Völter 00:03:54 I guess, since you are mostly involved with the need for speed, a franchise or series, I guess most of the things we’re going to talk about kind of have a background in this kind of game. So if we don’t say anything else, we basically talk about racing games in some sense, although the concepts of course are applicable to the others as well, but that’s your background.
Andrew Brownsword 00:04:13 Yeah. My background in the last couple of years has been a knee for speed, and we also did a skateboard

Markus Völter 00:04:18 Escape. Yeah. Okay. So, um, if we look at these racing games, um, can you give us, I mean, maybe it should some say one more thing. You did a keynote yesterday about this topic and I saw the keynote. So I might, I might can kind of implicitly reference things that are saw in the keynote. Um, so one thing you showed was how these kinds of games have evolved over the last whatever 20 years from the pixelated, you know, beginning of the nineties to what we have today. Can you give us a little bit of a, you know, kind of timeline regarding, I don’t know, screen resolution level of detail, how these things changed?

Andrew Brownsword 00:04:57 The, uh, the state of affairs when I joined the game industry in 1990 was that the screen resolutions were approximately 320 pixels wide by 200 tall. Um, and to give you some idea on a typical TV or display, you’d be looking at it and you almost have to use your imagination to realize that this thing in the distance is a car. I mean, it, it, it was very vague and blocky and, um, the number of colors we could draw, you know, in the beginning was know on the Macintosh, for example, it was black and white and on, uh, on the PC, some of the displays only had 16 colors per pixel. So the, the, the range of colors we could show was very, very limited. And the computational capability of the computers was so limited. We couldn’t do even three D graphics, right? And so we would have pre drawn 2d sprites that we would draw on the screen in the correct position.

Andrew Brownsword 00:05:57 And when the car got car got farther away, we would actually switch to a different texture that was drawn to be smaller and tricks like that. And so the whole thing was an illusion of 2d layers displayed on this low resolution screen. And the audio was, was little better than blips and boops. And then, uh, shortly thereafter as machines started to become faster and heavier duty, math became possible screen resolutions, um, slowly started to increase. We moved to 3d and then a little while later screen resolutions jumped up, VGA arrived on the screen, and we were then up to six 40 by four 80. And we usually had enough compute power that we could run at 20 frames per second. And once you get to 20 frames per second, now you start to perceive real motion previously at 10 frames per second, it was pretty chunky. And you really had to use your imagination as you get the 20, and then hopefully 30 frames per second, as you know, movies are 24 frames per second. And that is sort of the threshold of where you can see continuous motion and actually the human perception. You can, you can perceive smooth motion all the way up to around 60 frames per second.

Markus Völter 00:07:11 If I go to the movies and I can see that I watch 24 frames film, and you see rapid movement, I think I perceive in, can you in continuities? It, it kind of feels, it looks chunky a little bit, or is that my imagination?

Andrew Brownsword 00:07:23 You’re absolutely correct as big pans and stuff like that happen. You can definitely perceive right at the edge of your perception and even the TV speed of, uh, well, the North American TV speed of 30 frames per second, it looks smooth. But when you see a 30 frame per second game versus a 60 frame per second game, there is a clear difference to most people. Some people have a hard time perceiving it. We’re all a bit different in our visual acuity, but 60 frames per second is pretty rock solid, smooth for most people.

Markus Völter 00:07:54 So that’s what you’re shooting for today.

Andrew Brownsword 00:07:58 So these days there’s a trade off and usually a game particular game has to make a decision. So resolutions have gone now far beyond the six 40 by four 80 and at a high definition to have television resolution. We talk about the number of vertical lines. And so we’ve gone from four 80 to seven 20 P and now to 10 ADP. And the frame rates of the TV is 60 frames per second or higher. Some of them display higher, but we generally can only send a new image to the TV at 60 Hertz. And the game has to make a choice because 60 looks a bit smoother, but it’s pretty subtle effect. 30 looks almost as smooth. And if you have twice as much time to render the frame of the game, you can get twice as much detail into a single frame.

Markus Völter 00:08:50 The actual picture might look better, but you get fewer of them. Exactly. So it’s a trade off between smooth motion

Andrew Brownsword 00:08:56 Versus quality of the image. Okay. Quite often games will choose quality, the image, and it really comes down to a game play choice, right? Like some games play better at six.

Markus Völter 00:09:09 I actually, I saw during the keynote yesterday, you, you gave a little demo of the need for speed shift. I think it was called correct. And, um, I have to say, I don’t follow games that much, except for angry birds, I guess, and flight simulator. But, um, then it was pretty impressive. I mean, it, it, it really, the level of detail that went into the rendering, especially, I mean, all these different shades and you showed this one slide with know scratches on the, on the car and stuff. So this was, it was really impressive. So, so if you listener, haven’t played one of these games reasonably try to, I don’t know, go to YouTube and look at one of them, make sure it’s a HD resolution is pretty impressive. So, um, anyway, so we’re going to talk about these kinds of games. And I guess we were going to talk about two or three aspects.

Markus Völter 00:09:57 One, is that the ingredients? So what goes into building one of these games, like, you know, simulation, rendering, audio, things like that. And then we’re going to look at some of the more current challenges you were looking at, which is concurrency and multi-platform, and then you have identified a bunch of patterns that you use when building these systems. We’re going to talk about some of these. Okay. So, um, let’s start with some of the ingredients and I guess the basis for a game like need for speed shift is obviously the, the simulation that goes on that kind of decides what happens as opposed to how it’s shown and how it’s rendered in a visual and audio way. So how, how do you go about simulating the physics? How exact is it? I mean, is it real physics? Is it all fake?

Andrew Brownsword 00:10:39 It’s definitely not all fake. Um, so underlying the game engine, there is a basic FIM physics simulation, and we call it a rigid body simulation. And at that basic level, there’s objects in the world and they’re modeled with momentum and they have, um, you know, tensors, and, and all these other physical concepts and the simulation is updated at generally 30 or 60 frames per second. Right. Um, and the physics engine is responsible for taking the forces that are applied to the vehicles and any other objects that are moving around, turning those into changes to the vehicles, velocity, momentum, momentum, and acceleration, um, and then, uh, applying gravity and then detecting when objects collide together and then generating reasonable responses when they have these collisions, as well as events like the noises in the sand, that particle effect.

Markus Völter 00:11:31 So the, the, the, let’s say the math that goes into simulating, this are real physical formulas, but the level, or the way you have to tune the overall amount of processing you have to do is basically the granularity of the object. Is that, is that fair to say, or do you actually do cheats on the, on the way you calculate things? So

Andrew Brownsword 00:11:54 At the rigid body level that I mentioned, the physics is actually fairly stable and some EA games, for example, use, uh, there’s a licensable havoc, a product called havoc that is a physics engine. You can buy, it’s a piece of middleware that you can license. And, uh, other games use internal internally developed physics. But the key point is at the level of the rigid body, the physics is pretty much actual, real mechanical physics, Newtonian mechanics, or whatever. Um, the place where the fudging and the tuning and the, the not quite real physics comes in is in the actual simulation of the vehicle. And it really sits on top of the rigid body physics, and it generates forces to move the bodies that are the car.

Markus Völter 00:12:42 So you don’t simulate the whole engine and the way the tires interact with the pavement, this is more or less approximated.

Andrew Brownsword 00:12:50 The engine is definitely an approximation. We don’t simulate the cylinders or anything like that. So we have curves that we go, and we looked at the manufacturers data curves, and we generate a rough simulation of a model of that. That’s a fairly high level model. Um, but you mentioned tire physics. On the other hand, we actually have quite sophisticated tire physics models that do, they’re not, you wouldn’t find a physicist. Who’s working for the automotive industry, basing car design on these things, but we are tuning it carefully so that it gives the right feel. And it exhibits quite a few of the attributes, the actual real tire physics exhibit, but we don’t take it to an extreme, right. That’s for sure.

Markus Völter 00:13:35 You said that, uh, fun and a good feeling is more important than actual, complete physical accuracy.

Andrew Brownsword 00:13:42 Absolutely. So it’s really key that the player feel like he’s driving the real car and strangely enough, trying to actually accurately reproduced the numbers that the real cars get on the track, doesn’t give that feel. Right? Part of the reason for that is because the driver, the player of the game isn’t sitting in his Audi or whatever it is, right. Driving around the track, feeling the G-Force is moving him, holding a steering wheel, unless you have a really sophisticated game set up, you’ve got a little game pad control on your hands. And through that, you are trying to experience racing around the track. Yeah. So it takes a fair bit of tuning, artistic license. And there, there is a, um, there’s a real art to defining the feel of a game to making it feel gritty and aggressive. And at the same time, most game players, aren’t professional race car drivers, and we want them to feel successful, right? So it is a game first and foremost,

Markus Völter 00:14:40 In addition to, um, the, the car and its interaction with the road, you also have to simulate things like, how does the car deform if you crash into a wall, and if you accelerate too fast, how does the smoke, you know, leave the tires and stuff. This is not really real, right? This is all, this is approximated these guys.

Andrew Brownsword 00:15:00 Yeah, absolutely. The, the damage is, um, designed to return a, a good visual look at something that looks impressive. Something that doesn’t break the illusion of reality, like the suspension of disbelief is a term we use the user wants to be in the game playing. And if you see something that just looks wrong, like you can see through the polygonal model of the game of the car, you don’t want to see that. So we carefully deformed the models so that we don’t break the illusion, but it still looks like a car that got seriously crunched. In reality, we don’t actually do much to the model of the car. So even though the front end may have been compressed in by six inches, we probably don’t actually move where the collision happens. We don’t need to because you can’t notice.

Markus Völter 00:15:48 Yeah. Yeah. And, and, and things like a particle. How do you say swarms kind of, you know, if you have many small particles that move together, that’s, you would use the swarm kind of algorithms where you define, you know, one where you say you have a bunch of primitive rules, how the particles interact with each other, and then you would kind of emerge, you know, that the baby had behavior of the small of the cloud of the particles would emerge from the, don’t do this for the smoke, probably just to detail.

Andrew Brownsword 00:16:16 Well, actually that’s not true. We have, we have specialists who their job is to set up these particle systems. And each of the particle kinds of particle systems has its own set of rules as you call them. Right? Little algorithms that govern the motion of the particles and smoke is actually a really important one to get the motion. Um, I know we’ve spent a great deal of time working out how to get believable volumetric smoke, and there’s different kinds of smoke. Some smoke comes out of the tailpipe. Some smoke comes off the engine after you crashed the, some of the most important smoke in a racing game comes off the wheels. And it’s different, whether it comes off the front wheels or the back wheels, all four wheels, depending on which, uh, which engine, uh, which wheels the engine is driving. Right. And so, um, you know, smoke actually curls around the tire, as long as you burn out and all this stuff has to be modeled. And it’s modeled not from physical correctness point of view, but from a believability point of view, and then it’s balanced against the trade off. If we can’t render an unlimited number of particles. So we have to carefully design them so that they represent a reasonable load to render and yet generate a pleasing effect visually.

Markus Völter 00:17:30 Yup. So let’s move on to a video presentation to the optics basically. Um, I guess one, I mean, one question I could ask initially, I know it’s not the case, but I can still ask it. Um, is this rate tracing going on? Is this actual simulation of light race or, uh, what of ways of rendering the actual scene are used?

Andrew Brownsword 00:17:52 So it’s all polygonal triangle rendering and it’s, um, it’s not right. Tracing a low for some purposes, we will trace rays. I’ll usually not for graphics strangely enough. Um, however, um, it’s a, a shaded lighting model and different games use different techniques. So some of the latest games are using what’s called deferred shading. Um, and the idea is you, you render all your triangles into the world scene, and then you apply lights after the fact on the rendered result. Um, traditionally in older models, you would render the polygons and as you drew each triangle onto the screen, you would figure out the lights that are shining on that polygon and color it appropriately.

Markus Völter 00:18:43 So, um, how do I have to imagine actually programming this visualization? Um, is it that developers actually let’s say drop pixels, or do you have pre prefabricated pre-rendered sprites as you call it back in the day? Do you have, um, certain patterns that you maybe apply to walls or to the street prebuilt? Or how do you, you know, how do you, on which level of granularity do programmers who implement the visualization algorithm? How, how detailed do they have to actually render each pixel? How does this work? I have no idea about graphics programming as you probably can notice.

Andrew Brownsword 00:19:19 So that’s yeah. So, um, so what we get from the art team is three D models, which are essentially a cloud of vertices in three D space. And, uh, these vertices are connected by edges, which form triangles and applied to these triangles is a bunch of properties. So lighting information or, um, specular information, there’s a variety of different properties that described the, the triangle. And there’s a generally a texture sometimes quite often, a whole layer of textures, some of which are simply the, the 2d visual image that you would see if you were looking at that part. So you can imagine a brick wall, and it essentially is a photograph of a brick wall, and that is drawn onto the triangle. Um, some of the textures are, are not directly physical representations. Things like a bump map will affect how light reflects off the texture.

Andrew Brownsword 00:20:24 So it defines shape so that we can then on the fly calculate how that surface is interacting with the lights that happened to be in the environment. So with the textures in the models that are produced by the artists, these are fit into the rendering engine and programmers interact with this first and foremost, by marshaling them in off disc, into main memory and then arranging for them to get to the GPU so that the graphics processor can actually use that data. And then they are fit into the graphics processor along with what are called shader programs. And there’s generally two kinds of shader programs. Um, there are new kinds coming, but first what runs on the program on the GPU is, uh, called the Vertex shader. So every Vertex in a triangle is made of three vertices, the three corners, um, every Vertex gets processed by this Vertex shader program.

Andrew Brownsword 00:21:17 And it does things like it takes it’s the vertices position in three space and works out where that’s going to get projected onto on the screen. And it’s going to calculate the normal, which is perpendicular to the triangle, um, and various other parameters. And these are then passed along to the second program, which is called the pixel shader. And the pixel shader gets the inputs from the triangle and that those are generated by the Vertex shader and a triangle of three of them. And what the hardware does, is it interpolate from one Vertex to the other along the edge. And so you get an array of pixels once it’s, rasterized onto the frame buffer. Yeah.

Markus Völter 00:21:59 So, so one thing maybe that most people probably know is that the abstractions you work with on the GPU really isn’t the pixel it’s, it’s these triangles, and there are layers and there are all kinds of 3d magic going on on the GPU itself. Right. So there’s not just pixel processing.

Andrew Brownsword 00:22:18 Yeah. There’s both the vertices get processed. And then the GPU does this interpolation magic and it works out exactly which pixels go into it. And then it runs the pixel shader to actually calculate the color that it wants to right.

Markus Völter 00:22:30 Put out the back. Yeah. So, um, GPS are actually quite sophisticated today. As many of our listeners probably know how much of the work of the overall rendering is done on the CPU and how much what’s the workshare between the two.

Andrew Brownsword 00:22:46 Uh, well, that would depend on how you measure it. Um, I would say that probably 90% or more of the computation is happening on the GPU, the vast majority of it, what the CPU is doing is figuring out what to draw and importantly, what not to draw and then submitting all of that to the GPU in the form that GPU can, can deal with. And the CPU ideally never actually touches each Vertex and never actually touches each texture. Now, there are exceptions to that, especially on a machine like the PlayStation three, where there is a lot more computational capability in the CPS, because then we want to load balance between the two. So we’ll move some of the calculations over onto the CPU, but fundamentally the GPU is doing all the work on the vertices and the pixel.

Markus Völter 00:23:38 Um, how do you program GPU? Can you write C code and run them on the GPU or how, I mean, it’s obviously there, these things are highly parallelized, so right. You have to have certainly parallel algorithms. We also have specialized languages for that, or is it a, is it C plus plus?

Andrew Brownsword 00:23:54 Well, it’s definitely not C plus plus. Um, so when you’re programming graphics problems, you’re either running a direct X where you’re running open GL, right in direct X, there’s a language called HLSL, which stands for high level shading language. And it is a very C like language that’s been extended to have a very common graphics primitives. So float fours, for example, is a type in the language. And there’s a bunch of built in functions that are commonly used for doing, um, graphics, calculations dot products and stuff like that. And so it’s a language carefully tuned to do the sort of calculations that you want to do in your Vertex shader or your pixel shader. And that gets compiled down into native GPU assembly code basically. And, uh, in open GL you have GLSL, which is graphics, library, shader language, and it’s very, very similar to HLSL.

Markus Völter 00:24:50 Okay. Let’s um, we will come back to the, to the graphics stuff and to the challenge of multi-core and generally parallelization later, um, let’s briefly move on to the audio. And, uh, yesterday you mentioned that, um, to render, if that’s the right word, uh, the audio, the sound of a car, you had some sometimes have 27 different kinds of sounds that you mix together to give a, you know, a good audio representation. So there seems to be going on more than I thought in the audio.

Andrew Brownsword 00:25:20 Yeah. So I pulled the number 27 out of the air. Um, but there are many, and in fact it might be more than 27 or different components. Um, there are many different elements that go into a single engines car sound. And when you consider we have multiple cars happening at the same time, there’s a very rich audio environment. In fact, uh, many times at any given moment, there might be a hundred or more sounds being blended together to sort of render your audio canvas as it were. Um, some of them are so incredibly subtle that you can’t pick them out consciously, but it really adds up to a immersive kind of environment. And if you’ve ever watched a movie without audio and then turned on the audio, you realize how massive and impact the audio environment has on the experience. So it’s really crucial. And, uh, the need for speed team in particular has won numerous awards for years on the, on the, the level of audio fidelity they have.

Andrew Brownsword 00:26:19 So for the engine sounds in particular, we have a internal proprietary technology that I can’t really talk about to model how the engine sounds as it revs through the, the range. Um, and it, it really, the emphasis is on engines that rev really quickly cause you’re in a racing car you want to hear and feel your engine screaming through the RPM. And you want to be able to hear when the blow off valve goes or the turbo goes, or that exhaust note as you hit that power band, there’s a lot of elements that really go into that. Yeah.

Markus Völter 00:26:54 I guess that’s something that most people don’t think of. I mean, visually it’s obviously impressive and you, you clearly realize that rendering nice visuals is important, but for the sound, I guess most people don’t, I didn’t think it was that important, but

Andrew Brownsword 00:27:08 Yeah. And that to no end drives the audio guys crazy. Yes. Okay.

Markus Völter 00:27:18 Uh, another thing that goes into games obviously is the game play itself. How do the, well, the obvious things is, you know, if you think about quake, how do the opponents behave? So there is this AI thing going on, is it really AI or is it basically scripted?

Andrew Brownsword 00:27:35 Well, it’s a funny question. Um, in reality, in most games it’s highly scripted and to a lot of us, that’s kind of disappointing. And 10 years ago, or almost 10 years ago, there was a, a lot of talk about this. Uh, what’s called emergent behavior and they wanted to have all the entities in the game and they all wanted to have their own behavior. And so you’d get things happening in the game sort of spontaneously that nobody had ever predicted or planned. Right. Um, and then the game designers went, but wait a minute, that means we can’t control it all. And so they sort of backed away from that and said, no, we want to control the player experience. Cause we want this carefully crafted really highly tuned experience. That is very enjoyable. And so it’s, it’s a strange dichotomy because the, the highly crafted experience is at odds with a smarter AI. And so some games really go down the AI path and they try to have smarter algorithms and more emergent behavior and that sort of thing. But it’s at odds with the story it’s at odds with the carefully crafted experience. So there’s a balance happening there. There’s definitely AI controlling the other opponent racing cars in a racing game, right. There is logic in there. That’s working out how best to drive around the corners. We give it a lot of hints, but they

Markus Völter 00:28:59 Simulated in the sense of, you know, what you talked about before with foresters and stuff, they are really scripted going around or do they, is there also a physics simulation engine kind of running for each of the other

Andrew Brownsword 00:29:11 Cars? There’s absolutely a physics simulation running for all of the opponent cars. Now it’s varies depending on games. And so various racing games in the past and the need for speed series. There’s been a varying degree of physics applied to each of them. And when you encounter traffic as an optimization, when the traffic’s far away, we don’t care so much about the physics of the traffic vehicles. So then they’re more scripted, right. But when they get close or all the opponent vehicles generally are going to be running the full blown physics and in some of the games, they are in fact controlling the car the same way the players control the car. Yeah. We’re actually inventing the steering and the gas and the brake inputs and the shift,

Markus Völter 00:29:49 Your controls, and it then goes through the same simulation pipeline.

Andrew Brownsword 00:29:52 Exactly. And it’s running. So if you have eight cars in the race, we have eight different instances of the physics

Markus Völter 00:29:58 In case anybody ever thought about where your computing power goes, that’s edited every 18 months, according to Moore’s law, I guess we’ve just found it

Andrew Brownsword 00:30:06 It’s games. Yes. Games and flash, I guess.

Markus Völter 00:30:11 Anyway, it’s impressive

Andrew Brownsword 00:30:13 Games. Have a voracious appetite for competed for consuming computing power. Yes.

Markus Völter 00:30:18 Yes. So, um, there is also other things that go into a game, the menuing system, the online communication with other players, uh, social media, uh, social, social, yeah. Social, total networks these days, these days as it just, we, we don’t cover that too much because it’s more or less straight forward. And we won’t have a lot of time to cover the more interesting stuff. But if you want to say something briefly about that,

Andrew Brownsword 00:30:46 Um, I mean, I sort of agree, actually, being in the game and playing the game to me is the really interesting part. And it’s a really interesting computational problem. Um, that said there’s a huge amount of complexity involved in all the social networking and all the online on all these aspects. Online is a fascinating technical problem. How do you deal with latencies and bandwidth limitations and whatnot, but I agree it’s a huge topic. It’s worth a huge amount of discussion. Um, but certainly we’re not gonna be able to cover it here. So,

Markus Völter 00:31:17 So then let’s move on to some of the, you know, let’s, let’s dig a little bit deeper into how you build these games. Um, so I guess a starting point is to realize that these, these games and the systems they run on are actually, you could call them real time embedded systems, right? You have predefined limited resources. And I think yesterday you mentioned that the PS three has half a gigabyte of Ram only

Andrew Brownsword 00:31:41 The PS three and the Xbox three 60, both have 512.

Markus Völter 00:31:45 It’s not a lot. I’m surprised. So, um, so in other words, I guess resource management is one of the main challenges in building such a game.

Andrew Brownsword 00:31:56 Absolutely. Uh, it’s been said many times, at least internally at EA that resort game development is primarily about resource management. We typically have four or five gigabytes of data sitting on a DVD or a blue Ray disc. Um, and somehow we have to use it all in a machine that has one eighth, as much as that. And so there’s a lot of loading or streaming or some other marshaling of any domain memory. And once you get in into main memory, you have to figure out how to manage your available CPU time and your memory bandwidth.

Markus Völter 00:32:29 Yeah. And do all of that within one 30th or one 60th of a second, you know, doing all these different simulation, audio, rendering, video rendering, sending it on to the GPU. All this has to happen within, within one, let’s say time slot

Andrew Brownsword 00:32:41 Exactly. Real time. Exactly. And, and, you know, it’s, it’s soft real time, as I said, because if you miss a frame, it’s not the end of the world. Yes. If you miss a frame consistently, it’s going to seriously degrade the user experience though. So we try to maintain that and interesting way to look at 30 frames per second or 60 frames per second is if you’re running at 60 frames per second, you have approximately 16 milliseconds, a little over 16 milliseconds to accomplish everything you need to do in that frame of the game. That’s your rendering, your audio, your physics, your animation, taking your inputs. There’s a whole lot of stuff, including online and all this other things. Yeah. So that’s a lot of work to do in a very short period of time.

Markus Völter 00:33:24 So let’s talk a little bit about the implementation, which languages do you use mainly

Andrew Brownsword 00:33:28 Almost exclusively C plus plus. Okay.

Markus Völter 00:33:32 And you do that because you like it, or because there is no alternative that gives you that kind of performance control,

Andrew Brownsword 00:33:38 Uh, on the game consoles, it turns out that there is no other alternative really

Markus Völter 00:33:46 Aren’t any compilers probably.

Andrew Brownsword 00:33:48 Yeah, exactly. So because the, the Sony and Microsoft are providing a C plus plus compilers and no other components. So yeah, C plus plus that’s it, the reason they’re only providing us those things, because there really isn’t an alternative language. And that means as a practical matter, all games are written in C plus plus. So all game consoles provide C plus plus, and we’re stuck in that ecosystem.

Markus Völter 00:34:11 Any, yeah, I guess that the question is useless then any frameworks tools you use to make your life simpler, I guess, since it’s all specific to, let’s say the peers three, there aren’t any frameworks or tools that Joe programmer would know, because these are probably not available for these platforms. So you have your own frameworks, but nothing like off the shelf,

Andrew Brownsword 00:34:31 We certainly with any EA, because it’s a large company, we have quite a few frameworks of our own. Uh, we also license frameworks, uh, EA in places has used, uh, the havoc physics syndrome, for example, that I mentioned earlier. And if you go and look in the industry literature, there are various other middleware packages that, you know, EA may or may not use, but other game developers do,

Markus Völter 00:34:52 I guess one, one, we will talk about design patterns or architectural patterns later. But one, as far as I know, one key building block in the way you build your systems is that you do implement the engines in C plus plus, but then the actual, uh, the scripting of opponents, the actual design of how things look, this is done in separate data driven styles, right? So you don’t want to change the C plus plus code if you want to change some parameters of how your opponent drives.

Andrew Brownsword 00:35:24 Yeah, absolutely. So many systems are at a minimum parameterized heavily, uh, and that sort of scales up to fully data-driven systems. And then in the extreme, we get all the way up to scripting languages. So in the game industry, a very common language to embed in a game engine is Lewa. For example.

Markus Völter 00:35:43 Yeah. I thought about it. I was actually planning to do an episode about that for years, but never got around to do it.

Andrew Brownsword 00:35:48 It is an excellent language because it embeds very, very well. And its performance is reasonable. We, we carefully control the scope of what we use it for. Right. Very, very specific tasks that aren’t performance intensive.

Markus Völter 00:36:04 You ever thought about using domain specific languages that you build specifically for your kinds of use cases? I mean, one key challenge. We’ll talk about it in a minute is parallelism, and I’m not sure where the Lewa has a lot of very useful abstractions for that, or maybe it might happen at the C plus plus world, but the other state same statement is true. C plus plus is not very good for parallel stuff. Have you thought about doing a domain specific languages where you may be even be able to, let’s say flexibly switch between generating C plus plus, which is fast, but in flexible and running it in an interpreted way, which is

Andrew Brownsword 00:36:36 Absolutely. Um, and over the last decade or two, um, what has happened is as systems become more and more data driven, they’ve eventually gotten to the point where we belatedly realized that they are essentially domain specific languages. They become data that drives the C plus plus engine, but they are a limited language. They’re not Turing complete by any stretch, but they are a DSL effectively, but we got there in reverse, right. We just progressively parameterized it until, Hey, guess what? We’re almost at a language, rarely do we start at the language side and design backwards. We’ve done that for gameplay languages on occasion. Sometimes we use Lua, but other times we have internal languages that are purpose built for that kind of thing, because maybe we don’t want to have that much generality, or we want to figure out how to make it perform better than the generic Lua, for example. Um, and certainly it has been a topic, uh, internally about, can we do any better? Can we get performance improvements, like rather than, you know, Lewis slow and so try to avoid using it only using it in strategic places. Can we actually turn around and create a DSL that gives us a performance when I have yet to see that happen in practice. But actually the second talk that I gave this morning was essentially about that topic. Hmm. Okay. And I believe it has much broader implications to the computing industry. In general,

Markus Völter 00:38:13 I’m asking for two reasons. One is that my own area of expertise and work is domain specific languages. So I’m always interested in that. The other thing is yesterday morning, I interviewed Martino Gursky who does Scala. And he just won a research grant for developing domain specific languages for really domain specific concurrency and parallelism as scholar embedded DSLs. So this topic about DSLs and higher levels of abstraction and more clever libraries and translation engines,

Andrew Brownsword 00:38:42 Um, is

Markus Völter 00:38:44 Clearly something that that’s on the, on the rise. And so I was just curious.

Andrew Brownsword 00:38:49 Yeah. And I’m actually a big proponent of that. And at the end of my talk this morning, somebody actually specifically asked about Scala and its ability to, or it’s a framework that it provides in order to build the ASLs within it. Um,

Markus Völter 00:39:03 They are, I mean, kind of off topic here, but, uh, other ski told me that they are actually going as far as not running the embedded DSLs that are embedded in Scala S scholar programs, but these DSLs just build a syntax tree and then they have a compiler to Kuda or to C to really build completely separate programs.

Andrew Brownsword 00:39:22 Well, that’s very interesting to hear, because this morning I was suggesting that they should generate open cl yes. And EA is in fact, a member of the open cl standards committee. And this is a cross platform platform that I believe we can target. And it presents a, um, a tasking model similar to one of the design patterns that we’ve adopted within EA.

Markus Völter 00:39:45 Okay. I guess we should put this to the discussion when we shut down the podcast here. So, um, we do, you talked about, um, concurrency already. So I guess one of the central challenges of what you’re doing is multi-core not just, I mean, if you run brightening a PC game, you have multi-course in the, in the CPU, but I guess these three views are highly parallel paralyzed already. So any, how do you deal with that?

Andrew Brownsword 00:40:11 So we’ve been dealing with it since the late nineties when GPS first started to appear on the scene. And the first step was let’s just offload the actual rendering. So back in the mid nineties, we were actually rendering every single pixel as CPU code, but when hardware started to become available, that would do it for us, that hardware now started to become able to run asynchronously. And that means the CPU sets up the work, hands it over to the GPU. And now you have two processes happening in parallel. Now that’s not the conventional multi-core model, but it definitely gets things happening asynchronously. And you start having to figure out how to partition the work effectively, the arrival of the PlayStation three and the Xbox three 60 was a bit of a shock to the system for us because all of a sudden we were confronted with our primary market, the machines in the primary market, having three or six or eight hardware threads.

Andrew Brownsword 00:41:10 So you had to parallelize existing code, I guess, have you didn’t rewrite everything? Well, it turns out paralyzing existing code is really hard. And so in practice, what we do is we find pieces of a game engine and we extract them out, turn them into tasks. And one of the design patterns that we’ve adopted is a, we call it job manager within EA, but it’s called a task graph or a task manager. And what you do is you define a specific piece of a program, a Colonel, a computational Colonel, if you will, and then you schedule that to run on other processors, right? So your main line code decides there’s a bunch of work that has to be done, puts it in a queue to get done. And then the work processors such as the SPS that are found in the cell processor in the PlayStation three are constantly pulling work off of those.

Markus Völter 00:42:02 And then you just have to manage the queue. Doesn’t get too big and stays on real limited. Then you know, that it’s actually

Andrew Brownsword 00:42:07 That’s right. And then sometimes you have to pick up the results as they come back. Although some tasks just go to the GPU and they never actually come back to this.

Markus Völter 00:42:16 Um, another design parents, since we’re already talking about it, um, you mentioned yesterday, are these vector based interfaces. So if I get it correctly, you can let me know if I do or don’t, uh, instead of saying, here’s an object, I call a method. You say here’s a bunch of objects. I want the following method to be executed on each of them. And then some lower level can find out whether there is only one core or threat to do a sequentially or several do it in parallel. That’s kind of right.

Andrew Brownsword 00:42:45 That’s kind of right. Um, part of the problem that we’re trying to solve here is you don’t just get currency from having multiple processors, individual processors, individual threads, hardware threads, um, have concurrency within them. Some of that shows up as Cindy instructions. Some of it just shows up as, um, pipelining and overlapping instructions and concurrent memory accesses. So there’s all these different elements of parallelism. And when I talk about it, I tend to use the label of vectorization when you’re going to optimize for this kind of concurrency that’s at the instruction level, I just lumped that into it and call it vectorization. And so when you take an algorithm and you vectorize, you’re trying to take advantage of all the concurrency that you can get at the instruction level at the memory organization level, simultaneous transactions on the bus, because things can overlap on the memory bus and that kind of thing.

Andrew Brownsword 00:43:40 Once you get all that sorted out, um, we, we find that many algorithms, data, parallel algorithms, at least we can get a 10 times performance speed up by doing that sort of optimization. The problem is if you haven’t designed your software to do a lot of the same calculation to the same kind of objects all at once in a batch, you can’t apply those sort of optimizations. So if, um, uh, an example I use all the time is calling, which is essentially the process of deciding what to draw and not to draw. So if you have thousands of things to draw, but you can only afford to draw a few hundred, you have to quickly toss out, toss away the things that don’t need to be drawn. Cause they’re not in the view frame or whatever that process can be quite expensive. And if you walk through your graph of objects and as you each object, you say, Oh, call your call method to see if you’re visible or not. Then it goes through this call chain. And eventually it says, what’s the view for us drum. What’s my bounding box. Do the calculation to see if it’s visible. If you do that thousands and thousands of times at 60 frames per second, you find you quickly use up all your CPU,

Markus Völter 00:44:52 Wait, 18 months to get a faster one. Yeah.

Andrew Brownsword 00:44:54 And we just don’t have that option. And in fact, single processors, aren’t getting that much faster. And

Markus Völter 00:45:00 So the game consults haven’t been, uh, you know, changed much in the last six or seven years. Anyway. I mean the Microsoft Xbox is ages old.

Andrew Brownsword 00:45:09 Yeah. So I mean our console generations last six to 10 years or whatever they last. And so we have to get faster and faster on a given machine. And that means that what you really need to do is you need to identify the large pieces of work and you need to batch up the calculation. And once you have it all, as instead of going through a thousand objects individually, calling them what you want to do is you want to have a thousand and bounding boxes all at once. And then you can write one high-performance algorithm that all at once does a thousand bounding boxes or whatever your calculation

Markus Völter 00:45:43 Does destroyed the object oriented purity of your object model a little bit, because you have to, you know, define different kinds of interfaces.

Andrew Brownsword 00:45:53 Absolutely. And not just a little bit, but it can, it can completely derail your, your object during the design. Right.

Markus Völter 00:46:00 Another challenge, I imagine you mentioned peers, three X-Box PCs. How do you get the platform? Independence is C plus plus good enough to kind of transparently do that. Or I’m also, I mean, I guess the parallel nature, I mean, some have several cores, other stoned. So you have to even have different ways of parallelizing things depending on the platform. How do you handle that?

Andrew Brownsword 00:46:22 This is where the 80 20 rule comes to the rescue. So the basic rule of thumb is 80% of your performance happens in 20% of your code. That means the other 80% of the code isn’t especially performance intensive. So you write it in C plus, plus you try to make sure you don’t do anything platform specific. And for the most part, it ports pretty well to all the games. And maybe you provide a layer of portable libraries, the high differences between file systems and memory management and all that sort of stuff. That 20% that’s performance intensive. Um, typically you’re going to do different versions for every platform and there might be some tools that help us stay agnostic to the platform. But yeah, there’s going to be a fair bit of, of machine specific tuning.

Markus Völter 00:47:13 Of course, all the stuff you do with your DSLs and data driven kind of stuff is obviously platform independent anyway. So that’s not a big deal.

Andrew Brownsword 00:47:21 The implementation of the DSL, if that’s what you want to call, it may be tuned for every platform. Yes. Okay.

Markus Völter 00:47:27 Hmm. So let’s look at a couple of more of these patterns you’ve found or implemented or used. And I guess one pattern you, I think didn’t explicitly mention yesterday, but which I kind of jumped like into my eyes, this whole rendering pipeline kind of thing is obviously pipes and filters, right? I mean, this is just one step, create something that’s piped to the next filter or stage, which has then piped to the next one. Right. Well, you would think, okay.

Andrew Brownsword 00:47:56 Um, so different rendering engines take a, a different approach to these things, but, uh, generally the rendering is broken down into your rendering different surfaces. And so for example, you might render a environment map. And what that really is, is it’s a separate rendering of the world from the point of view of being inside a shiny object, a car, for example. So you, you choose a wide angle lens and you render the world in a low resolution of detail. So now you have this picture of this, of the world that’s around the car. And then once that’s rendered, you use it as a texture that you then feed into a later rendering pass. When you’re actually going to draw the car that the user’s going to see, and you use that reflection in your pixel shaders to reflect the world. So when you’re looking at the car, you can actually see the buildings that are going by and it makes the car look like it’s in the world. And so it’s kind of like piping from one stage to the other. Um, but it’s not actually done using that sort of terminology or that sort of mechanism in the, in any rendering engine I’ve seen.

Markus Völter 00:49:12 And I guess, um, we should look at some of the patterns you do use, and it’s easy for me to name those because they were on your slides. One of them was the SIM simulation presentation split. That sounds like model view controller.

Andrew Brownsword 00:49:25 It is quite similar to model view controller. Um, you have the simulation and it’s essentially updating the game state. And so you run the simulation for one frame, and now you have in some form, another sitting in memory, maybe spread throughout memory, but you have this state of your simulation world. And the job of the presentation is to render that both audio and graphically and the force feedback on the controller. So games that are using this pattern, we’ve just covered, um, have a fairly formal split between those two halves and they try to carefully control the data that passes between them so that this data can be captured, can be used for a replay, for example, um, plus because the two are decoupled like that, once the simulation finished, it’s doing its calculation, it hands, this nice little package of data over to presentation. And then it can, because the program was concurrent on multiple threads, the simulation can carry on with the next frame. And thus you’re overlapping the calculation of the next frame with the rendering of the current frame. And you get parallelism at a coarse grain level out of that. Everybody’s busy all the time. Exactly.

Markus Völter 00:50:38 You mentioned his services. I don’t remember what it was actually.

Andrew Brownsword 00:50:41 Yeah. So services are like an animation service or a rendering service or an audio service. And so they’re essentially just singletons in the program. They are factories for managing the kind of objects that you would have, um, in any given subsystems. So the, you know, the physics service manages rigid bodies, for example. Okay.

Markus Völter 00:51:02 Another one is the data driven system stuff, which we already talked about. Yep. Um, we already talked about top manager and vector based interfaces. Yep. The last one I wrote down was the pre-baked data.

Andrew Brownsword 00:51:15 Yeah. The key here is any computation that you don’t have to do while the game is running is a computation you don’t have to do well. The game is running. It’s a pretty simple concept. Yes. So basically it says if we can precalculate stuff. And my favorite example here is lighting information. So a lot of games have fairly intricate lighting and sometimes the, um, the calculations that go into lighting can be very, very intensive. Uh, one of our games, the lighting calculation to light the entire world would run for three days on high end PCs. So you can’t do that in real time on any machine, much less on a $500 game console over $300 now, cause they’ve all come down in price. But so what we do is we precalculate it all, we store it in highly optimized data structures. And then we bring in an off of disc and we feed that pre-calculated data to the, to the rendering engines.

Andrew Brownsword 00:52:14 It’s not all completely canned. There’s still some dynamic calculations happening based on that. Um, but the idea is get your data into exactly the form that the game machine needs it rather than a more generalized form. And an extreme example of that is week often pre-baked data to the point where we can literally take the file. That’s sitting on disk copied into memory and begin using it immediately. There’s no serialization or DC realizations, memory image, direct memory image sitting on the disc. And sometimes there’s a bit of tweaking we want to do fixing up pointers and that sort of thing. But by and large, the closer we can get it to the memory resident image, the better cause then you just issue a read to the disc. The operating system delivers the block of whatever megabytes and you get a pointer to the beginning of it. And quite often you can just pass that straight to the GPU and it starts drawing. Right?

Markus Völter 00:53:08 One thing that you showed yesterday is also the, um, the skateboard guy driving around on his skateboard. And, um, I wonder how the movements actually are kind of diverting back to them, uh, you know, to the simulation thing here. Um, how the, the guy who rides the skateboard is rendered, is that described algorithmically or do you use motion capture or something like that? So

Andrew Brownsword 00:53:32 First of all, what do you mean by rendered

Markus Völter 00:53:34 Wrong word, finding out how he moves.

Andrew Brownsword 00:53:37 Right. Okay. So, uh, in skate it was a combination of prerecorded, um, motion captured animation. So motion capture is a process whereby you get an actual human actor, you put them in a funny black suit with white markers all over them. And then you have special cameras that track those white markers. And he goes, and he does the moves. So we have a motion capture studio at EA in Vancouver, and we’ve had all sorts of football players and American football players and baseball players and golfers. I think we had tiger woods in once, awhile back, um, and skateboarding. We had the skateboarders and they come in and they, they do the various moves that we want to represent in the game. We have cameras from all sorts of angles and we capture the data and a whole bunch of software does a lot of preprocessing and turns it into three D data. Then we take that animation data and we build up a database and in the game, when a player wants to run in a straight line and then turn right, we go and look into the database and we find run in a straight line and we find turn right, and we can play these things back. And then, then there’s all sorts of techniques to blend from one animation to another, combine them in different ways. So maybe his legs are running and his upper body is doing something different, like shooting a gun

Markus Völter 00:54:59 Yesterday. It looked pretty good. Although I thought it was a little bit too fast, if that’s the right word. It, it, it, it seemed to me as if the bodies that moved were massive enough, you know, some of the hours, the word track had in German inertia, right. It felt like that was a little bit, it felt like it was a bit, a little bit too light somehow. I don’t know. Do you know what I’m talking about?

Andrew Brownsword 00:55:23 Yeah, I do. And it’s a really finicky tuning issue when building games, because you want to have fast action and very rapid response to user input. Um, but there’s a trade off. If it’s too fast, then it feels like it’s too light or too many verbal. And so there is a real game play balance going on

Markus Völter 00:55:46 He’s realistic versus funny or fun. Exactly. Right. Okay. So let’s briefly talk about the process you use to develop these games. I mean, it was probably a couple dozens of people at least involved in this game. And I guess, um, you also have like stakeholders, we’re not programmers like artists and, and, and, and, and things like that, or people like that. I imagine those being quite interesting as customers from a programmer’s perspective.

Andrew Brownsword 00:56:11 Yeah. It makes for very interesting and dynamic teams and, and it’s, it’s a hugely creative and productive environment. Um, I have a hard time imagining what a team of just software engineers is like after spending so long in the game business. Um, because as a group software engineers are typically quite left-brained right. They’re very logical and methodical and you would hope so, at least you would hope so. Artists on the other hand are much more right. Brain they’re much more, uh, holistic and visual. And that their approach to problems is very, very different. And you bring these groups of people together, and at times they clash, they just don’t understand each other. But once you get a team that is mashed together and working effectively, it’s intensely creative because these complimentary skill sets. And so it’s a, it’s a great environment. Um, for the games we’ve been talking about, the team sizes are typically anything from 20 to 50 engineers or 20 to 60 engineers.

Andrew Brownsword 00:57:13 Um, at least that many artists, modern games for the PlayStation three and Xbox three 60 are extremely content heavy. So the, the amount of like a movie already, almost absolutely in the amount of people you need to build all of that content. I mean, you know, every single building you see next to the racetrack, the racetrack itself, all this data, all the, the textures that go on the polygonal models, all of this stuff has to be created out of essentially whole cloth. And that just takes a really large team. And then you have animators who are responsible for tuning, the animations that come out of the motion capture studio. You have the people in the motion capture studio, you have our audio people that are generating the sound effects, like the skateboard going across the sidewalk or the engine sounds. And you have guys that go out and record the sounds of actual car engines, and they have all sorts of stories about, you know, flying over to Europe and going into Ferrari and getting sounds of the latest and greatest engines. Lucky guys, I wish I could go, but, uh, so there’s all different parts of the team. And so I think at largest need for speed team, I saw it was approaching 200 people and it defends depends on where you define the boundaries. So if you include marketing people in it and then all sorts of others, it can be quite large.

Markus Völter 00:58:36 I guess then the, the distinction between the engine in C plus plus, and the data driven higher level stuff, and maybe the runtime tuneable parameters, it was also a way to decouple these kinds of people, at least a little bit. So you don’t have to have the C plus plus programmer involved whenever the artists, you know, this guy needs to move a little bit faster or a little bit slower or something.

Andrew Brownsword 00:58:56 Yeah. Sometimes I wish we were more decoupled, but, um, yes, absolutely. The reason that you want to expose those parameters is because the key to building a good game is rapid iteration because what, there’s no magic formula that makes a great game. And what you want to be able to do is try things rapidly. And by trying things rapidly, that means you get to experiment with what works better. And so there’s rule of thumb that basically says the more times you can iterate the better. And that means speeding up the iteration time as a program, or think about it as your compile link run cycle. The faster that is, the more times you can, you can find and fix bugs and you can refine your code and make it perform better. It’s the same thing with game design, you can go in and you can refine the art and the gameplay and all the different aspects.

Andrew Brownsword 00:59:49 So by exposing tuneable parameters, especially those tuneable parameters, which can be done during game play. Like ideally, while the guy is driving the car around, he can be changing the coefficient of friction on the tires. So we get to the right feel of it, right. If he can just keep driving around the track and he never has to quit the game and rebuild and redeploy, right? You don’t want to do any of that stuff because it can take a very long time. So ideally, while the game is running, he can tweak all those parameters and get just perfect.

Markus Völter 01:00:21 So we have to leave this room in about five minutes, so we’re almost through anyway. Okay. So last question. How do you think will games look in five or 10 years? I mean, it’s obviously a question you get a lot of many times, but still, I mean, what do you think, except of course, you know, framework, frame, rail frame rate, we’ll get hired level of detail we’ll get higher. Is there any fundamental change coming or is it just going to be, everything’s better? 500 different sounds.

Andrew Brownsword 01:00:51 So, first of all, I don’t know that I agree with that frame right at 30 or 60, there’s not really much reason to go higher. So we’re, we’re pretty much at a good point. Resolution might go higher, but really we’re aiming at HDTVs. We don’t expect to see a lot of resolution increases there. So there will be some increase in fidelity, but we’ve already hit a point of diminishing returns, right? We’re at a point where if you double the computational power, we can throw at the graphics. We haven’t made it look twice as good. In fact, this last generation was a bit of a struggle for us because we’re at that point of diminishing returns. Yeah. They do look better. Yeah. They are in higher Rez, but what’s the next generation going to include. And my speculation is we’re going to see the differences in behavior. It’s going to move better. It’s going to do things smarter. It’s going to behave in more believable ways. And that’s going to be a challenge because it’s easy to sell graphics. Right? Right. You can show a screenshot, say, look how much better that looks, but behavior, you can’t show screenshots. Right. And it’s a more subtle effect. So I think it will lead to better gameplay, but it’s going to be a hard, it’s going to be a hard story to sell to the consumer. Why what’s the improvement? You can do

Markus Völter 01:02:09 Three D games, you know, I mean, really it fit you with the classes

Andrew Brownsword 01:02:13 And our games are doing three dams and, you know, performance improvements will help there cause essentially you have to re render the twice. Right.

Markus Völter 01:02:20 Exactly. Okay. Um, I think we’re through anything else you want to say, you know, like famous last words, uh, at the end, uh, I’ve had a great time in Munich. This is a beautiful city and I’ve, uh, very happy to have had chance to visit. Good. Great. And I’m happy to have had you or going to have you on se radio. It was a very nice conversation. Thanks.

[End of Audio]

This transcript was automatically generated. To suggest improvements in the text, please contact [email protected].