SE Radio 517: Jordan Adler on Code Generators

In this episode, SE Radio host Felienne spoke with Jordan Adler about code generation, a technique to generate code from specifications like UML or from other programming languages such as Typescript. They also discuss code transformation, which can be used to migrate code — for example from Python 2 to Python 3 — or to improve its internal structure so that it conforms better to style guidelines. Adler is currently the Engineering Director for the Developer Engineering team at OneSignal, and he was previously lead API Platform Engineer at Pinterest and a Developer Advocate at Google.

Show Notes

Related Links

• Jordan on Twitter https://twitter.com/jordanmadler
• Jordan’s website
• Python future
• YelliCode

Related Episodes

• Episode 452: Scott Hanselman on .NET
• Episode 474: Paul Butcher on Fuzz Testing
• Episode 379: Claire Le Goues on Automated Program Repair

Transcript

Transcript brought to you by IEEE Software magazine.
This transcript was automatically generated. To suggest improvements in the text, please contact [email protected] and include the episode number and URL.

Felienne 00:00:16 Hello everyone. This is Felienne for Software Engineering Radio. Today with me on the show is Jordan Adler. He has been a professional software developer since 2003. He’s currently Engineering Director for developer engineering at OneSignal. Previously, he was API Platform Engineer at Pinterest and developer advocate at Google. Welcome to the show Jordan. Today’s topic is code generation. So let’s start with a definition. What for you is code generation?

Jordan Adler 00:00:46 That’s a great question. So code generation is a technique you can use in software engineering where essentially your software is producing code as an output rather than some kind of expected user behavior. So for example, a common code generation technique would be transpilation wherein unlike a compiler, which compiles programming code into machine code, a transpiler compiles or translates programing code from one language to another. So a common one of these would be a TypeScript, right? A TypeScript converts into a JavaScript who conducts some type checks along the way. That would be an example of transpilation which is a type of code generation.

Felienne 00:01:33 Yeah, that’s really an interesting question and answer for example, because that leads to the question, like why are we generating source code? Why are we not just typing source code? Right. So what is the benefit of generating JavaScript from TypeScript or in other contexts generating certain pieces of software? If we can also type that, right. I get it for assembler, no one wants to type bit code or assembler, but why JavaScript, it’s fine. Why are we generating this?

Jordan Adler 00:02:00 Yeah, there are lots of different reasons to do that. You know typically the answer is productivity of one reason or another, right? So if you are trying to write piece of software and there’s a lot of duplicate code in that piece of software, perhaps it’s duplicated because you are one of five different teams, each trying to build a system and they all interact with each other and maybe they use different languages, but they all have the same kind of interface, with the same specified method of interacting with each other, you might want to procedurally generate a kind of that interface code so that when you actually change the way that the servers communicate with each other, you only have to change them in one place instead of five places. So that’s a common reason. Another common reason could be to, like I mentioned, with the TypeScript JavaScript, perhaps you’re conducting some kind of checks and in the process producing code that is consumable by some other tool.

Jordan Adler 00:02:54 Another example might be lots of folks have Kubernetes, YAML, right? That becomes unwieldy and repetitive after a while. And so there are tools out there that can actually produce Kubernetes, YAML for you based off of tempering. And so that process effectively generates code, declarative code that is kind of Kubernetes consumes. And so there’s a lot of different kind of reasons people might want to do this, but typically they boil down to productivity. You have some kind of machine or some kind of system that expects — either kind of a computer system or system of people — that expects, kind of, code to come in at one way and transpilation can kind of enable you to fit that standard, or it’s a technique you can use to fit that requirement while reducing the cost actually.

Felienne 00:03:38 Yes, generally it’s quicker. And it might also be less error-prone because you can do some checking before you actually generate the code. So you know you’re generating correct code for a definition of correct.

Jordan Adler 00:03:49 Absolutely you test for correctness, you can duplicate code, so you can kind produce multiple different versions of the same input, right? So the process of doing that as opposed to having someone write it out, is a lot quicker and less error-prone. Absolutely.

Felienne 00:04:04 Yeah. That makes sense. So you already sort of hinted at some concrete examples, but can you give a certain example of a situation in which you use a code-generating tool to solve a specific problem?

Jordan Adler 00:04:17 Yeah. So one example would be we have this tool called clitool that we’ve built, sort of a prototype, and what it does is it creates a — it injects, kind of, the code into an application to add an SDK into the application. So we have the code base — so, Android app or iOS app, for example; you can run this tool, it’ll scan the programming code for that application and inject, or conduct the right changes to actually inject the required changes to the code to be able to include the SDK. So this is a kind of code-transforming process or technique — a code transformation where you take one piece of code, you output another piece of code, but you’ve modified the code in some way; not unlike transpilation, but the difference here is we’re not converting from language to another, we’re just kind of keeping it in the same language. Maybe we’re semantically changing the behavior of the application.

Felienne 00:05:15 Yeah. So we’re like enriching an existing code base with some features. And later in the episode, we want to dive into code transformation specifically as like a separate process from code generation. I’m also wondering like, are there anti-patterns? Are there situations in which you would say that code generation might not be the right solution?

Jordan Adler 00:05:38 Yeah. I mean, oftentimes it adds quite a bit of complexity, particularly in your build tool check. So, if you have a situation where you think you might be able to save developer time by code generating some piece of the code base before kind of building and producing it, now that kind of adds on to your build process. So that can add time to each build that you do, both in terms of when the software is actually shipped, but also in terms of development, right? So you kind of have a local development loop — you have to build, you have to test, you have to iterate, you know, if you have kind of code generation in the mix during that kind of tight developer loop, it’ll end up taking longer. So, oftentimes the trade-off here is yes, I’m spending a lot less time writing code, but I’m spending a lot more time waiting for code to be generated. That is a trade-off that you have to make potentially. And the productivity gains will have to outweigh the cost of both establishing the code-generation pattern, which is complicated certainly and rife with issues, but also in terms of the cost of kind of using it and maintaining it, which includes quite a bit of complexity in the build chain and the time cost and execution of that chain.

Felienne 00:06:52 Yeah that makes sense and I want to talk about this whole build process of code generation also deeper in the episode. But one question maybe that sounds a little bit abstract still for people that have never used code generation tools is like, what does a code generation tool look like? Do I write code to generate code? Or is this a visual tool where I sort of collect the interfaces together and then it generates code from a visual model, from something like UML? What is code generation look like, practically?

Jordan Adler 00:07:23 That’s a great question. You know I think in practice, all of those are kind of common UIs for dealing with code generation. There are tools that you can use, kind of in a one-off basis — visual tools, for example, to build out, say, SQL specifications, like a set of SQL statements to create tables. There are a lot of tools out there, table designing tools that produce as an output some kind of SQL statement or series of SQL statements that can be consumed by a database. That is a case, certainly. Another common one — perhaps the most common one — again, going back to the IDLs case, if you have something like Swagger, which is an API specification (open-API specification previously called Swagger), you can have in YAML or JSON a definition of a REST API and run a CLI tool that procedurally generates from that specification client libraries or perhaps servers or pieces of server code that is then consumed by a Java application that fills out stubs of that interface, right? So it can vary in terms of interface. It can be CLI-based; it can be GUI-based. It can be something you use once as part of your development process and never use again. It can be something that you use every single time you build, and it can be something you use manually when you pull something from upstream. It’s a technique that could be used in many different ways, for sure.

Felienne 00:08:48 Nice. So that gives us a lot of ways to apply code generation in projects. Now we have generated code. So the code has been generated with one of the variety of the tools that you just described. So then now what? Do I manually read this code? Is there some sort of verification, or do I verify the generation? What do you do in that case? Like, do you ever look at the generated code? Is it ever necessary to inspect that or is it sort of correct by construction?

Jordan Adler 00:09:17 Oh, absolutely. And you know, you can establish a pattern by which you can kind of procedurally generate code and then have that be tested in a way that enables you build confidence that it’s error-free. For example, when I was at Pinterest we were using code transformation to convert all code base from Python 2 to Python 3 as part of the migration we were doing at that time. And that process, you know, as we were kind of converting bits and pieces of the code from Python 2 to Python 3, we could deploy a piece, you know, convert a small chunk of it, deploy it to a portion of our overall fleet — let’s say 2% — and then if 2% of our fleet is running this new version with these new modifications and it’s getting all the same API requests and returning all the same outputs and not having any new errors, not producing any new issues, we can probably say that it’s safely kind of consistent between the two versions, and we deploy it. So, in cases where you have a deploy process where, you know, canary-like, or have some other processes, statistically eliminating kind of risk and you can move forward carefully, then automating the process of deploying code generations is not unreasonable.

Felienne 00:10:35 Yeah. And so I wanted to say, like, this is a situation in which you already have running code — you have a baseline, right? — and you know what it’s supposed to do and you can migrate parts of it, but this is, of course, not always the case. So, I was wondering if you also have examples of experience with sort of freshly generating code where you do not have a baseline to test against?

Jordan Adler 00:10:55 Oh, absolutely. And in most cases you really should manually inspect your code. So, even when we were working at Pinterest on this this project to convert from Python 2 to Python 3, we were routinely manually inspecting the changes that were coming through. And honestly, like, some of the code transformation we had, they were not error prone at all, right? They were fairly straightforward — you know, convert this function, add parenthesis after print so it’s no longer a statement but a function. That’s a pretty straightforward thing to change until you start throwing in complexities like, well, what if we have our own function called print that we shadow, right? So we have kind of monkey patched our own print function. Or what if we have some kind of special label in our code called Print that, you know, we’ve modified in some way, or what if we have function calls that look like print and perhaps the regex that we used to convert the code or, or whatever technique that we used to actually implement the code transformation was a little overzealous and so we have an error?

Jordan Adler 00:11:57 And so, we’d often kind of run through and manually review all the changes as part of our PR process that would actually happen. However, if you were to run code generation in automated fashion… For example, we have, at OneSignal, API client libraries that I mentioned — again, that we procedurally generate from opening from openAPI specification files — and so, the output of that can change from version to version as we pull in changes from our upstream openAPI generator Open Source repository. We pull them in manually. We rerun the code generation and then we review the changes that occur before landing them because you can’t say for certain what the changes will be. So that is more of a manual kind of review process than something like sort of a canary-based or even kind of the PR inspection, which is much more kind of scrolling through thousands and thousands of changes and looking for outliers, as opposed to kind of really deeply inspecting every single line that’s changed trying to understand it.

Felienne 00:13:04 Yeah, that makes sense. And I guess there’s also a difference between if you are the person that is authoring the code generation tooling, or if you’re simply using something that has been extensively tested, then probably you can rely a little bit more on the fact that the generation will be correct because it has already been tested by many other people.

Jordan Adler 00:13:23 That’s a really great point, Felienne. And I think you’ve hit on something interesting about code generation, which is that it often involves collaboration between people. It’s a technique that is pulled out when two teams or two groups or two pieces of software have to interact with each other — two or more really — and so, having that kind of consideration of ok, where is this code coming from? Who wrote the code generator? and understanding that is as much of a process of understanding how to integrate and deploy this technique in your code base as anything else.

Felienne 00:13:56 So let’s talk about practicalities. Yeah. You already mentioned that this code generation will then be part of your build process, which might be time consuming, but also you get some interesting questions like what do I do with the generated source code? Do I check this in to version control, or is this typically something that you would put in and just ignore? Because, well, if you need it, you can just generate it again. I can imagine that for reasons of traceability, maybe, you also want to ship the generated code so you’re sure that everyone looks at the same version of it? What are your best practices there?

Jordan Adler 00:14:30 Yeah, I think it’s going to vary. I don’t think there are kind of standard approaches. Again it’s an unfortunate answer when it comes to code generation and transformation and really kind of more broadly, compilation and consideration of managing code, there are lots of different ways to treat code as data and lots of different patterns of using that. I have seen cases where people have generated code — for example, in Java, right? — and then created, you know, modified the exact same file to change out the stub functions and actually implement them. And then on updates to the API where you can kind of then procedurally generate the changes to the server function, then you can just kind of get a patch file, run that against your file, and then manually edit it. Right? So. that can work if you have a good mixed code in the same files if you’re going to be manually editing and reviewing it. If you’re going to be automating it, I probably would not have them in the same files.

Jordan Adler 00:15:39 I probably would also, you know, whether or not you check them in depends on whether the generated code is more of an intermediary object or more of a kind of desired output of some kind. And so that will depend, right? And so for example, with the API client libraries the generated code is the product, right? And so, for us having that be checked into the version control actually makes sense, not in the repository that contains all the code that generates it. So we have a code that, one repo where all the code is generated for the client libraries, and then ten other repos for each of the client libraries. One for each of the other client library: Java, Go, C#, Rust, and so on.

Jordan Adler 00:16:19 And so, the reality is that you will need to kind of use whatever approach makes sense. My only cautionary statement here and kind of the good rule of thumb here is when you’re working with a language that’s typed, you want to take advantage of that typing. And if you’re using code generation in a way that basically creates an intermediary layer between the procedurally generated types and the types that you’re actually using in your handwritten code — in other words, if your handwritten code and generated code have two totally different type graphs, and they’re not connected at all, then your type checker’s not really doing its job. And that’s a problem. So you do have to be conscious of that. But other than that, I would say there, there’s no kind of hard and fast rule, and it really depends on the situation.

Felienne 00:17:13 Yeah. I think I can add an example there from a project that I work on myself, because sometimes it’s also about like what tooling do you expect people to have? So we have a backend that’s in Python and most of our open-source developers actually work on the Python side. And then we have a little front end that’s written in TypeScript that we then transpile to JavaScript. So we do check in the generated JavaScript because just because we think that it’s a hassle for the Python developers to have to generate a Javascript themselves, they might not have NPM. It might just not be ready for that type of tooling. So as like a courtesy to people who are like, oh, here’s a generated code. If you’re not changing anything in the front end, you don’t need to compile or transpile the code. So sometimes it’s also about, do you require the users or the contributors in your project to also install all the code generation tooling, which might sometimes be also complex to deal with. So that’s maybe also a consideration that you can have that not only who will, or who needs to generate the code, but also who will sort of feel like installing all the tools that make the code generation happen.

Jordan Adler 00:18:15 That’s a really interesting point. And kind of actually, interestingly enough, is an illustrative of the difference between commercial applications of this technique and open-source or academia where you want volunteers, you want people to join. And so you want to minimize the cost that the threshold effort to contribute code. And that’s not true necessarily in a commercial setting where I’ve been doing most of my practitioner work, right? In a corporate environment where I could say, well you know, tough.

Felienne 00:18:45 Tough, yes, you just have to do what I say. Yes, exactly.

Jordan Adler 00:18:47 Right. Install this thing, or I added it to the device management, so you don’t even realize it, but you already have Java compiler.

Felienne 00:18:56 Yeah, because sometimes this can really be a big blocker. Like, I was looking into another code-generation tool and then it’s like, yeah, I have to install Eclipse and this version of Java. I never use Java. And then there’s sort of need for open-source work. It is a threshold like, well, if it requires me to install Java, then I don’t feel like doing this. Maybe it’s not worth it. So that’s the tooling angle, and it’s very right, that you point this out is very different in Open-Source projects where indeed, we want to make it as easy for you as possible. We don’t want to force Python developers to install tooling that are like, what is this? I’m not going to need that.

Jordan Adler 00:19:33 Yeah, that’s a great point. There’s a lot of tool kits out there, Open-Source tool kits for generating or building code generation tooling. One of them is called YelliCode, which is written in JavaScript or TypeScript rather. And that one is one that we ended up using for a lot of our web SDK. So we procedurally generate glue code that sits on top of our web SDKs, specific to react or view or angular. And so we’re able to produce those kind of — procedurally generate high level SDKs for these frameworks on top of our web SDK. But we didn’t want to do that using the same kind of Java-based tool used for backend stuff, right? And so YelliCode is this really nice kind of TypeScript tool chain that exists for building these things. I have to imagine to some extent it exists in part because of what you were saying, right? Like, a lot of these things existed beforehand, but none of them kind of in the same tool.

Felienne 00:20:28 Consistent, yeah.

Jordan Adler 00:20:29 Consistent, yeah exactly, or compiler.

Felienne 00:20:33 Yeah. We will definitely add a link in the show notes to the YelliCode tool. Then I was also wondering what about documentation? Right? So if I’m generating code, where does my documentation live? Do I generate documentation that’s in the generated code for when people inspect the generated code? Or is that documentation typically placed wherever I’m writing the specifications for the generation, whether that is in a different programming language or in a visual tool? Or is this something that lives in a markdown file where it just says, this is how you generate the code and this is what happens? Are there any best practices there?

Jordan Adler 00:21:10 Yeah. I mean, I think that the best practices when it comes to documentation is, yes? All of them, you know, I think it will depend. So to give you an example, we’ll often procedurally generate, like I said, API client line items, right? And that includes our API reference in it. So we have a Python classes that are stubbed out that include docs strings or documentation kind of inline as Python developers expect them. And that comes from our YAML file, the open APS, open API specification kind of YAML file that says, okay, if you call a put on this path on our server, that is actually this function and here’s what it does. And here are the parameters and so on. And so that, kind of, YAML files consumed procedurally generates and actually creates the client libraries. And so we have kind of one place where we kind of update those API reference documentation and can then propagate that downstream to 10 different client libraries very easily.

Jordan Adler 00:22:10 So that’s one place for documentation and so that’s kind of that inline, you know, documentation in kind of the resulting client libraries. We can also procedurally generate just an API reference itself, right? So kind of a markdown, think of it as, instead of producing a TypeScript output of this kind of API-specific, sort of producing a markdown output. And opening that generator, the Open-Source project includes an output so you can procedurally generate, markdown documentation — or other kinds of documentation actually — to be able to host and serve alongside the client libraries. And that’s kind of another form of documentation. Yet again, we also have the documentation in the open API generator project itself, which explains how to use it, right? So that’s kind of one piece, but in our own kind of repo where we host all the code that actually executes as part of our tool chain open API generator and includes all of our patches to the downstream libraries. That repository also includes instructions for people who are working on our client libraries on how to specifically use it for us. Right? Which includes, by the way, how to patch the readme for the resulting client libraries to have kind of manually crafted readmes that procedurally generate client libraries from the upstream templates are not always super useful and readable. So there’s documentation API references being kind of inserted into the code that’s being resolved in as well as produced as an additional target that we can serve alongside our client libraries, as well as the documentation that exists for the developers using or working on our system and not the ones that are consuming the code by system.

Felienne 00:23:48 Yes. Yeah. So, indeed there are these different forms of documentation. That’s probably a good idea to have it anywhere. And if you so specification about what you’re going to generate you might as well generate that specification as a comment in your code. So let’s go from code generation more towards code transformation. We have already talked about this a little bit, but what exactly is code transformation? Now we have a process in which the input is code and the output is also code, but then there’s also code defining the transformation? So what does code transformation look like for you?

Jordan Adler 00:24:25 So if you think about code generation / code transformation as both things that output code, right? Compilation also outputs code. So, compilation takes in programming code outputs shoot them. Transpilation takes in programming code, outputs programing code, maybe in a different language. Code generation takes in something semantically and outputs code, right? It doesn’t have to be code. It can be some kind of configuration object or something like that. Code transformation, however, takes in code and outputs more or less the exact same code, but having been modified in some way. And so code transformers, sometimes called code modifiers, they can take a variety of different shapes in terms of how they’re implemented, but really what they try to do is produce something that’s basically the same language, but with some modification in the code itself. Either semantically, in the case of, say, a code transformer that’s trying to change the behavior of a function and maybe you have to change everywhere it’s called as a result, right? If you have a very large code base, you might not want to do that manually. You might write a little code transformer to update the function everywhere it’s called to change the parameters that are being passed around. That’s is a kind of one consideration transformative, like how code transformation is different than other techniques in the space.

Felienne 00:25:48 Yeah. So your example made me think of a refactoring, right? So adding a parameter or changing the order of parameters, this is something I can do in the IDE. I right click a function in most IDEs, and then I can reorder the parameters. So that is a refactoring, but also a code transformation. Like, is refactoring an example of a code transformation? Or is it not because it’s not really done with a code generation tool?

Jordan Adler 00:26:14 I think refactoring is a common goal or common cause or use of code transformation. When we talk about find and replace in the IDE, so if you pull up Eclipse or something and do a find and replace, that is a code transformation. Right? You’ve found code; you’re replaced it. Switch statement in Vim, that’s a code transformer, right?

Felienne 00:26:34 So then we’ve identified one tool to do code transformation with the IDE, but I guess there’s also other tools in which we write code to script the transformation or to visually manipulate the transformation? What are tools that you typically use for code transformation?

Jordan Adler 00:26:52 That’s right. So, if you take code and you’re trying to transform it, the tools that you will use will depend on the language itself. So we talked about YelliCode before. Yellicode is kind of a toolkit for parsing, so it’s a toolkit for making code transformers. And so it has elements of it that enable you to parse languages and represent programming code in a given language, say TypeScript, as a data object of some kind. And really like if you think about, what is a code generator? What is a code transformer of some kind? Well, it starts by it’s really a two-step process, right? Step one, get code into data. Step two, you know — I guess three steps if you’re transforming it right? — munge that data somehow. And step three would be kind of producing or outputting that data back as code again. And there’s lots of different ways that you can do that. And lots of different tools you can do that with. You can roll on your own, certainly. Or you can use compiler tool chains that often have that first step covered and the third step which is convert code to data and data back into code.

Felienne 00:27:59 And then what you are manipulating in between is the data representation, which will often be a parse tree, I guess?

Jordan Adler 00:28:07 So, it can be a parse tree. So now we are getting deeper into parsing and for folks who have taken compiler classes, you might remember some of these things. But you can use an abstract syntax tree, which includes enough of the information for you to be able to take a representation of programming code and turn it back into source code. Because remember, not all representations of programming code can be turned back into source code. Once you’ve stripped out white space and comments and so on, you can’t immediately turn it back. And so, a lot of compilers will have multiple steps: it’ll go, abstract syntax tree, and then it’ll trim that down to a concrete syntax tree, and then they’ll change format and use byte code of some kind that actually gets piped into, say, the JVM or python’s virtual machine. But in our case, we’re going to go part of the way. So for Python, as an example, we can actually use Python’s AST module — the thing that Python itself uses to represent Python programs as code. And pipe code, you know, read code from text and put in there, and then once it’s in its AST then we can modify it as we like. But there are other ways too. For example, you don’t have to use a complex compiler tool chain. You can just use regex or even kind of look for strings and manipulate strings; really, any way that you can kind manage text as strings you can use for code too.

Jordan Adler 00:29:33 But the less context-aware that your implementation is, the more risky it is in terms of the error proneness of the output, and the less … because you have to imagine if you’re running this code transformer on multiple different kinds of code bases, not all code bases are created equal. If you test on a million lines of code but a particular pattern is never seen, there’s some kind of bug in your transformer that you just don’t know about and won’t be encountered until someone else picks it up and uses it. And so you have to think about that as you’re designing your transformer, but certainly the simplest possible implementation could be a bash script that is basically a one-liner call to find and replace and set or vim, or something like that.

Felienne 00:30:22 Yeah. And of course it can be easy, but also more error-prone. If you are transforming Python 2 to Python 3 and you just want to add brackets around every print, you could do that with a little bit of string magic, but then maybe you’re not really sure that every print you encountered is actually really the print that you want to transform. So, let’s talk a little more about this case study because you have worked on this Python 2 to Python 3 transformation project, and I would love to hear more about, like, did you do everything automatically, or what are some edge cases that had to be transformed manually? And what was your approach? Can you just take us through that project, how you approached it?

Jordan Adler 00:31:00 Absolutely. And so I talked about this project at PyCon a few years ago, I’d say it was about 2017, you should be able to find that online if you like.

Felienne 00:31:08 Oh, we’ll add a link to the show notes.

Jordan Adler 00:31:14 Awesome. In Pinterest’s Python 2 to Python 3 migration, we used a tool called Python-Future, which was produced by an outfit called Python Charmers out of Australia that I’ve been collaborating with. And Python-Future includes a number of tools that are useful for this endeavor of going from Python 2 to Python 3 in a system. The first thing is a set of code transformers, code modifiers, that take Python 2 code and convert it into Python 2 code, but in a way that is more aligned with, or more gradually, incrementally more consumable by Python 3, right? So there is a set of things that are syntactically different between Python 2 and Python 3. As an example, print moves from a statement to a function, so we have to put parenthesis around it now, right? So, it’s no longer a special-case function call. That can be done with a code transformer, and Python actually included a function called __future__ which in the Python world we call dunder future — “under” for double underscore. So dunder future is a directive you can include into your Python code to say, ‘Okay, I’m going to run this under Python 2, but I want it to behave like Python 3 for this specific type of change.’ And so, what we did at Pinterest was we went through these code modifiers — code transformers — and kind of left our system running on Python 2, but incrementally made it more able to run under Python 3.

Jordan Adler 00:32:50 And it starts with these code modifiers and these, kind of, directives to the Python 2 compiler that says, or Python 2 machine, that says behave more like Python 3 in this way, right? So kind of incrementally, including backwards-breaking changes from a future version. Kind of hard to explain, but you have to imagine for a moment that, essentially, we’re kind of choosing to gradually cause that breaking change to occur. A lot of that was added, by the way, in Python 2.7, which came out after the Python 3. So this was added after the Python 2 migration process really started, which was years before Pinterest creation. So Pinterest was one of the last companies to engage — in part because of the size of the code base — to engage in this process. And so it starts with the code transformers: you manually, incrementally make it more able to run with Python 3. Then we have the Python-Future project includes some what’s called Future. So, instead of underscore underscore future underscore underscore, it’s future. So, from Future, import so on. And you can import monkey patch functions. So for example, you can import a version of the string object creating function that creates string objects that are more like Python 3 than Python 2. Once you produce Python 2 code that behaves more like Python 3 and is running on a Python 2, then you can start bringing in these future functions or future classes that are basically runtime shims that model the behavior of Python 3 under Python 2. So you can start coding against Python 3 API in your Python 2 code base, by pulling in new stuff into Python 2 from Python 3.

Felienne 00:34:48 Yeah, so you can migrate while you are also adding new features to this existing code base. That’s what you’re saying, right?

Jordan Adler 00:34:55 That’s right. Yeah. You can migrate while using features that would typically not be available in Python 2. Or specifically, the API that changes under Python 3, you can pull in more and more of those changes either through directives to the Python virtual machine or through these, effectively, userspace implementations of core Python objects that are consistent between
Python 2 and Python 3. This is in contrast, by the way, to another approach that you can use is to do the Python 2-to-Python 3 migration, which is basically if statements. You can say, “if Python 2 do this, if Python 3 do that,” right? And that pushes the complexity into, or makes the complexity in our code base as opposed to, kind of, this module we’re using in the library and stuff.

Felienne 00:35:44 Yeah, because if you have the complexity in the code transformation tool, at one point hopefully you are done. So then you no longer need that complexity, and then you end up with a cleaner code base that is 100% Python 3.

Jordan Adler 00:35:56 That’s right. So when at the end of this project, the final stage, when you’re actually taking this code that could run on the Python 2 or Python 3 by virtue of these directives to the virtual machine as well as this kind of userspace versions of Python 3 classes and functions, you can take that code, run it on Python 2, run it side by side under Python3, confirm that they behave the same and then actually stop running under Python 2 and then remove all those directives that are — you know, the cleanup patch is a lot smaller, right? It’s just, remove a few lines from the top of each file to remove those directives.

Felienne 00:36:34 Yeah. So let’s talk about tools for this project. So what did you use to write transformations in or to define the transformations with? Was that this YelliCode tool that you were talking about — because that was a JavaScript tool — did you use that here, or did you use something else?

Jordan Adler 00:36:48 So YelliCode, it is Typescript-based, it is JavaScript-based. So it is not what we used here; also, I think it came a little bit later. So Python-Future uses the AST class that exists in the Python standard library. So this is actually the thing that Python itself uses to parse Python. We use in Python-Future as well. We basically take in code, we read it in, use the AST module so it’s kind of reading code, turn it into an AST object, which is the abstract syntax tree. And then we transform it. We look for specific — so we do a typical tree walk, we look for, for example, maybe look for a node that is a function call type. And once you find a node that is a function call type, you want to find out what function it’s calling, and you can pass and say Print, right? So you can write a little piece of code that says, ‘Hey, once you’ve got the abstract syntax tree, look for the node that has a function called Print’ and then once we’re in there we can change the AST in some way. But if we never find it, then we don’t do anything.

Felienne 00:37:49 So this is tooling then that sort of depends on a certain programming language. Does this exist for any programming language? Can you transform Java with a similar approach, or is this a very Python thing to have build in?

Jordan Adler 00:38:04 This is definitely very Pythonic. Most compiled languages don’t have some version of this. Most — or maybe most is kind of, I’m not sure if it’s most, but many interpretive languages do. So Python, Pearl probably have some version of an abstract syntax tree class or some way to model Python code or Pearl code or PHP code, for example, in that language itself. But most of the time you won’t see that. And in fact, compilers you may have to reach for a compiler tool chain to dig into there. So, for example, LLVM is a kind of compiler tool chain project that’s out there and has what are called compiler front ends, which basically take in source code as text and produce what’s called an intermediate representation, which was code as data in some way. You can use LLVM front ends often — in fact, all code transformers all use LLVM because LLVM has amazing coverage on the front end side. And so, basically, your front end is: take let’s say C# code, turn it LLVM intermediate representation. And then your back end is just: turn back into C# code. So you can just write your own little fake compiler that calls the LLVM, ‘Hey, turn this C# code into intermediate representation then modify the intermediate representation and turn it back into C# code.’

Felienne 00:39:35 So, what is a scenario that you would want to do that where you use this? Is this purely about using, like, compiled languages, or are there other differences between this and the Python tool?

Jordan Adler 00:39:48 In this specific case of, let’s say, an LLVM, IR, and AST, I don’t know what they may have in difference. Now, as I mentioned earlier, there are representations of code as data that are not easily converted back into source code because they don’t have those white space or comments or other parts that frankly aren’t meaningful to the machine, right? If you’re actually turning it from source code to machine code, if your tool that you’re using to build your code transformer is really intended for code compilers, then you may not be in a good situation. But you can find versions of this for almost every language that’s out there. And it’ll be very kind of tech stack specific, and so you’ll have to do your own research, but those are some of the ones that I’ve used.

Felienne 00:40:38 So, of course, we want to also know about the pitfalls, right? What are some of the things that you ran into when doing this big migration? What are some of the mistakes that we should not make?

Jordan Adler 00:40:51 I mean, I think probably, there are lots of pitfalls. I think probably the most immediate one that comes to mind is not all use cases are going to be the same. So you have to remember that. When you’re reading documentation about code transformation of some kind, you will find instructions or guidance that is generally true but may not be true for your specific case. Keep in mind, when I was working with Pinterest and we were transforming a multimillion line code base, we found everything, right? We really battled hardened the hell out of that Python-Future project. And you know, I think that you have to be conscious of that whenever you’re working with code transformer code out there is, whatever you’re picking up, chances are it hasn’t been applied on code bases as unique or as varied as, kind of, the totality of all code in existence and therefore how it applies to your specific code may not be how it is intended to apply, and there are probably bugs in there too. So I guess, as there are bugs with any kind of software, bugs that exist in code transformation software can be very difficult to detect if you’re not kind of being intentional about it and can be extremely difficult to debug. Because it’s basically like, code’s removed, code’s changed. It’s just really hard.

Felienne 00:42:13 So talking about transforming multimillion lines of code projects, what about performance? Like, such a transformation, did it take like an hour? A day?

Jordan Adler 00:42:25 Well, in the case of Pinterest, our migration took months — probably on the order of years, frankly. But you have to think about the project that you’re embarking on, what you’re trying to achieve, and kind of what your desired outcome is before you reach towards a tool. And if you find yourself in a situation where code transforming gets you more confidence, as it did for us in Pinterest, then great! So, a multi-year project was cut down into something that was fewer years, right? But the running of those tools, those manual code transformers, was just one part of that project. And so, you have to think about how your project shape is going to be different if you use this technique. If you are trying to make a change, and you’re pulling in code transforming as part of that change in an automated way — so if you’re incorporating code transformation as part of your tool chain, for example — that will, as I mentioned earlier with code generators increase your build time, and so that can become problematic as well..

Jordan Adler 00:43:32 So yes, they can take time to run. There is a performance cost here, and depending on how you apply the technique or, kind of, what you’re trying to achieve, the trade-offs may not be there. And they may end up being yes, it takes longer to actually run the command and I’m spending more time waiting, but I’m spending less time typing the same things over and over and over again. And so that is the trade-off that you have to think about. And sometimes that takes a view of the timelin, a temporal window, that is bigger than just the build step or just the actual part of running the code itself, the code transform.

Felienne 00:44:13 Yeah. So I guess what you’re saying is that running the transformation itself in such a big project is not really where the performance issues exist because in such a big project, it’s just maybe if it takes an extra hour, it doesn’t matter if this is a project of a few months.

Jordan Adler 00:44:28 Right. And also like we chunked it up. So, we ran 10 pieces of 10 files at a time, for example, out of a thousand files. And so each run on each file may have taken a little bit of time, sure. But that process of chunking it up and doing it in that way and having some automation there, netted out with something that was much faster than if we had manually done it, right?

Felienne 00:44:53 So you already mentioned something about making sure that the code was the same because you could deploy it to a subset of users and see if not too many errors occur, but that is like the code as the running artifact. But I was also curious about sort of the code as an artifact for reading. Did you also make any improvements while transforming to maybe some stylistic issues? Did you also try to improve the code base, improve the readability of the code base, or at least not make the code readability worse? Because the interesting difference between transforming code and generating code is maybe with code generation, you don’t necessarily need to then maintain the generated code, but with this, these sort of transformation projects, then once you’re done, people will then manually continue to work with the code that you’ve transformed. How do you make sure that this transform code is reasonable for a person?

Jordan Adler 00:45:48 Yeah. I talked a bit earlier about abstracts syntax trees and concrete syntax trees and how one major difference is that they include space and comments — the parts of the source code that are not relevant perhaps to the machine itself that’s running code, but rather to the programmer who’s reading it. And so if you have a code transformer that eliminates those things, that removes them right, then the output code that you have is going to have those things stripped out, and that’s going to be less useful to the developer. So certainly that is something that you have to be conscious about when you’re running a code transformer is you don’t want to eliminate or change too much of the white space or comments, certainly, if you don’t have to. There also exists a set of tools out there called autoformatters or prettiers, or something like that. Sometimes called tidy pools. Think of it a kind of like a linter.

Jordan Adler 00:46:39 So if a linter does static analysis, which is basically turn the source code into data and inspect it somehow and return a result: this is a bad call, or this is a broken pattern, or this looks good or whatever. So that’s a common linting case. A prettier will take a code, actually add white space as needed, or comments where appropriate, break up lines, do whatever, change semicolons where optional — all the stuff that are stylistic changes that historically people would spend lots of time arguing in comments on pull requests overnight. You know, “no semicolon here.” “But it’s optional.” “I don’t care.” Now we have basically a tool that you can run before you check in code. That kind of auto-pretties your code. So there’s prettier in JavaScript land. Lack is a tool like this for Python. I think you’re going to see something like this in lots of different languages where there’s sort of like, okay the Open-Source community said, here’s the style that we want more or less standardize around because every little shop having their own opinion, and having a config file on every repo for script specific to my code base doesn’t actually improve readability, right?

Jordan Adler 00:47:54 What really makes a difference to readability is that everyone expects code to look a certain way. People can quickly look and say, okay I see this pattern call visually. And so the cognitive process of looking at a piece of text and recognizing calls in a certain way is a lot better when there are markers present or spacing is as expected. And so it’s really important certainly for productivity not to eliminate that stuff, and I think if you have a code modifier that you produce and it removes white space and comments, it is broken — unless that’s a desired goal, right? In which case, you probably shouldn’t be shipping that little thing anyways because it’s probably a part of a bigger thing like a compiler.

Felienne 00:48:39 So, I guess what you’re saying is that you want to keep comments in place. You want to keep white space in place. And in some situations you might want to, if you are transforming anyway, also run the codes through a prettifier tool so that the output looks the same in similar cases, making it easier to read for future developers.

Jordan Adler 00:49:01 Yeah, and if you’re doing a large transformation project, you’ll probably want to do that prettier run before, right? Because a prettier, an autoformatter, it’s supposed to be a semantic noop, right? It’s supposed to have no change to the semantics of code. It just looks different. And so doing that first, and then running that big patch out the door, semantic noop, you can make a change easily … then you create some sort of tool chain, CICD kind of process that auto-pretties code before it gets pushed up, then that will kind of minimize the thrash to developers in your code base.

Felienne 00:49:39 Nice. That’s really good advice. Just peeking at my notes. So this was actually everything I wanted to talk about. Is there anything we missed? Any important tips or best practices, or more stories that you have to share about code generation or transformation?

Jordan Adler 00:49:55 I think that I talked a bit about kind of the different techniques for actually getting code from text into data. We talked about regex, we talked about text markers, AST, and for folks who are interested in learning more, that is a great place to start. Start by playing with code. You know, take some script that you’ve written. See if you can turn it into some sort of data object in one way or another, and try and manipulate that. And you can use tools that are out there for your benefit. But if you’re really trying to learn and grow what you know, I think it’s great to build something yourself, even if the tooling is out there already. I would definitely encourage people: get curious, check it out. It doesn’t take much to try and practice this technique, and once you’ve kind of learned it, you’ll find yourself with a new tool, a new power that you can use — really a superpower that you can leverage to make not just yourself more productive, but all the people you work too, and that’s a win-win.

Felienne 00:50:57 I think that’s a great closer of the episode. Knowing how to parse and transform code, it is like a superpower.

Jordan Adler 00:51:04 Oh yeah, definitely.

Felienne 00:51:06 So any places where we can read more about you — like, your blog, your Twitter, any links we should add to the show notes?

Jordan Adler 00:51:13 Absolutely. I have a website: jmadler.dev and you can also find me on Twitter @jordanmadler. And to learn more about the Python-Future project, which you can add to the show notes as well, is Python-future.org.

Felienne 00:51:36 Yeah, We’ll make sure they’re on the show notes. Okay, thanks for being on the show today.

Jordan Adler 00:51:41 Thank you so much.

[End of Audio]

SE Radio theme: “Broken Reality” by Kevin MacLeod (incompetech.com — Licensed under Creative Commons: By Attribution 3.0)