Christian Mesh, tech lead of the OpenTofu project, speaks with host Robert Blumen about OpenTofu. They start with the history of terraform, terraform providers, license changes to open source projects, the origin of OpenTofu as a fork of terraform, and the structure of the OpenTofu organization. They further explore compatibility issues for HCL, providers, and modules, performance issues, and adoption, as well as significant features in the OpenTofu-included dynamic-provider iteration, and the roadmap for the project going forward.
Brought to you by IEEE Computer Society and IEEE Software magazine.
Show Notes
Related Episodes
- SE Radio 289: James Turnbull on Declarative Programming with Terraform
- SE Radio 405: Yevgeniy Brikman on IaC Best Practices
References
- The OpenTofu Manifesto
- Static Evaluation of Provider Configuration PR
- OpenTofu 1.7.0 is out with State Encryption, Dynamic Provider-Defined Functions, and more
- OpenTofu 1.8.0 is out with Early Evaluation, Provider Mocking, and a Coder-Friendly Future
- opentofu/rfc/20240513-static-evaluation-providers.md at main
- Top-Ranking Issues #1496
- Dynamic Provider Configuration Assignment · Issue #300
- Use Cases | OpenTofu
- Can OpenTofu Become the HTTP of Infrastructure as Code? – The New Stack
- OpenTofu vs. HashiCorp Takes Center Stage at Open Source Summit – The New Stack
- Github – OpenTofu Providers
- cam72cam (Christian Mesh) on GitHub
- Schedule for OpenTofu Day
- CNCF KubeCon 2024 Keynote
Transcript
Transcript brought to you by IEEE Software magazine and IEEE Computer Society. This transcript was automatically generated. To suggest improvements in the text, please contact [email protected] and include the episode number.
Robert Blumen 00:00:19 For Software Engineering Radio, this is Robert Blumen. I have with me Christian Mesh. He has worked for 14 years as a software engineer, including Space Telescope Science Institute. He’s the Tech Lead on the OpenTofu project, which will be the subject of our conversation today. Christian, welcome to Software Engineering Radio.
Christian Mesh 00:00:41 Thanks for having me. Apologies. My voice is a little bit raspy. I’m just getting back from KubeCon where I gave a couple talks on OpenTofu and had a great time meeting a lot of the users and developers there. It was excellent.
Robert Blumen 00:00:54 Yeah well, a lot of talking goes on at those events, that’s for sure. Was there anything else about your background you’d like listeners to know?
Christian Mesh 00:01:02 Sure. I’ve worked at a variety of software jobs at a variety of levels throughout the industry. Although I will say my time at OpenTofu has so far been remarkably unique. It’s an opportunity for me to work on a project that I’ve cared about for quite some time, and this is going to sound cheesy, but fight for the user. My main goal day in and day out is making a project that works for the people who are trying to use it. I and the other developers are all service-focused, focused on figuring out what we can do to make other people’s lives easier, and being in that role day in and day out is fantastic.
Robert Blumen 00:01:35 Thank you. OpenTofu is a fork of the Terraform Project. We have covered Terraform previously in Episode 289 and 405 on Terraform Best Practices. Before we go into OpenTofu, can you explain what Terraform is?
Christian Mesh 00:01:51 Sure. Terraform is a tool that is used to manage infrastructure’s code. You are able to define a configuration, which then talks to whichever Cloud provider or even the pizza place down the road using the provider framework to deploy and manage your infrastructure. It’s based on a relatively simple declarative configuration language pioneered by HashiCorp, and it really took the ISA world by storm a decade ago and has been adopted throughout the industry, both through open source and closed source projects. Its success resulted in a number of companies being built around it and built on top of it, and that will kind of tail into some of the other things we talk about today. But as an end user, for someone who’s not familiar, you write a config file saying, I want this infrastructure between these Clouds or these particular areas, apply it, and it then goes and does it for you. So instead of having to go through the AWS console or the GCP console and do all that by hand, it is a way to manage that infrastructure as a team in code, in a pretty straightforward fashion.
Robert Blumen 00:02:53 In your response providers came up. Can you explain the boundary between Terraform and providers and where do the providers come from?
Christian Mesh 00:03:03 Sure. providers are based on a specification put up by HashiCorp. They have a GRPC protocol and now that’s a technical detail. It doesn’t matter too much, but it is a standardized protocol that many people have adopted. So at a higher level, a provider is something that allows you to talk to some other service. So OpenTofu and Terraform don’t understand what AWS or GCP is out of the box. There’s a little caveat there, but for the most part, they don’t understand how to manage your infrastructure. They understand their configuration language and they understand how to download providers. Providers then say, hey, I can manage, for example AWS, ALB, I can manage an S3 bucket. I can manage resources on Google Cloud. So it is the bridge between your configuration and OpenTofu to their infrastructure.
Robert Blumen 00:03:50 Terraform is very well known for provisioning Cloud service providers. The two that you mentioned just now, AWS and GCP, but it’s not limited to Cloud services, it can really provide almost anything. Correct?
Christian Mesh 00:04:05 That’s correct. One of the folks I work with on a regular basis uses it to build thousands of docker containers. It’s not a standard use case, but it’s fascinating to see all the different ways people can use these tools. So a couple more notes on providers. They are written by both HashiCorp originally. They’re originally all built into Terraform back in the day, but they decided to break it out to make it easier to author providers. So if you’re a smaller Cloud or you have infrastructure on prem or some other scenario, you write some Go code that tells OpenTofu and terraform exactly how to manage that infrastructure. It’s an open protocol. There’s, as last count, about 4,000 in the OpenTofu registry. So it’s, there’s a few of the large ones that HashiCorp maintains themselves with help from the community and from the corresponding Cloud provider. But it’s a gamut spinning pretty much every corner of software engineering and infrastructure.
Robert Blumen 00:04:54 OpenTofu started out as a fork of the Terraform project, which is a, and may still be open-source. I’m not completely clear about that. Can you explain why OpenTofu founders felt the need to fork off of Terraform?
Christian Mesh 00:05:11 Sure. So I’m going to step back in time a little bit to go back to something I said earlier about how this was started by HashiCorp. So they started this project, they created Terraform, they created the providers back in the day, and they branded themselves as open-source community focused. And a lot of people were comfortable adopting the thing, because of that focus. That’s one of the reasons they were able to take the world by storm. Unfortunately, what happened is about a little bit over a year ago, HashiCorp decided to change the license, Terraform, and they can do this because of something called a contributor license agreement. So when you change a license on a software project, you need to get sign off from all of the contributors that have contributed code. In most cases, that’s what the license requires. So when I as a developer contribute code to Project X, if I have not signed off on a license change, they cannot relicense the whole project.
Christian Mesh 00:06:03 They can try and remove my contributions and then work around that, but it’s the idea that the license is in most cases, fairly immutable. Unfortunately, the contributor license agreement means that as soon as you’re contributing code to HashiCorp’s, Terraform or a lot of other projects, we’ve seen MongoDB, Elasticsearch, there’s a large number in the past five years that have effectively pulled the rug out from under their communities. So in the Terraform case in particular, they switched to something called their business license. Now that is, again, I’m going to preface a lot of things I say today with, I’m not a lawyer. I’m not speaking on behalf of any lawyers or anyone who knows more about this, but my understanding and based on their FAQ is, it’s not truly open source anymore. It is visible source, which means they have additional constraints on who can use the software, who can contribute to the software, which is a problem.
Christian Mesh 00:06:52 So the CLA means that they can change the license to whatever they want, functionally speaking, they could move it close source if they wanted to. They did a step in between, which is a really tricky move to get around and to work with. So as OpenTofu, just as some concrete example, we can’t use LLMs in any part of our process because the code is visible, but it is unclear if an LLM learns off of Terraforms code on GitHub. What if someone contributes code to OpenTofu? If we’re using an LLM, does that constitute a license violation? We don’t know, and it’s an illegal gray area. As far as the enforceability of Terraforms license, I can’t really speak to that, but this is a pattern we’ve seen more and more in software companies starting an open source project, getting a lot of recognition and following because of it, because the community is helping build it. They see ownership of it, they see that it is a project they can contribute to, that they can build on top of, when in reality, if it’s one company with a contributor license agreement, that might not actually be the case. And unfortunately, we’ve been seeing that a lot.
Robert Blumen 00:07:51 Are the companies that founded OpenTofu, is there a specific worst-case scenario that they’re imagining might happen? Or is it more simply a lack of clarity about what could happen and that in itself is a bad thing?
Christian Mesh 00:08:08 So there’s a mixture of a lack of clarity and explicit terms within the license and their FAQ that mean that a lot of the companies that sponsor OpenTofu, not all, but many of the companies that sponsor OpenTofu, I’m going to say, are fairly sure that they cannot run modern versions of Terraform in their infrastructure. Again, this is a very complex legal area, but to the interpretation of the companies that have found it open to who, many of them are uncomfortable running Terraform in their infrastructure, and there’s a pretty strong sentiment that this was one of the main reasons for the license change. So taking it as a step back, Terraform has a corresponding product at HashiCorp that is in the form of their Cloud offering. That is, it is something that HashiCorp is trying to push as one of their big products. They try and get people to use it to store and manage and run their infrastructure.
Christian Mesh 00:08:54 So instead of running Terraform on your developer laptop, you’ll run it as part of Terraforms Cloud and or Corp’s Cloud. And there are some distinct advantages to that. What ended up happening is with any product, there is always room for innovation. So a number of these companies before the license change spun up around Terraform. Now sometimes it was adding on functionality that didn’t necessarily overlap with what HashiCorp was offering. In other cases, it was indirect competition. They’re in a lot of cases building similar features in these hosted platforms. So by changing the license, HashiCorp made what I would consider an anti-competitive move that was trying to lock other people who were using Terraform out of the market and all of their customers therein. So OpenTofu was a direct response to that. A lot of these companies are direct competitors, so you have a zero space lift a harness, and there’s others that are in this space.
Christian Mesh 00:09:44 And apologies, it’s been a busy week coming back from KubeCon, but a lot of these companies saw the writing on the wall. They say, this is no longer a platform we can build upon. This is something that is risk to us and to our customers. It’s unclear what the license implies for both parties. And again, I can’t speak to that directly. So the day that this happened, the founders of these companies that had built on top of Terraform got together and put together the manifesto. They broke down barriers. Again, these companies, they’re trying to build the best products so that they can win their other competitors’ customers. It’s a competitive environment. This is what the foundation of OpenTofu is. And all those barriers were set aside in a day after the license change happened, lines of communication were opened, and a manifesto was created.
Christian Mesh 00:10:26 It’s currently on the front page of the OpenTofu website because a fork is the last option. Ideally, you want to work with the community, you want to figure out a way forward. Unfortunately, that manifesto fell on deaf ears and we were forced to fork the code base after Terraform 155, which was the last open source version under the Mozilla public license, and spun up a team that I know lead that builds new features on top of OpenTofu. There are some additional complexities to that, but OpenTofu is currently under the Linux Foundation, the trademark, the organization, and it’s also, if someone disagrees with even how the Linux Foundation is running it, OpenTofu is easy to fork. This is by design. This is kind of a demilitarized zone between these companies and it’s allows it to be sponsored very heavily by these companies. But community driven and I can talk more about that later if you’d like.
Robert Blumen 00:11:15 Let’s talk about the OpenTofu.org. You talked a little bit about what motivated it to start. What are some of the founding companies okay, if you don’t remember all of them.
Christian Mesh 00:11:27 Sure. We do have a sponsorship page on our website that lists all of the companies involved. Off the top of my head because these are the ones that are actively contributing, like have signed up to Contribute Developers. Whereas it’s not, which doesn’t cover all of the contributions to OpenTofu, but the developers working on it Space Lift, who sponsors my development harness, M Zero Grunt Work and Scaler, those are all companies that are sponsoring developers directly on it.
Robert Blumen 00:11:54 Any key points from the manifesto that you have not already covered?
Christian Mesh 00:11:59 I don’t believe so. Other than it was the olive branch to say, listen, we think you’re making a mistake by changing this license. We’ve all invested, we’ve evangelized this product. We have built on top of this product. We’ve contributed back to this product. HashiCorp had this narrative that, oh yeah, we’re all these other companies are freeloading on top of Terraform. But in reality, it’s become increasingly hard to contribute back to Terraform in any fashion. And that’s one of the reasons that as soon as OpenTofu repo was created, we got a deluge of PRs. We couldn’t accept all of them right away, but that’s why there was so much community interest right off the bat, because it’s open and we’re more willing to have, I mean, this is my personal opinion, that we’re more willing to have conversations about what the future of the tool is and don’t have a strong idea of what OpenTofu should be in the future other than community driven.
Robert Blumen 00:12:45 We will later in this interview talk about of course, the features in OpenTofu. I am also going to ask you if you recall any of those PRs in that original deluge. Let’s put that on hold for a moment and stick with OpenTofu. It has some kind of corporate identity as an org. What is the nature of the org? Does it have employees, things like that?
Christian Mesh 00:13:07 That’s a good question. So currently there is, I’d call it maybe a coalition, I don’t have the proper word for this, but there’s an organization of companies that came together that realized that they needed a project that is truly open source that they can build upon and that’s what OpenTofu is. As far as the day-to-day organization, there’s a lot of people involved in the peripherally, but primarily there’re the engineers that are the core team, which I lead, which are from all of those companies I mentioned before that handle reviewing PRs, triaging issues, figuring out what the next couple steps of the roadmap are. But doing support in the Slack channel on GitHub above us is the Technical Steering Committee. It’s currently comprised of a member from all the companies that are currently contributing engineering resources to OpenTofu. They meet once a week, they take a look at the roadmap, any technical issues or legal issues that come up and decide either to try and figure it out themselves and give the core team some guidance or to escalate to the Linux Foundation.
Christian Mesh 00:14:01 At the end of the day, the buck stops with the Linux Foundation. They have a lot on their plate and lots of projects. So we try not to be too much of a thorn on their side. But anytime a major change to our governance, any projects happens, we run it by them first because the goal is that no one company, no one individual has control of OpenTofu. That is one of the core tenets of what we believe and the level of freedom I have as Tech Lead. I’m a Space Lift employee and this is one of the jokes I told at KubeCon, which is frankly true is I don’t know what Space Lift is. I’ve read their marketing page, I know roughly what their product entails, but I can’t speak to how it compares to other tools in the space. They don’t come to me saying,hey Christian, we need this feature in OpenTofu to push our agenda for the next release of Space Lift.
Christian Mesh 00:14:44 Or I didn’t even know what the release cadence looks like. It is remarkably hands off. I was actually quite skeptical when I first joined the team that it was going to be okay, you get 20% to do what the community wants and 80% to do what Space Lift wants. But no, it shows that I don’t know what Space Lift does. All of their customers come through the standard issue pipeline get triaged just like any other person wanting something out of OpenTofu. It is truly a space that still surprises me that all these companies devote all these resources and are completely hands off to the future of OpenTofu. It is what is the community asking for? How do we win hearts and minds? That is our goal and it’s been remarkable to be part of a team that’s focused purely on that.
Robert Blumen 00:15:22 Version. 1.5.5 of Terraform was the four point Terraforms. Now one nine something OpenTofus had a number of versions. Would it be a goal to stay compatible or are these now two products that naturally will evolve in different directions because different people have the best idea of what they think it should be?
Christian Mesh 00:15:44 That’s an excellent question. It’s an ongoing concern and a question we get frequently. Currently, we’re aiming to be as close to a drop in replacement as possible as people migrate over. We recently had Fidelity, they spoke at OpenTofu Day at KubeCon. They migrated 900 environments over to OpenTofu really without a single problem. A couple people had to update their aliases in their Shell scripts, but other than that, it was a drop-in replacement. As far as the future’s concerned, this is a conversation we have every single day, both amongst ourselves and the core team and with the community. So actively there’s a discussion happening on Slacks. So grunt work, which builds on top of Terraform and OpenTofu has a concept called Stacks Terraform in one point 10 is introducing something called Stacks and they have some patents around it, which makes things complicated and other companies have similar features.
Christian Mesh 00:16:30 It’s unclear if OpenTofu is going to implement something similar itself or add functionality that allows tools to be built on top of it that can provide this functionality. There’s always this question of what role does OpenTofu fill in the ecosystem? Is it closer to assembly? Is it closer to like a WYSIWYG editor? Like there’s the whole gamut within the ecosystem and it’s an interesting conversation to figure out where OpenTofu is today and where it should be going. But again, this is always a conversation with the community. As far as me personally, I’m hoping that we can keep compatibility as long as is reasonably feasible, but as these two projects diverge, I don’t know what that’s going to look like. I know OpenTofu has some features the Terraform doesn’t have, and I know the reverse is true and we’ve had a lot of community questions, but one of our community members has created something called Can I.TF? And that is slowly being built out to sort of document the differences between the tools, especially as they change over time.
Robert Blumen 00:17:25 This raises another point in my mind, which I understand is not the focus of this interview. We did an episode on this podcast about automated rewrites AI and other kinds of tooling when people have to migrate enormous code bases from let’s say Python X to Python Y. And this area is one that’s actively under development in the industry for many reasons. That’s not really a question, but you can respond if you’d like.
Christian Mesh 00:17:55 Sure. As far as the migration for OpenTofu currently, the only thing that really happens is we have to replace the provider address within the state file. As far as tooling built on top of OpenTofu to help with migration, I think that’s something we’ll consider in the future as the feature sets diverge. Again with the compatibility promise, oh, this is actually something I should have mentioned earlier is the compatibility promise is something that both OpenTofu and Terraform have, which Terraform created in when we forked, we adopted that says that we’re going to maintain compatibility with 1.0. We’re not going to be breaking backwards compatibility amongst ourselves, which means there’s going to be that core feature set that 95% of people use day in and day out. That is unlikely to change within the next couple of years. As far as new features go, that gets a little bit trickier of a conversation, but we’ll handle that as it comes.
Robert Blumen 00:18:41 We’ve been talking just now about compatibility of Terraform. Earlier we talked about providers which are written by different people and they interact through an RPC interface. Are providers going to be compatible with both branches of the fork or will providers now also need to potentially fork and potentially support two different cores?
Christian Mesh 00:19:04 So at this point, our plan is to maintain compatibility with the existing protocol. HashiCorp has been adding things to it, but I would be remarkably surprised if they ever make a breaking change to that protocol. They have stuff in there that goes back honestly probably close to half a decade if memory serves, would they still support odd and old patterns that were built back in the day. So you’ve got 4,000 providers written by probably 3000 plus companies and individuals. So making any breaking changes to that protocol is, I would say ill-advised. Terraform has been adding additional things to that protocol, which we’ve been adopting as needed because that is still published under the MPL version two. But as far as where we go in the future, OpenTofu may start adding additional features that aren’t in Terraform and figuring out how to give providers an option to implement that without completely destroying the workflow or becoming incompatible with Terraform is a big problem we have to solve.
Christian Mesh 00:19:58 We have some really brilliant engineers that are currently noodling on it, but GRPC is a pretty flexible entity and there’s options that we have moving forward, but our main focus is on, honestly, first and foremost is not breaking compatibility with that protocol. That is, as far as I’m concerned, set in stone. But you can with things being added on top of it. But the providers themselves, I hope we maintain a scenario in where they don’t have to choose OpenTofu for Terraform and they can be built for both. The tooling may get more complex.
Robert Blumen 00:20:25 But something like where if your provider and there is a difference between the two, you might be able to handle that with a branch in your code, say if we’re in this environment, do this. Otherwise it’s not supported.
Christian Mesh 00:20:39 Yeah and to add onto that, one question I get very frequently is when I’m switching from OpenTofu, do I need to do anything with proprietors? Do I need to change them? I hear that there’s this new registry, so what does that look like? My response is HashiCorp decided that when they changed the license on Terraform, they also,to make it in my opinion, more difficult to fork. They changed the terms of service on the registry. The registry is what allows Terraform and OpenTofu to connect to GitHub, to download the providers. That metadata was now locked within HashiCorp’s vault. So when we forked OpenTofu, we had to build our own registry and we scraped GitHub very thoroughly to find any potential providers and added it to our registry. So we now are effectively maintaining two sets of metadata. One which is the Terraform registry that HashiCorp maintains and the OpenTofu registry, which OpenTofu maintains and the community can submit providers and modules to.
Christian Mesh 00:21:31 But the interesting thing is, side effect is that Pulumi, one of the projects in the same sphere is no longer able to use the Terraform registry and actually had to switch over to use OpenTofu registry. And we’re happy to have them and we’re actively in talks with trying to break some portions of OpenTofu out into libraries so that we can deduplicate some effort there. But when you do migrate it, it will be using a separate registry under the hood, which again was just one of those fallouts from some of the HashiCorp’s actions they took around the time of the fork.
Robert Blumen 00:21:57 If I understand this, the provider registries are kind of a proxy or a layer in front of the open-source project on GitHub. So a provider project could use GitHub, publish their work as everyone does, and then it could show up as a provider in one or more registries, Terraforms, OpenTofu, or anyone else without the project having to have two versions or be in two places. Is that correct?
Christian Mesh 00:22:27 That’s correct. Exactly. So what they’ll likely do, if I was authoring a provider today, I would submit it to both registries, Terraform and HashiCorp, they have their own license, you agree to. You sign into your GitHub account and they actually get a web hook set up within your project that you fire off anytime you do a release. OpenTofu wasn’t really in the position to ask every single of those 4,000 authors to go through and set up a web hook for us. So we took a little bit of a different approach. So our registry, we automatically scrape pretty much the entirety of GitHub’s projects that we know about. We look at the RSS feeds for releases and we automatically update every single provider in every single module. And there’s, I think 25,000 modules of memory serves every 15 minutes. So as a module author, let’s say you just wrote a provider for OpenTofu, you publish a release on your GitHub page, you go to OpenTofu, you create an issue using the template, paste in your provider’s name, hit submit.
Christian Mesh 00:23:26 One of the core team does a quick review, takes a look, make sure everything looks good, merges that in which is just a small little metadata file. And our registry is, if you’re familiar with the Home Brew Registry, we actually worked with some of their developers when designing ours. It’s based on existing patterns that work well in open source. So our registry on GitHub is just a giant set of metadata files and then every 15 minutes, if you publish a new release, we pick up on that via the GitHub’s RSS feed and automatically publish the new version in OpenTofu. No additional configuration needed from the side of a provider, which we’re happy about.
Robert Blumen 00:23:58 We’ve talked a lot about the history, infrastructure, organizational structure. Now I want to get into what the feature differences are. I have used a lot of Terraform and probably like every developer and every software, I had the thought, well I really wish it did this or that thing. And you can only add so many features to anything and some features might not work very well with the existing set of decisions. So I have a list of some of the features that are highlighted on the OpenTofu website that are in 1718. We could go through those or maybe if you would like to highlight what you think are some of the top new features in OpenTofu, we could start from there and then I can ask you about others if we have time.
Christian Mesh 00:24:45 Sure, yeah. I can give a quick overview of some of the bigger pieces of work we’ve done so far. There’s lots of small little quality of life pieces that I don’t need to get into today, but our things that have been asked for quite some time, the biggest thing for 1.7 was State Encryption. We actually had a company, I believe it was a German bank, send one of their employees to join the OpenTofu team for a few months to build this feature. Because they were maintaining a Terraform for internally in order to meet compliance to the best of my knowledge.
Robert Blumen 00:25:11 Before you go into this, talk about what is the state and why would someone want it to be encrypted?
Christian Mesh 00:25:17 Sure. So the state is a JSON file. It’s really, it’s a set of data that OpenTofu uses as a, this is the state I think the infrastructure should be in. So OpenTofu deals with three realities. The reality of the configuration that the user has written, the reality of the state, which is what it thinks it applied and it’s what it thinks infrastructure should currently look like. And then there’s the actual whatever’s out there in the world that someone may have been twiddling knobs on. So OpenTofu when running an apply action will refresh and take a look at what is currently out there. It’ll load the state for what it thinks should be out there and it looks for any changes from the configuration since the last time it applied changes. So this data may include IP addresses, ports, configurations, secrets. There’s a lot of stuff in there that you probably don’t want other people getting their hands on.
Christian Mesh 00:26:06 So we took the approach at least initially of saying that, well you might have a bucket where this is encrypted, like for example, S3 can configure encryption there, but perhaps you want to store your encryption keys in a different Cloud provider of all things just for added security or maybe the solution of having it not encrypted in transport is a problem. So our solution was client-side state encryption. So you set up your keys in the OpenTofu configuration, either passed in via environment variables directly in the config, which we don’t recommend or via the AWS or GCP key management system. And we have provisions for rotating keys and all of that migrating to and from encrypted states. So we actually had someone post a, I don’t know off the top of my head, but there was a blog post about the time we started this work where someone realized that hey, if you have automated pipelines that are add certain commands to OpenTofu and someone gets access to your state, you can actually change the list of providers in the state file and have it downloaded and install providers that potentially execute arbitrary code.
Christian Mesh 00:27:02 So the state is a very critical piece of OpenTofu and it should really be protected. Terraform has an answer to that feature, which is ephemeral. That’s something that we’re working on 110. I have not honestly looked at it in great detail yet. We will be evaluating whenever we’ll do something similar. But when I’ve talked about it in the past, it’s two complimentary feature sets, it’s the probably good idea to not store secrets in the state in the first place, but to a certain degree your state is a secret. It is a how your infrastructure’s configured, where it’s configured, all of these details that if someone gets access to that particular file that might open up a door you might not expect.
Robert Blumen 00:27:36 I ask you to pick some of your favorites. I want to talk about one now. This sounds really cool. Dynamic provider defined native functions. Why would somebody need that? What is it?
Christian Mesh 00:27:46 Sure. So you’ve written some HCL in the past, I’m sure you’ve probably run into comprehension or manipulating data structures. So I’m going to take one small step back real quick. So Terraform introduced a feature called Provider Defined Functions and that allows a provider to say, here’s a set of concrete functions that OpenTofu can call based on the configuration. So for example, the AWS provider has the eight ARN part. It takes a string and turns it into an easy to manipulate ARN object. At OpenTofu we saw that there were some additional hooks in the API that we could really play with. And what we now allow is you to define a provider that will take configuration that it can then use to expose additional functions. Now that could be a function that is, I can tell it what availability zone it’s in. It can say, hey, this is a valid thing for this zone or not, or it could do policy inspection and all sorts of interesting things.
Christian Mesh 00:28:41 The more fun ones that we really prototyped with is we have a Go in lieu of provider. So these are still experimental, Iím hoping to take some time in the next month to make them not experimental, but effectively what you can do is write a LUA file or even in line write LUA or Go and have that exposed directly as functions you can call within your OpenTofu configuration. This means that if you have a large policy that you’re manipulating, if you have data structures you’re trying to, let’s say you’re trying to sort by a key that doesn’t, is really hard to access or you just want to be type safe. Let’s say you want to have this large, let’s say you have this large data structure in OpenTofu that you need to pull apart and use in different areas. Maybe it’s coming from another data source and OpenTofu right now prior to this in OpenTofu you would really just have to write a really horrible spaghetti for comprehension structure that was really untestable and unergonomic. Now you could have a Go file that ships with your configuration and tests for it. So you can have strictly typed data manipulation in a language you’re familiar with inside of OpenTofu that can really patch over some of the gaps within the HCL language itself.
Robert Blumen 00:29:50 I’ve used Terraform. It has a library of built-in functions like the length of an array or, set intersection that are comparable to what many programming languages offer as built-in functions. They added modules in a release quite some time ago, which are something like Terraform functions. But there has always been missing the ability to define a new function and make it available as a first-class function. I understand this now, I missed the feature where they added functions that a provider could write and the providers are usually implemented in Go. So that would then give you the full power of the Go language to write functions that are on the same level of usability as built-in. Is that all correct so far?
Christian Mesh 00:30:39 Yes, that’s correct. And if you’re using Terraform at least currently you can only, you have to write a full provider and have all those functions defined ahead of time. Your configuration doesn’t get to choose what those functions are or what they look like.
Robert Blumen 00:30:54 The dynamic part of this feature then is able to load functions that are provided at the Terraform runtime. Is that the step up from the provider defined functions?
Christian Mesh 00:31:08 That’s correct. Effectively, the provider now serves as a bridge to whatever language that you want. Currently we support Go in LUA, but others have been experimented with like someone could write a JavaScript provider. So you could inline define JavaScript functions or in a separate file defined JavaScript functions and their tests and then have those functions be directly exported and able to be used just like the built-in functions within OpenTofu.
Robert Blumen 00:31:33 Prior to this conversation, if you’d asked me five minutes ago what is a provider? I would’ve said it’s a Terraform adapter between the core and something that has an API that affects infrastructure. It sounds like you’ve broadened the definition of provider to interface with really almost any external code. It doesn’t have to call an API on the backend.
Christian Mesh 00:31:57 Yes. Terraform originally made the changes to the API that that we then built upon and extended. You still need to write a provider to do as that adapter, but providers themselves have been able to do more things above the original idea that is just they connect you to a particular API.
Robert Blumen 00:32:16 Okay. Moving on to some of the new features. There was something that had not been released at the time I was researching called static evaluation of Provider iteration. Can you explain what that is?
Christian Mesh 00:32:31 Sure. That is one of the most exciting features we’ve worked on so far. It’s a big one. That’s one of the reasons the 1.9 release, our 1.9 release has been taking a while, but we’ve got the alpha out for that. But what, what exactly does this mean? So static evaluation or early evaluation, as some people call it, is the ability to evaluate expressions before the state is available. So we originally added early evaluation, the static evaluation in order to let you define your sources for modules in a static fashion. So you could have a config file that defines the version of all of your modules. So a module is a set of Terraform code or OpenTofu code that gets pulled in when you initialize the project and provides additional functionality in an encapsulated way. These are usually published and versioned, but in a lot of cases you’ll be using 20 copies of the same module or 20 subsets of the same module throughout your infrastructure and we’ll want to upgrade those all at the same time or switch them to a different source at the same time.
Christian Mesh 00:33:34 So there’s something that needs to be known before the state is available. And in order to do that, we have a separate evaluation engine that sits before OpenTofu really gets into its groove, doing its graph reversal and actually doing the application planning and application of changes. We’ve then realized we can expand that and use that functionality in a bunch of places. We support it in backend. So your backend configuration can be a little bit more dynamic. You don’t have to pass it in via the environment command line variables. You can have it a lot more integrated. But my friend and coworker Ronnie, she gave a talk at KubeCon about this that I recommend taking a look at. But getting back to the original question, early evaluation means that akin to a pre-processor, but there are some differences, but for providers, going with a concrete example, you might have the AWS provider and the AWS provider only functions within a single region.
Christian Mesh 00:34:30 So in order to do any sort of multi-region configuration, you need to have multiple copies of that same provider. So you’re copying and pasting config. Additionally let’s you have a module that you want to have that module. Let’s say you’ve got a hundred resources in there or some complex configuration and you’re passing a bunch of information into it from your root. Currently you have to copy and paste that module and all of its inputs and outputs as well, which is unfortunate. So this is something that has frustrated people a long time Terraform quite a few versions ago, I don’t remember the exact revision, added a for reach feature to the language such that modules and resources can be given a set of data that can then expand them. So you could have what most people would love is a resource per region, maybe a bucket per region.
Christian Mesh 00:35:17 Unfortunately, each of those instances of the module or the resource would previously have to use the same exact provider. So an OpenTofu. Now as of 1.9 alpha two, you can add a 4h expression, which is, it is just like, think about it like a map of data. If you, especially if you’ve used 4h in resources modules, it’s effectively the same concept. You can have a provider which is configured in one instance that’s configured in multiple different ways, which you can then refer to from a module or a resource and give it a particular key to use. So you could have one of the best examples is the AWS provider configure that for five different regions and then create a resource that uses for each expression as well. And that will then use that provider and connect that resource instance to that provider instance.
Christian Mesh 00:36:10 So your bucket in US East1 is connected to the provider in US East1. In order for that to function, you’ve effectively removed the giant foot gun of copying and pasting your configuration for every single region. There’s a lot of other examples of this, but that’s the one that’s most people will be familiar with in some fashion. And it is one of the top voted issue in OpenTofu and is I think top five and has been for the better part of, I think at least five years in Terraform.
Robert Blumen 00:36:37 Let me see if I understood that. I’m aware with AWS, it’s not only that I need an AWS provider, but the provider connects to a specific region of AWS, I might be building out infrastructure that spans multiple regions. So I would need a different instance of the AWS provider for each region. But otherwise a lot of code might be the same as a programming problem. Generally I would say, okay, I’ll write a loop or a dick or something that just assigns the regional provider to run on the regional object. That turned out to be not straightforward to do until this changed because the evaluation of which provider you use was more static or came too early in the runtime. How did I do?
Christian Mesh 00:37:22 That’s it exactly. That information needs to be known really before you get too deep into evaluation of OpenTofu, evaluating the configuration in the state. By adding this early evaluation feature, we can figure out what providers we have ahead of time and we can be a little bit smart on what’s being used where. There are some limitations to early evaluation. For example, you can’t have a provider that depends on data in the state for that 4h. The 4h expression cannot depend on like results from a query from a database or something of that nature.
Robert Blumen 00:37:53 That raises another question that might have fit in earlier had I thought about it. As far as the different kinds of shared resources that you bring into a Terraform project, we talked about the Terraform itself providers, there are also libraries of shared modules that you, you can do things like there’s an AWS VPC module that will create a VPC with subnets and you can have gateways, all the typical things you might or might not want to have. These are also open-source projects that are maintained on GitHub and can be loaded into Terraform. To what extent are the existing ecosystem of modules compatible with OpenTofu and do the module writers face the need to possibly fork?
Christian Mesh 00:38:41 That’s an excellent question. Currently we support the same modules. Again, this is, it depends on the language and we haven’t really changed the language in too dramatic of a way. There’s a couple new features that exist on one side of the other that folks need to be aware of, but in practice most people don’t need to worry about it today. But one thing we did add is something called the .Tofu file extension. This file extension looks is almost identical to the .TF extension, but in OpenTofu it has a special meeting and Terraform just ignores it. So if I’m a module author and I want to use a new feature that might be slightly different in one of the few cases between OpenTofu and Terraform, I can have a Terraform file that gets overridden by a Tofu file only when it being run by OpenTofu. So they can kind of use that as a, in the same branch with the same code. I can support both and just use that as sort of like a feature flag one way or the other. It’s also worth noting the JetBrains in their latest release, I think they just put this out a few days ago, they added full support for the .Tofu file extension and we’re working actively adding support for some of the other new features.
Robert Blumen 00:39:40 That concept reminds me of the overlay pattern that you see in a number of places. It’s pretty common in Kubernetes, with YAML or with Systemd services. You can create an overlay file that overwrites a specific portion of a default without having to edit the original. Do you know if you were inspired by this pattern in any other software?
Christian Mesh 00:40:05 Not off the top of my head. It’s more so replacing the entire configuration file. There’s something a little bit more similar to what you described, which are the underscore override files. Those are more used if you’re, honestly, I don’t recommend you use them. They’re not terribly well supported in either tool. But like the, so if you do underscore override after the file name or after the file name before the extension, it flags some interesting functionality within both OpenTofu and Terraform. But the main thing is for the .Tofu file extension is it’s a pretty simple set of rewrite rules where if the file exists, OpenTofu will load that .Tofu file over the TF file. And in Terraform, at least today, they completely ignore the .Tofu file.
Robert Blumen 00:40:42 We haven’t covered all of the new features, but we’re running short on time. Either we could talk about one more new feature or in the deluge of PRs that came into OpenTofu. Were there any ones that you remember as being significant?
Christian Mesh 00:40:57 I think there’s a few that stand out. The ones I get really excited about are the performance improvements. They’re usually interesting little puzzles that I get that I then get to dive into. But I think it was more so the volume and the interest from the community, the people who are chomping at the bit to say, I’ve had something I’ve wanted to do with this tool for a long time and I feel like I finally have the opportunity to do so. I don’t have too many specific examples, but I would recommend if someone’s curious, take a look at the release notes. The core team does a fair portion of the work for every release, but we’re consistently overwhelmed by of the amount of ideas and support coming from the community.
Robert Blumen 00:41:36 Does raise another question. Is performance of the Core Engine something that OpenTofu is trying to compete with Terraform on?
Christian Mesh 00:41:46 It’s something that, I don’t know if it’s a direct competition, if that’s the way I’d put it, but I think it’s something that’s important to a lot of our users and it’s something that we focus on. There’s a guy who frequently will send me on the OpenTofu slack a profile of his ridiculous setup. And it’s again, it’s one of those things where he’s, it’s not necessarily that we, the way we recommend someone use the tool, but it’s fascinating what people use it to get into and I’m happy to jump on and take a look at that and try and fix that case. Because nine times out of 10, there’s someone else out there that’s having the same problem. I will say that some people have contributed to both OpenTofu and Terraform and Performance is one of those locations. We had a user come in and submit a PR to both OpenTofu and Terraform, and at that point, I guess the code was similar enough that they could send the same PR to both. I didn’t look at the Terraform side, but in some cases, we do have the same performance improvements in other cases. I know OpenTofu, especially like for provider functions, takes a different path and we’ve had some, we’ve had a user analyzing the performance there and in some cases, OpenTofu has a pretty big advantage. Honestly, as far as where the future goes, I mostly just want to make users’ lives easier and if that means that we can reduce the cycle time, that’s something to focus on.
Robert Blumen 00:42:57 I want to switch over now talking about the roadmap. Talk about first how is the community involved in the roadmap?
Christian Mesh 00:43:05 Sure. I think the most visible way is the top thing on our GitHub issues is a top voted issues post that’s been pinned. That is something that we look at pretty much every day to see what users are asking for so they can give their thumbs up on an issue and that directly corresponds to where it lives in that list. That allows us to very quickly see what are users most passionate about. This isn’t the final thing we look at. Sometimes if there’s something that one of the developers gets really excited about or if someone puts in a PR, then that might prioritize it as well. But it is entirely based on what the community’s asking for. We do take a look at what Terraforms release notes have every time they put a new release out and we take a look in in within the core team what we think would make sense to pull in out of the box.
Christian Mesh 00:43:53 As far as features go, again, we can’t pull in any code, but we can look at documentation, we can look at the release notes and we can most of the time figure out roughly what’s going on there and implement that ourselves. But we don’t do that for every single feature. Some of the new features that are added to providers, we haven’t added for a release or two because there’s not really anyone using them yet. There’s a lot of stuff that’s added to Terraform that is more focused on their Cloud offering and what’s going on there, whereas we’re entirely focused on what the community’s looking for. But if a community comes to us and says, hey, there’s a discrepancy between OpenTofu and Terraform here, or if there’s a feature that OpenTofu only implemented part of because that was what they thought the community wanted, then we’ll go back and include that in a subsequent release.
Christian Mesh 00:44:37 The actual release milestone itself is comprised of a number of issues that the core team is interested in getting into the next release, but it’s very fluid. If the community is very focused on performance optimizations or adding certain new features, we’ll change our plans for release. We’ll start pulling that in, reviewing those PRs and getting those in. And it’s dynamic. We are not the ones that have an idea of what OpenTofu is going to look like in five years. That’s one of the hardest questions I have to answer is we don’t know. And that’s intentional. Our job is to figure out what the community’s asking for, what the community needs and to build the best tool for them. And I know I sound a little bit sappy there, but that’s what I believe.
Robert Blumen 00:45:14 Can you pick one issue on the roadmap that you think is quite interesting or worthwhile?
Christian Mesh 00:45:21 The one that I look at a lot is I tend to think about improvements to backends. I mentioned earlier that OpenTofu doesn’t really understand what AWS GCP and other Cloud providers are. There’s one major exception to that, which is the backends. A backend is something that can, a remote state backend is something that stores the state. So when you’re talking about your state file, some people just commit it and get other people, put it in an S3 bucket. It can go in a variety. Sometimes it goes in a database that’s fluid, but there’s currently a set of backends that are built into OpenTofu, but there isn’t a great way of someone writing their own backend. And there are some pretty significant limitations of backend. So if the AWS API changes or is enhanced in some way, you have to wait for OpenTofu to patch itself and then push a release that includes those changes.
Christian Mesh 00:46:14 So one of the things is S3 added an option for locking so that our Dynamo DB setup is no longer needed. That’s something that will have to go out in a release at some point, but IT users will have to wait for that. One thing we’re trying to work toward is first building a better HTTP backend. That’s something that exists today that a lot of companies build around. It allows you to, it’s a relatively straightforward HTTP protocol that says it can accept a state file, provide a state file. It can do locking mechanisms, those sorts of things, but it does lack some pretty significant functionality. So we first want to improve that, then we want to start writing a library to make that easier. But once we have that library in place with compliance tested and helping people just build a backend that works and as well tested, once that’s in place, our goal is to start looking at adding GRPC support to that mostly because that’s what OpenTofu uses to talk to providers and as the next stage, start looking at integrating that within the provider protocol or something similar.
Christian Mesh 00:47:13 So it’s shipped in the provider binary or maybe as an additional binary that goes along with it. We’re still, the technical details on this are still a little bit sparse because we’re still figuring it out and there’s a lot of different ways to go with it. But functionally speaking, we’d love to be able to ship a backend binary similar to how we ship provider binaries today, where someone can say, hey, I want to use this particular version of the AWS API because it uses the type of credentials that I’m familiar with and I haven’t had time to upgrade my organization yet. This also means that it takes burden off of the OpenTofu developers. We could move all those backends into community projects so it’s not within the OpenTofu day in day out practice and people can upgrade them, we can release them asynchronously from OpenTofu releases if there’s a new AWS feature or GCP feature. If someone finds a bug and wants to patch it, it is, doesn’t have to be locked to the same release cycle. Similar to how providers are flexible today.
Robert Blumen 00:48:03 Do you have any data on the adoption so far?
Christian Mesh 00:48:06 I do. So OpenTofu does not have telemetry. We immediately rip that out as soon as the fork was created. But we do have sets for downloads of both the OpenTofu binary itself and for providers. And additionally the companies built around OpenTofu do have some internal metrics. So our actual downloads, I’m taking a look at the Space Lift blog. I’m sure it’s on some of the other blogs after OpenTofu Day that says KubeCon. It has been a pretty linear increase since we started. So we are in total weekly downloads as of today. We are at 250,000 downloads per day of OpenTofu. How that compares to Terraform, they don’t, as far as I know, they don’t publish that information. Or if they do, I’m not aware of it. But we have a, it’s been a fairly linear increase and we’ve seen that both in, again, the amount of downloads and the number of issues on GitHub and the growth of the community. The nice part is, although I am around for supporting users and the core team is around for supporting uses when they run into issues either in OpenTofu or just the problems faced in general, a lot of the time we’re actually beaten to the punch by other members of the community because it’s grown large enough that they can help each other out. And I love to see it.
Robert Blumen 00:49:15 We mentioned OpenTofu.org. Are there any other places on the internet that you’d like to point listeners toward?
Christian Mesh 00:49:22 I think at this point, OpenTofu.org is the main spot. We have a blog, we probably don’t post as much as we should, but after OpenTofu day, we should have the videos from that day that we’ll be posting. We’ll also be posting that on our YouTube channel and we’ll also occasionally post short videos there showcasing new features. And we’ve even done a couple live streams there just to give people a way of chiming in live and reacting to stuff we’re working on. And so there’s the OpenTofu community Slack, linked from OpenTofu.org. And we also have a Google meet that we do once a week as the core team that we invite anyone and everyone from the community to, so we can talk about what the core team’s been up to, what we’re trying to help different people in the community with, and just try and in general answer questions, kind of like how we’re talking today.
Robert Blumen 00:50:10 Is there anywhere listeners can find you?
Christian Mesh 00:50:13 So primarily on the OpenTofu Slack, on the OpenTofu GitHub. I’m one of those strange people who somehow has managed to avoid creating a LinkedIn up till this point. But yeah, if you’ve head over to my GitHub, cam72cam(Christian Mesh), I have a Google calendar link there. I welcome people to just schedule a time to meet and talk. Again, I enjoy listening to how people are working with OpenTofu, what questions they have, if they have any ideas they want to talk through, anything and everything. My schedule’s pretty open. Rather I can move my schedule around well enough that, yeah, if you’ve got ideas, you want to talk about an issue on GitHub, you want to talk about ideas, you’ve got, reach out and I’ll be here waiting. Excited for it.
Robert Blumen 00:50:52 Christian, thank you for speaking to Software Engineering Radio.
Christian Mesh 00:50:56 Of course. Thanks for having me on. This has been a pleasure.
Robert Blumen 00:50:58 This has been Robert Blumen for Software Engineering Radio. Thank you for listening.
[End of Audio]
