Search
Search
jens_neuse
August 9, 2023

SE Radio 576: Jens Neuse on Back Ends for Front Ends

Jens Neuse, founder of Wundergraph, joins SE Radio host Jeff Doolittle for a conversation about back ends for front ends, or BFF. Jens begins by explaining how a heavy integration burden is often placed on front-end development teams. When multiple APIs must be integrated, it can be challenging for client development in web, mobile, and desktop environments. Explaining how APIs should be treated as dependencies, just like packages, the episode explores BFF patterns and use cases, as well as the future potential emergence of a “git for APIs” standard.

This episode is sponsored by ClickSend.
SE Radio listeners can get a $50 credit by following the link below.
ClickSend logo

Show Notes

From the Show

From IEEE Computer Society

From SE Radio

Transcript

Transcript brought to you by IEEE Software magazine.
This transcript was automatically generated. To suggest improvements in the text, please contact [email protected] and include the episode number and URL.

Jeff Doolittle 00:00:18 Welcome to Software Engineering Radio. I’m your host Jeff Doolittle. I’m excited to invite Jens Neuse as our guest on the show today for a conversation about backends for front ends. Jens is the founder of Wundergraph, a next-generation BFF framework designed to optimize front-end, full stack and backend developer workflows through API composition. He is also the creator of graphql-go-tools, a Golang-based open-source project to build GraphQL middleware proxies and API gateways used by many GraphQL API vendors in the industry as well as many enterprises. Jens welcome to the show.

Jens Neuse 00:00:56 Hi Jeff. Thanks for having me.

Jeff Doolittle 00:00:58 Absolutely. In a recent conversation you and I had, you mentioned that a lot of front- end development teams bear the burden of integrating with lots of distinct APIs and it seems like there’s always new dependencies that they’re taking on. So from a high level, from your perspective, what can be done to reduce this integration and coupling burden?

Jens Neuse 00:01:17 Yeah, I think on a high-level perspective, what we currently see over the last couple of years is we decompose larger systems into smaller parts. We use more and more SaaS systems that provide individual APIs to solve a very specific problem. And what we’re actually trying to figure out is what is the right place to integrate all those systems into one unified API interface? And obviously there’s different places where we can handle this integration and I would say from a frontend developer’s perspective, it can be a huge pain if you have to do all of that with almost no tools that help you facilitate that, do it in the front-end. There’s security pitfalls, but also most front-end frameworks if you look at what they are actually good at in terms of, you know, binding data to the user interface and facilitating that. But what do front-end frameworks have to offer for this modern development use case where you want to integrate with multiple services and most of the time there’s really barely anything. So you’re looking for new ways to solve that problem. And I think that’s a huge challenge for front end but also full stack developers.

Jeff Doolittle 00:02:42 So it sounds like you’re saying that the frontend frameworks, while they may be useful, they’re not necessarily helping solve the challenge of how do I integrate with a bunch of different APIs as a front-end developer?

Jens Neuse 00:02:53 Yeah, exactly. And I actually think this is also a good thing because not necessarily is it the right place to solve that sort of problem. And I think the UI frameworks like React NextGen views, I think the way they are scoped is really great. And usually what you find is that you have this one front-end and it talks to this one API, that’s your backend. And that works very, very well. However, as we said earlier in the discussion, we see that we have more and more services and integrations we want to build with and then we have Payments and Stripe and other SaaS services for sending SMS and whatnot. And we have so many different systems we’re now using, I think for example at Wundergraph, we have at least 40 different vendors that we use ourselves that provide something to us. If you want to integrate all of that in the front-end or wherever you actually want to integrate that, it’s a challenge by itself.

Jeff Doolittle 00:03:57 Yeah. What’s maybe one of the worst things you’ve seen from a front-end integration standpoint? I mean, you just mentioned 40 vendors, so you have 40 vendors and they all have their own APIs. I mean, have you seen situations where front-end teams are literally bearing the burden of trying to integrate tens or dozens of different APIs together in the front-end?

Jens Neuse 00:04:14 Well, I think one of the obvious problems you can very easily see and that it simply shows the struggle of especially younger front-end developers, is when people go to stack overflow and ask where do I put my environment variables into my React application? And you see from the way how they phrased that question, it’s kind of obvious that yeah, obviously they like you should never put a secret of sort or an API key into your React application. And I think there’s a kind of like a gap in education and we need to talk more about, you know, composable architecture and different roles and responsibilities. Like what kind of logic should sit in the backend, what kind of logic should sit in the front-end? And for me personally, worst case you have a front-end that talks to many different systems with different protocols.

Jens Neuse 00:05:15 All the integrations happens in the front-end. You get huge waterfall requests when you visit a new page. Requests depend on requests, depend on more requests, you have like lists and then nested fetches, it all goes to different services. And at the end of the day, for me, what’s the biggest nightmare about this kind of architecture? You actually don’t really understand what this application depends on. And it’s very, very unclear if something is slow or doesn’t properly render or you’re unhappy with the scores your website gets in like Lighthouse, which is a performance check for websites, you don’t really know what’s the cause and it’s just very hard to track that down.

Jeff Doolittle 00:06:01 So when teams begin, some of them with integrated and everything in the client, what do they often do as a next step? Like maybe they’re feeling this pain, they’re realizing that oh it’s really hard to do retry and roll back logic in the front-end. It’s really hard to make sense of all this complexity of you know, integrations with all these different things. So what’s often the next step that those teams start taking? And then talk to us a little bit about the challenges of maybe what their next attempts tend to be.

Jens Neuse 00:06:27 Yeah, so as you have more and more modern front-end frameworks that also support server-side rendering, for example, NextJS has this API directory and what you can see is that a lot of people simply start writing or adding the API integration layer into that API route directory. And so they are moving this logic from the client to the server. It is still in the same repository and it’s kind of like the same level of complexity. What’s a bit better is we now have a security boundary between client and server, and we are potentially closer to the APIs that we’re integrating with. So we can eliminate some latency. But if you look at the core problems that we need to solve, you will see that a framework like NextJS and others are no different. They’re super agnostic in terms of helping you with API integration.

Jens Neuse 00:07:32 Like you can create an API endpoint, kind of like a REST API thing. And that’s what you get from the framework and the REST is on you and you need to ask yourself the question. We have so many APIs, a lot of them public like GitHub and Shopify and Stripe and others. Should every developer, should every application that integrates with them, should everybody write more or less the same kind of logic to integrate APIs. And we had a very interesting discussion earlier today because we did a livestream with the founder of SST, Dax Raad. We were talking about this topic, what’s the difference between building an application and building a library? And what we came up with or what our conclusion was is when you’re building a library like two libraries on different problems like SST and Wundergraph they are probably completely different because they’re solving so different problems. But when you compare a couple of
e-commerce apps or just the NextJS applications, they will actually share a lot of code. And that’s kind of, especially in the integration part, does it actually make sense that we write so much manual integration code?

Jeff Doolittle 00:08:54 Yeah, so teams might start moving to the backend but to your point, while they may gain some benefits of a security barrier and maybe they can reduce some latency, they’re essentially just moving the complexity to another place. And it sounds like from your perspective, something is trying to emerge here and I think one of the next steps you’ve mentioned that teams start to go to is they start realizing that whether they like it or not, they’re kind of now building their own backend for their front-end. So maybe lean into that a little bit, especially for listeners who aren’t familiar with what is backend for front-end, you know, where does the term come from? And then let’s talk about some of the pitfalls when teams discover that they’re actually starting to build their own BFF framework.

Jens Neuse 00:09:35 Yeah, I think it was 2012, I really hope, I don’t get it wrong, but it was around the time where SoundCloud came up with the term backend for front-end. And it’s essentially the idea that you have, or back then you have one single monolithic API or it could actually be also microservice architecture. So it’s agnostic in that term. But let’s say you have a bunch of services and then you don’t have one particular front-end but multiple and back in 2012 I think it was this trend to go towards mobile. So what you usually had is you had like I would say your main API surface and then you had a backend for front-end for the website, a backend for front-end for your iOS app and the backend for front-end for your Android app. Simply because the requirements of each individual app and the website were so different that SoundCloud thought, wouldn’t it make sense if we have a specific backend for each individual front-end? So it’s kind of like a, I don’t know, you could say maybe a shim, but it’s actually, I think it’s a bit more than just that. So it’s kind of like a layer on top of your API layer that simplifies cross-cutting concerns like authentication and API composition and other patterns pulls that into the BFF and then you don’t have to handle that in the front-end code.

Jeff Doolittle 00:11:05 Right. And it sounds like there too, if I understand you correctly, that they were building sort of platform specific backends for certain kinds of front-ends. So maybe here’s the mobile one that has slightly different API surface area than maybe the web. Sort of bringing the integration barrier down a little bit for teams that were maybe on a specific type of footprint for a device.

Jens Neuse 00:11:26 Yeah, exactly. Like if you think about app development more specifically, you’re probably aware that once you change an app, it needs to go through the app store, at least on iOS and Android I think it’s a bit more automated and you usually have to support a native app for a very long time because not every user will update that. So that means if you have a web application that is very fast paced, like you know, you can redeploy the web app, the front-end and the BFF and the moment they change, they can change together. So you can continuously change your BFF and how the website talks to that. For an iOS and Android app, you cannot continuously change them because you need to maintain those old versions. And you know, one of the core paradigms in software development is you want to keep things together that change together and you want to keep things separate that’s change separately.

Jens Neuse 00:12:25 So you have a different release cycle for iOS, a different cycle for Android and a different cycle for your web app. And also the requirements of each individual app are different. So I think it’s kind of makes a lot of sense that you figure out these boundaries and for each of those you have a BFF and that can handle versioning and can maintain old API versions. And that also gives you, that’s another interesting pattern here, it gives you a facade pattern where you can actually put a facade into your BFF that can sit or that can sit between the internal APIs and an old version of the Android app. And you can keep maintaining that facade. You can change internal systems but as long as you keep the facade up to date and maintain that, old applications will keep running. And that’s one of the big benefits here of using such a BFF.

Jeff Doolittle 00:13:25 Yeah, it’s interesting too because a lot of what you’re describing there in the more abstract is about layering and in his book A Philosophy of Software Design , John Osterhaut talks about deep modules. And I’ll put a link in the show notes to the appearance that he had on this show. But I think that’s what a lot of people end up discovering is, if we just had these very flat systems where it’s essentially a database with an HTTP shim on top of it with a frontend React application, then there’s a ton of coupling to your data models you’re going to have a ton of problems with how do you handle transactions and complexity, especially as you’d adopt other APIs and systems. But broadly speaking, there seems to be this growing recognition of a need for deeper layering. And of course layers have a cost, they can introduce some latency, they can introduce complexity, but it seems like we’re finding cases where this layering is actually quite warranted and make some things far simpler than the alternative.

Jens Neuse 00:14:17 Yeah, I actually think that sometimes we’re very enthusiastic about optimizing something down from 100 milliseconds to 95 milliseconds, but at the same time we’re wasting a whole month or two of precious engineering time and what’s the actual user benefit at the end of the day. So yeah.

Jeff Doolittle 00:14:40 Yeah, I agree performance is a feature, but if you’re eeking out three or four percent, you know, that’s different than if you can eek out, you know, 10X or a 100X in performance. Now I’m listening now maybe it’s worth that additional engineering effort. So one of the things you said to me in in a previous conversation, which I thought was an interesting concept was that you recommend that APIs be treated as dependencies like any other package. So talk a little bit about what that means and what the benefits are of taking that approach.

Jens Neuse 00:15:06 Yeah, so you know when we write code, let’s say we created next JSF and we decide we want to use some sort of dependency a servers or an SDK or we just want to add some code to our application. The way we do that is we NPM install a package and then we have something, it is called the package manager. We have a package JSON file and with the package JSON file, we have a list of our dependencies for code and we know exactly okay, we installed this package, it is now in in node modules and we know what our application depends on in terms of code. Now we’re building a BFF with our NextJS stack. So what do we do? We add Axios or install an SDK and with Axios we make some fetch calls to an API, where’s the package JSON? Where do we actually know what we depend on and why is not everybody shouting from the roof that something is wrong? Like, we have such a great pattern, you know, we have Go modules, we have crates in Rust, we have Maven in Java land, we have NPM and for APIs we have kind of implicit dependency management. And that for me, as I was building this solution and started talking about API dependency management, I felt like it is so obvious to do that. Right?

Jeff Doolittle 00:16:44 Yeah, it makes sense to me. I mean, you know, we’ve all heard of the two hard problems left in computer science, right? Cache and validation naming things and off I one problems. But you know, I typically say there’s two hard problems left in software engineering, information hiding and dependency management. And what you’re highlighting here is dependency management is more than just about package management, it’s also about when you integrate with an API of some kind, you are also taking on implicit dependencies. And boy I’ve heard the phrase, I don’t remember the source of it, but you know, let’s make the implicit explicit and I think that’s exactly what you’re calling out here, which I think is great.

Jens Neuse 00:17:19 I mean if you look at larger companies or even smaller ones, our startup with 40 different vendors, if you look at or genuine question for you, if you look at a recent company you work with and you looked at one of their backends and you glanced at the code of their backend, did you have an idea of what this backend depends on in terms of APIs? Was it obvious?

Jeff Doolittle 00:17:47 In terms of APIs? No, in terms of code modules as you said, absolutely it’s pretty straightforward. But as far as what APIs it depends on, it’s very opaque.

Jens Neuse 00:17:57 Yeah. And for me I think it’s a huge problem because we simply have all this code but we actually don’t understand what’s happening. It’s so intransparent and yeah, I’m very enthusiastic in actually making that more transparent and explicit in saying, hey, we have this BFF because we’re building a BFF for our mobile app, and this BFF it depends on these five microservices and one SaaS service for sending SMS. And now it’s very obvious and we can tell everybody in the company, this is what we depend on, this is our quality of service, we have this one operation, it’s actually slow because of that one microservice and we can trace all of that and I don’t know, life would be a better place.

Jeff Doolittle 00:18:42 Yeah. And you know, now if you have that, you can imagine that things like observability and telemetry become simpler tracing as you mentioned because now you have a clear sense of how these different actual dependencies are affecting your system as opposed to just treating code modules as dependencies and forgetting that APIs are dependencies. So I imagine there’s some challenges to this approach. Like for example, you know, I know a lot of people use things like open API and Swagger and you know, maybe there’s some skeptical listeners who are saying that’s enough, that’s sufficient. So maybe explain a little bit about where that kind of an approach isn’t sufficient to solve these kinds of problems we’re talking about.

Jens Neuse 00:19:21 Yeah, I think the biggest challenge with API dependency management and automating the process of building BFFs is actually that every API is kind of different. It’s actually a huge problem to find open API specified APIs that have like the same level of quality. So usually open API can, due to the nature of open API is used to document existing APIs, not always to go like API definition first and then implementation. So you’re varying levels of quality and then obviously you don’t just have REST APIs. So in my previous jobs we had GRPC, SOAP, weird XML APIs, custom HDP APIs. So if you want to implement API dependency management, you actually need to find some sort of common denominator to facilitate that.

Jeff Doolittle 00:20:22 Right. And unfortunately it seems like the common denominator typically is, I don’t even call them REST APIs because you know, if you read Roy Fielding’s thesis, which we’ll put a link to in the show notes, REST is extremely uncommon. True REST. But setting those debates aside, not every APIs an HTTP API, right? As you mentioned there’s GRPC, there’s GraphQL, there’s even things like Async API where you know, you start to look at, I mean the real word that matters in API from my perspective is interface. Well an interface is just a contract about, you know, data can flow between these two things and this is what the messages look like and that could have lots of different protocols and transports and things like that. So that definitely I think is one of the challenges. And when we reduce to the lowest common denominator, we often end up trying to push square pegs into round holes.

Jens Neuse 00:21:09 Yeah, I totally agree.

Jeff Doolittle 00:21:12 Well you kind of talked a little bit about it, but let’s dive into some of the patterns that are involved when adopting a backends for frontends approach. And you let me know before the show that you’ve actually set up a new website, BFF-patterns.com. So I definitely encourage listeners to check that out, but let’s talk a little bit about some of these patterns and maybe give our listeners a flavor for what they can expect in their world if they’re trying to build backends for front-ends for their systems and simplify the front-end integrations for their client delivery teams. So first off API adapter, let’s maybe talk a little bit about that first one and say what is an API adapter and how does that relate to the backends for front-ends approach?

Jens Neuse 00:21:52 Yeah, so API adapter is really, I would say one of the, like together with API composition, it’s kind of like one of the simpler models where you have, let’s say a composed schema of APIs. So you combine a bunch of APIs into what you could call, if we want to stay generic, let’s just say that’s a composed graph that represents the multiple APIs. And then we have an adapter on top of that for different flavors of our web application. So in this sense we have a mobile app that uses three services, then we have a web app that uses some other three services and we have a third party, API that depends on two other services. And in this case, the adapter is simply a subset of the whole composition of APIs we have.

Jeff Doolittle 00:22:46 Okay. So then let’s talk a little bit about API composition and maybe we can see how these two patterns tend to relate.

Jens Neuse 00:22:53 Yeah. API composition, I would kind of say it is very much related to API adapter in the sense that API composition allows you to say, okay, we have five different APIs and they can be REST, GraphQL or maybe even a database or something that doesn’t by itself speak HTTP and API composition allows us to say that we combine all of them into one single unified API and expose that to our applications. And what that essentially allows us to do is a client doesn’t have to depend on multiple individual services, but it can rather depend on this facade, which is the composition of our services. So it doesn’t have to understand the different protocols and languages and interfaces. It simply has one single interface it can talk to.

Jeff Doolittle 00:23:52 Okay. So putting these two things together now, just to kind of paint a picture for our listeners from the composition standpoint, I’m saying maybe I have to talk to three or four different APIs. These could be GraphQL, these could be REST, these could be databases, some combination of those. And step one in composition is let’s collect all those together into a singular, like API, like a super API. But then the adapter side of it is let’s not ship all of the super API out to our clients, let’s give them adapters that are catered to the specific needs of mobile or web or whatever footprint they may be dealing with.

Jens Neuse 00:24:29 Yes. So for example, let’s say you introspect a database and you create an API schema from that and you add the REST API and the GraphQL API, you can’t imagine how big the API surface actually now is. So do you want to expose that whole API surface that even exposes internals of your database? Do you want to expose that all the way to the client? Probably not. So you might want to have an adapter that sits on top of that that creates just another API layer, which for example could be a REST API or adjacent RPC. And that’s now the communication layer between the deployed application and your BFF, which also means that you can leverage this layer to enforce loose coupling because the API composition can change, but the facade or BFF, it can keep the contract between client and BFF intact.

Jeff Doolittle 00:25:30 Right. And here’s where you can get your information hiding and you can improve your dependency management because now your front-end is not directly coupled to your super API, but the super API is giving you the power to support these adapters and facades that meet the specific needs of the specific front-end application. So you kind of, it’s sort of like you’re trying to balance the details between meeting the needs of the front-end but also meet the needs of the integrator. So the integrator’s not having this complex mess of wires, you just kind of integrate it all together by composing it together into a, I don’t know if super API is the right term but .

Jens Neuse 00:26:03 Yeah. You know, for example there’s a couple of tools and over time they are a bit more popular and then a bit less. But there’s a couple of tools that generate you an API from a database. And some people love to connect this directly to the front-end. And that makes you extremely fast for the first couple of days. And it makes you super-efficient. And sometime later the application is deployed, someone comes up with a brilliant idea which is very regular, to change the name of a database table. And because the API is derived from the names of the table, you just broke your API, you just broke your client. So it’s always tradeoffs, like something that makes you, like you, you need to be suspicious when something makes you extremely fast because maybe that’s a tradeoff you’re currently making.

Jeff Doolittle 00:27:00 Yeah. I mean, you know, I could strap a rocket engine to my back and I might be really fast, but that may not be very safe or sustainable .

Jens Neuse 00:27:09 Yeah.

Jeff Doolittle 00:27:10 Well let’s talk a little bit about API coordination. That’s the next pattern from the BFF pattern site. What’s the problem here with API coordination that we’re trying to solve?

Jens Neuse 00:27:20 Yeah, so one use case that we keep coming back to is, as much as I love REST APIs, sometimes REST APIs are very granular and they want to cater different use cases. So they are not specifically designed for a given user interface, which is good. They’re designed around a resource like a shopping cart or something. And then what can happen is, and this is what some of our users keep running into, I need to make 17 REST API calls to populate the data I need for this view and API composition. You’re laughing .

Jeff Doolittle 00:28:01 I know I should be crying, .

Jens Neuse 00:28:02 Okay. Yeah. So because front-end developers, they didn’t have a better way of doing that and the fastest way of integration, it was really just front-end. They didn’t know better. With the API coordination pattern, what we do is we create an endpoint on our BFF and that allows us to talk to multiple services, do the 17 API requests or whatever we need, combine it all together and then send it back to our client. So our client makes one single request, our BFF handles all the coordination and that makes the whole thing a lot easier for us on the client side.

Jeff Doolittle 00:28:43 So essentially if you’re trying to coordinate calls to two different services, instead of having the front-end developer try to coordinate that call and then of course handle errors and retry and this sort of a thing, this is pushing that coordination logic back into your BFF so that the front-end’s not responsible for that coordination logic.

Jens Neuse 00:29:00 Exactly.

Jeff Doolittle 00:29:01 Yeah. I see this all the time. That’s why I was laughing. I mean if you don’t put workflow into your backend system, it will emerge in your front-end system and it will be painful . So that’s why I was laughing, but like I said, I should be crying.

Jens Neuse 00:29:14 By the way this brings me to the point you need to invite. I’m not sure if he was here yet, but you should maybe invite the founder of X state.

Jeff Doolittle 00:29:24 Okay.

Jens Neuse 00:29:24 Stately. Okay. It’s very interesting because you can talk with him about state machines. You just reminded me of like, you have state machines everywhere and a lot of the time state machines are like, it’s a different problem space, but a lot of the time we have state machines, we make them implicit as well.

Jeff Doolittle 00:29:43 Yes. Or they show up as if else logic, which is also a kiss of death for complexity.

Jens Neuse 00:29:48 Yeah.

Jeff Doolittle 00:29:49 So next pattern is API evolution. How does this pattern help us in a BFF world?

Jens Neuse 00:29:55 So I think that’s actually one of the most important patterns and it’s also because we want to change things. And it’s also kind of correlated to what we said before with the database that we directly exposed to our front-end. If we use the API evolution pattern, if we have a BFF that’s sits between client and actual backend, we can have an endpoint that represents some logic, we can have that on our BFF, and it’s essentially wraps some legacy API and over time we can slowly change the implementation without breaking the client. And that’s, that’s essentially how we can slowly move off like an old service or something. And if you have, not just web apps, but also mobile apps, stuff like that, you’ll always want to keep an old contract up and running. And so in this case, you can use the API evolution pattern.

Jeff Doolittle 00:30:58 Great. I think that’s actually one of the problems, like you mentioned before, the ability to spin up a database, rapidly ship, you know, quote API I am doing that in scare quotes and now you’ve basically directly coupled your front-end to your database. And what API evolution is solving for is the dynamic nature of the systems that we build, meaning they’re going to change, but how can we accommodate that change over time instead of creating brittle and fragile systems where every time something changes, we have to redo everything.

Jens Neuse 00:31:24 Yeah, that’s exactly the point.

Jeff Doolittle 00:31:27 Next, pattern is API lens. This one might be a little bit challenging to present since we don’t have the visuals that we can show but talk a little bit about API lens and how, maybe it’s similar but different to the API adapter pattern.

Jens Neuse 00:31:40 Yeah, I think API lens in a nutshell is we have, let’s say we have our super graph of API dependencies, that’s our composed API graph from that, we create an adapter, but we realize we have actually like two different variations of how we want to use that adapter. So we can have even one more layer on top of that, like an API lens that gives us like one more option to filter out something else. A use case here could be you compose your APIs, you create that adapter, which is your layer on top, and now a third-party vendor comes around the corner and says, hey, I want to integrate with you. And you don’t want to give them the whole adapter because that might have some internal things. So you put a lens on top of that that really just gives them exactly what they need and then that filters out everything else.

Jeff Doolittle 00:32:39 I see. And so from the adapter, it looks like it’s more about adapting for a specific kind of client, like a device or a web versus with the lens, it’s kind of an adapter as well. It’s a way to organize a set of endpoints and APIs together, but it’s doing it for a different reason. It might be, as you said, you want to expose things to another, you know, third party. Imagine another use case might be that you maybe want to organize things by something more related to the business domain. And so you want to be able to create a lens that that’s oriented around, you know, these all relate to a certain kind of user, for example. So we’re going to consolidate these together. So similar to the adapter, but kind of with a different purpose, I guess. Okay. And then how about API name spacing?

Jens Neuse 00:33:23 Yeah, so that’s something very early on when I started experimenting with the idea of treating APIs like dependencies, what I realized is that we don’t just need to steal the dependency management pattern from code, we also need to steal the name spacing pattern. Because it’s kind of obvious that if you combine two or more APIs, especially like from public renderers, or even if you just combine two databases, it’s very easy to have naming conflicts. Like everybody has a user, nobody cares. Like when you create a database, you don’t namespace your types because you, you don’t think of that or these names being used in another context. So what we actually need to do is when we combine multiple APIs and treat them like dependencies, we need to figure out a way to namespace things so that you don’t have a user in Stripe and a user in Salesforce.

Jens Neuse 00:34:29 And then it’s kind of big use. What is what and should the fields of the two users merge, or what do we actually do? Like they’re not the same thing. So at the end of the day, what the solution here is that you have a Stripe user and you have a Salesforce user, and that way you can, you can very easily identify them, distinguish. And that’s also what you have in let’s say, TypeScript for example. You import a module and you can alias that and that way you avoid the naming conflicts. And that’s essentially what we also need for APIs if we apply API dependency measurement.

Jeff Doolittle 00:35:07 Yeah. And that makes sense too, that the more APIs you start integrating with, the more risk of collisions you’re going to run into. So having these kinds of patterns in the early stages may seem like, well, why would I need that? I’m only talking to one API. But as you extend and expand and you start talking to more, you’re going to start running into these situations where you’re going to need some of these techniques to help you avoid collisions and things of that nature.

Jens Neuse 00:35:30 Yeah.

Jeff Doolittle 00:35:31 Cool. Now we’ve gone through the patterns of backends for front-ends. Let’s talk through a couple of the use cases for backends for front-end. So the first one you mentioned is API consolidation. So talk a little bit about what API consolidation is and what problems it helps us solve.

Jens Neuse 00:35:47 Yeah, so this is actually where I more or less started my career in this space. I was working for Germany’s largest news website to online. We were building essentially a BFF out of the requirement that, and we did that in a manual way back then. So if you think about a news website, we had a content management system, we had data providers that gave a, like football data, financial data, sports weather. Then we had integrations with Amazon affiliate, we had our own internal systems that spoke like GraphQL. We had something, we had REST APIs, all sorts of things. And so this whole use case, it’s about combining APIs, composing them together into a single unified interface and make it simpler for a developer to develop against a hetero organicist set of systems.

Jeff Doolittle 00:36:52 And that really sounds like what a lot of people are trying to get to, but they’re doing it through pain driven development, right? By trying to integrate in the client, or as you said, they’re ending up building their own backends for front-ends, you know, approaches. But it sounds like, and we’ll get more into this later in the show, but it sounds like there’s some patterns emerging for maybe how to do this in more consistent ways as we’re trying to simplify our lives of our, of our client developers. Another use case you mentioned is a universal SDK, which I get really excited when I see Universal SDK, but I’m going to let you explain what it is and why it relates to backend for front ends.

Jens Neuse 00:37:28 Yeah. So when we started working on the backend for front-end space, the first step for us was to build this API consolidation layer where we translate different protocols and different APIs specifications into the common denominator, and then have the user or allow the user to speak one single language against all these services. And what we realized is at some point that you cannot just, so in our case, I want to make it a bit more specific. So we allow our users to talk GraphQL to all their API dependencies, and that was our starting point. GraphQL in this sense is great because it has one feature that no other API style has because it’s a query API style. You have one component of the graphical spec, which is the selection set. And with the selection set, you can select, as the name suggests, a subset of the API surface.

Jens Neuse 00:38:39 And that is essential for API integration because if you combine and compose multiple APIs, and you might even nest them, you need to have a way of cutting off the tree of data because you don’t want like to eat out, you don’t want to query everything, right? You want to limit what you have. And obviously you could do something like that with REST APIs as well. There are specifications, sparse field sets, for example. And other ways to define, I want to have a subset of a resource, but I think GraphQL is just better in doing that because it has such a native feature. We have the specification, but regarding the universal SDK, what we realized after a while is, our users they are not happy with a pure graphical approach. Because sometimes you want to get some data from a service and you might want to join it with another data source, but you want to massage the data a bit.

Jens Neuse 00:39:43 You want to map it, or you simply want to create a facade that is not derived from the backend. So from your API dependencies, but you really want to have a decoupled facade that is defined by itself. So the facade defines, okay, here’s an operation, has an input, it has response shape, and then you can implement that. So we learn from our users building BFFs that simply having something like a graphical API gateway, it is not enough to implement a BFF. They sometimes want to have some custom logic. And so we allowed them to write custom logic with TypeScript. And from TypeScript, we also allowed them to call into predefined operations against the GraphQL schema that we have generated. And that caused a little bit of friction because whenever you wanted to create a custom operation that also uses GraphQL to talk to your API composition, what you had to do is first you define the GraphQL operation, then you wait a little bit for the code generation process that creates some types.

Jens Neuse 00:41:03 So everything is type safe. And then you can use that from your operation definition. And that caused a little bit of friction because it was just a little bit slow and yeah, we weren’t happy with that. So we had a new idea. What if we take the API composition, so the, the whole graph of all our APIs, which is essentially a GraphQL schema. What if we take that and generate from it an SDK? So that in our case, you can simply use TypeScript and talk to all your API dependencies in kind of like an ORM fashion. What this allows you to do is from a workflow perspective, step one–you add your API dependencies to your BFF. Step two–you create an operation in TypeScript, step three–you use the universal ORM, and you talk to all your API dependencies as if they were just a database and you had an ORM on top of that. And that it makes the whole process of integrating and combining APIs, it makes it so smooth. You don’t have to think about, okay, is this SOAP, is this REST, is this pure PC? No, it’s really just you have a universal SDK, you just call into the services you need, you massage the data, you pass it back, BFF done.

Jeff Doolittle 00:42:31 Okay. And you’re using the language that you’re used to and you’re using constructs that are common to your language. You’re not having to sort of paradigm shift. Like that’s one of the things that ORMs are meant to do as well, right, is I’m not so much worried about SQL. If I need to, I can go under the hood and I can get there, which anyone who has experience with ORMs know you often have to do. But most of the time I’m a layer of abstraction removed and it just simplifies my interaction. So in a similar way, I don’t need all the gory details of HTTP configurations and GRPC configuration or whatever. I can just say I’m dealing with objects, I’m dealing with classes and types.

Jens Neuse 00:43:06 Yeah.

Jeff Doolittle 00:43:06 And final use case is, API upscaling.

Jens Neuse 00:43:09 Yeah, I think we slightly talked about this before. But it’s kind of like related to the legacy pattern. Where you have your facade, your BFF, you have that BFF, depending on some legacy API. And then over time you replace that with newer services. That’s for example, something that I actually see a lot when people decompose monolithic systems into smaller composable parts. You put a BFF in front of the monolith that keeps the contract intact, and then you, you start strangling the monolith slowly and outsourcing some of its functionality into another service, reconnecting that through the BFF and with that, keeping the facade between your BFF and the client intact.

Jeff Doolittle 00:44:05 Right so in this case, you’d be using API coordination, but not necessarily for some kind of a business workflow. You’d be doing it to keep your legacy system and your new system in sync until the time that you could finally shut off the legacy system and migrate everything, everything over to your new approach.

Jens Neuse 00:44:22 Exactly.

Jeff Doolittle 00:44:23 Yeah. Which is great because we see these headlines like, you know, large fan company ditches microservices, it goes back to the monolith. And of course, if you dig into the details, you realize that there’s more to the story than that. And what I think we’re really struggling with as an industry is finding what’s the right level of granularity. Like, you know, too many things with too many interconnections are not good. And too few things with too few interconnections are not good. And what’s the right answer? Well, it’s somewhere in the middle, but it seems like that’s often what we’re grasping for here. And tools like this would give the ability to experiment even with kind of how you factor the granularity of some of these things. But you can shield your front-end teams from worrying about the details of that, which seems really powerful.

Jens Neuse 00:45:03 Yeah. You know, often when I hear these stories, I feel like we should actually talk more about how to properly structure monoliths, because I usually feel like you can get very, very far with a well-structured monolith. But the problem is if you are not good at software architecture and you’re unhappy with your monolith, why do you think it’s going to be better with the distributed system? Like I don’t know, you’re adding network boundaries and all sorts of complexity. And I don’t want to say that I’m against microservices, not at all. It’s just, I just recently had a conversation with someone who was talking about, they were using Apollo Federation, which is a GraphQL framework to building microservices. And I asked them about their team size and they said they are a team of one, and they had two applications. And so I don’t know, but as a startup, as a small startup, we have 13 engineers. We have one single monolith because we really cannot afford, like, first of all, we all work together. We’re still one team and we just cannot afford the coordination and overhead of maintaining multiple services. Like we just have one very boring backend. And that’s, you know, you built exciting stuff with boring components.

Jeff Doolittle 00:46:36 Well, you said something before in the show, I believe you referred to composable architecture, and that’s the right answer. It’s not microservices versus monoliths. It’s how do we build things that are composable? And if you’re doing that, then you can be somewhere on a spectrum of those things. But you know, I appreciate what you’re saying too about how your team affects these things. And yes, it’s ironic that a single developer team is, you know, managing a full, you know, Apollo Federation or maybe managing multiple Kubernetes clusters with like one developer and it’s like, maybe you would’ve been better off with a monolith and try to find your product market fit without a bunch of additional unnecessary complexity. But hey, I don’t know your context, so, but definitely something to think about.

Jens Neuse 00:47:19 I want to add one thing because you mentioned composable architecture. There’s this recent trend of new startups that do authentication in some way or the other. And what I usually do is I look them up, I go to their docs and I type into the search Open ID Connect, because I’m building an API integration product. And what is amazing about composability and specifications, open specifications like Open Id Connect is if you build a new product, because our service integrates with Open Id Connect, obviously we delegate authentication. And yeah, you build a new product, you just implement the Open Id Connect spec, and you can immediately work with so many other systems and services and it’s just amazing. And so many of them don’t implement Open Id Connect, but build some fancy UI and stuff. I don’t know, I think that’s the wrong direction.

Jeff Doolittle 00:48:22 Yeah. To me it would be like if I’m building the cool new appliance for your home, but it requires a special electrical outlet and you have to call an electrician and put in a new panel and put in, right? And like, here’s the special plug, but this appliance is so awesome. It’s like, wouldn’t you rather just your appliance plug into standard for electrical plugs? Wouldn’t that be better for your business? But again, we don’t know your business, but it’s again, definitely good, good food for thought,

Jens Neuse 00:48:47 Good analogy.

Jeff Doolittle 00:48:48 In the time we have left, talk a little bit, you mentioned Apollo Federation as a contrast, but maybe dig a little, a little bit more into for us, what does Wundergraph specifically do to help support teams that want to build backends for front-ends and how has it maybe differentiated from some of the other vendors and options that are out there?

Jens Neuse 00:49:05 Yeah, so I don’t want to say we invented the idea of API dependency management. It’s not important who invented that. I guess probably some other people have talked about that before. But we’ve built a framework to really facilitate creating BFFs super simple and to give you like a well-rounded package where you can say, okay, I want to build a front-end for that. I want to have a BFF, I can add my API dependencies to my BFF, that can be REST, SOAP, GRPC, anything databases in the future we want to support async API as well. Because I think that’s really exciting use case combining different messaging patterns, sync and async. And then essentially the workflows I described it a bit earlier. You create a BFF, you add API dependencies and you can then define operations against your API dependencies.

Jens Neuse 00:50:06 And from that we generate for one the BFF API service. And second we generate whatever front-end framework you want to integrate that with like a view clients well to whatever. And that it really streamlines everything from building full stack applications. We integrate with integrate S3 for example, for file storage. So that’s just, you know, the mentality what, what you just said with the standardized plug-in Wundergraph, you can say in the front-end code that we generate, you can use the same API to upload a file. And in the backend side you can configure the S3 storage and that can be whatever you have like AWS or digital ocean spaces or menu. And we do the same pattern with authentication. You plug whatever you have there, but the client side, API is login.

Jens Neuse 00:51:01 And so yeah, we really try to abstract away common patterns that we see in the, in the BFF space so that not every developer has to reinvent the wheel. And I think one of the most exciting benefits of this approach and going that route is that you can make the API dependencies of your systems explicit. And something that we’re going to do in the future is that we want to help companies map in real time what depends on what and what uses what servers. And what is super interesting about this use case is we talked about API evolution earlier. One of the huge burdens of evolution and companies especially bigger size to move fast is you have some application that depends on 10 different services, the 10 different services they want to change because it’s not hardware, it’s software.

Jens Neuse 00:52:03 So you want to improve or change those services if you don’t know who depends on your services and you don’t know how they are using your API, how can you ship a change with confidence? And so what we want to bring to companies is first of all, we map out what depends on what we have all these facades so we know exactly, okay, here’s a front-end, it depends on this and that. And then we give backend developers or individual teams the tools to extrapolate if you make a certain change to your API and we can run that in continuous integration. If you make a certain change, it will affect the following services. What should we do? Should we abort that operation? Should we add an extra middleware to keep that compatibility or how should we treat that? And so what we really want to do is embrace APIs as dependencies and give developers the tools to move fast, to change things, but not to break things.

Jeff Doolittle 00:53:10 Yeah. And that’s great and from those patterns we looked at earlier in the show, even as you’re describing that I’m thinking I have now, I have options as a developer, you know, I’m making an API adapter an API lens. Is this an API coordination we’re going to support too, or am I going to do the more like the API evolution details like we talked about? So if I have some tools and patterns that I can follow, then I can say, here’s the pros and cons of these different approaches to solving this problem. Instead of what ends up often happening, which is, well here I’ve got bailing wire and duct tape. Let’s just throw something together and cross our fingers and hope it works. So I like that empowerment piece there of being able to say, these are the tools in my arsenal that I can use to solve these complex problems when things change and let’s be honest, requirements change all the time. So if we try to design systems, that are coupled to requirements, it’s no wonder that we constantly have pain.

Jens Neuse 00:54:02 Yeah.

Jeff Doolittle 00:54:02 So what do you envision is the future of APIs? Where would you like to, like a lot of what you’re describing here, it sounds like this glorious future where you’ve managed our API dependencies and we know how things interrelate and we could do impact analysis on when things are going to change. So where would you like to see the industry go as far as the future of APIs is concerned?

Jens Neuse 00:54:22 So the way we see ourselves currently is we are in the kind of pre-docker era in terms of containers. If we translate that to APIs. So there was an era where we didn’t have docker, we didn’t have containers, and we were just manually installing software dependencies and trying to run that on different platforms. That’s how I see the API landscape today. And what we’re now building or what we have built is, I would say this first step, the dockerization of APIs. So we’ve built our open-source framework to enable API composition and to enable this whole workflow of APIs as dependencies. And the next logical step for us, and that’s how I see the future, is that we build a platform to enable real collaboration between API producers and API consumers. Because what I see today is that a company that like Stripe uploads and open API specification to GitHub.

Jens Neuse 00:55:36 And that’s not really how I see collaboration. The way I see collaboration is that we have a platform where people in the open-source community, they can contribute APIs, we can see what depends on what, how are APIs being used, public and private. We can see how APIs evolve our time. We can discuss and propose changes. We can do all sorts of collaborative things. And most importantly, we can empower developers to continuously change APIs and services without breaking each other. And I would really love to see a future where something happens. Like, you know, you have an API specification, it’s published on what I call the GitHub for APIs. It’s published on that, someone opens an issue, asks for a change because they have a suggestion how to improve that API. At some point later there’s some sort of whole request mechanism or something.

Jens Neuse 00:56:44 This new feature is being added. Once that is closed, there’s a notification that gets published. Everybody who’s consuming that APIs get notified, hey, here’s a new feature, change lock, blah, blah blah. And then everybody else who’s using that API through a standardized BFF, all they do is pull an update on their API dependency. They now have this new feature they can use, for example, the universal SDK integrate that and the whole workflow is done. And now the API provider gets the immediate feedback through analytics because the BFFs, the facades and the API provider, they can coordinate and exchange data. So the API provider would get immediate feedback that people are now using this new capability they exposed. And I think that would just be an amazing future of API collaboration.

Jeff Doolittle 00:57:42 Yeah, I really like that vision of being able to understand my dependencies, understand the impact of potential changes and actual changes, and then being able to have patterns that I can pull off the shelf for accommodating those changes as well. And just the consistency and the clarity that that could help provide the industry seems like a really bright future. So It’s awesome to see that yourself and people like you are working on helping us to solve those kinds of complex problems. So if people want to find out more about what you’re up to, where should they go?

Jens Neuse 00:58:12 So we have Wunder graph.com. This is our main website. You can learn more about what we do there in general. Then obviously on GitHub. So everything we build is open source. You can go to github.com/wundergraph. There you will find a couple of open-source repositories. And then if you want to follow me on Twitter from time to time, I tweet something–it’s JensNeuse_de.

Jeff Doolittle 00:58:40 Great. And also want to remind listeners again about bff-patterns.com, which is also another resource where you can find out more about backends for front-ends. Well, Jens, thank you so much for joining me today on Software Engineering Radio.

Jens Neuse 00:58:53 Thanks for having me.

Jeff Doolittle 00:58:54 This is Jeff Doolittle for Software Engineering Radio. Thanks so much for listening.

[End of Audio]

Tagged as:
Join the discussion

More from this show

Menu

Recent posts