Timothy Beamish of BenchSci discusses React and Next.js, two of today’s most popular front-end frameworks. Host Philip Winston speaks with Beamish about components, routing, JSX, client-side and server-side rendering, single-page applications, automatic code-splitting, image optimization, and more. Beamish also details his experience moving a real-world application to Next.js.
Transcript brought to you by IEEE Software magazine.
This transcript was automatically generated. To suggest improvements in the text, please contact [email protected] and include the episode number and URL.
Philip Winston 00:00:17 Hi, this is Philip Winston for Software Engineering Radio. My guest today is Timothy Beamish. Timothy has been a software developer for more than 20 years with a focus on front-end architecture. He’s currently a staff engineer at BenchSci, overseeing a suite of front-end applications, written in React and Next.js. Prior to BenchSci, he was a senior staff engineer and web team lead at Plenty of Phish, one of the largest online dating sites. Welcome Timothy. Is there anything I left out of your bio?
Timothy Beamish 00:00:45 No, I think that’s good. Yeah, front end mostly focused, dipping into the back end a little bit, and yeah, built tons of tons of websites of over the last few years.
Philip Winston 00:00:53 Great. So let me just set the stage for the show a little bit. So, React is a very popular open source front end library, I’ll say, released by Facebook in 2013. And Next.js is a framework for React released by Vercel in 2016. So, we’re going to divide the show into three parts. First, we’ll talk a little bit about React itself, with an eye towards the following discussions. Then we’ll talk about Next.js and go through a number of features and discuss the advantages and some of the disadvantages of Next.js. And finally, we’ll talk about a recent migration to Next.js that Timothy oversaw at his current company. This part of the show is based on a blog post of Timothy’s called Moving House to Next.js that we’ll put a link into the show notes. I also wanted to mention that Timothy and I both currently work at the same company, although kind of at opposite ends of the company, so we don’t really encounter each other day to day. Before we dive into React, I want to highlight a previous Software Engineering episode that discussed React in pretty good detail: Episode 382, Michael Chan on Learning ReactJS. So, to start off, is React a framework or a set of components or a library? How would you characterize React?
Timothy Beamish 00:02:12 React is a framework for building front-end applications, and it’s important to know that it only does one thing and it does that one thing well. It doesn’t do everything. It doesn’t solve all the problems of the front end. Its job is to take data and represent that as a state via UI. So, it just places a bunch of UI on the screen, which is telling you what your data situation is currently in the front end. As soon as that data changes, the front-end changes, the UI changes. And that’s pretty much all it does. There’s lots of other problems you have to solve on the front end, and React doesn’t solve any of those for you, which is where kind of Next.js comes in to help out a little bit. It helps solve some of the other problems.
Philip Winston 00:02:52 I saw the name React and looked it up, and it said it’s kind of based on reactive programming, which is kind of what you were saying: the ability to handle data updates to otherwise static content. Is that a fair characterization?
Timothy Beamish 00:03:09 Yeah, so in the days before React — in fact in the days of static webpages when the web was pretty boring — you’d go to a webpage, it would render; if you wanted to go get some new information, you’d have to render a whole new page again. Then Ajax came along and that blew up the internet, and it meant that webpages didn’t need to load in entirety to change themselves. They could dynamically shift over time without loading the whole page. Little pieces could reach out and ask questions of servers, get answers back, and re-render just portions of the internet. And this opened up webpages to be dynamic applications, but it also creates a lot of complexity because now you’ve got to manage which parts of the page are updated, what the state of different parts of the page, different parts of the page have to be in different types of state.
Timothy Beamish 00:03:56 You have to have some sort of global management system to handle all that. And we had tools like jQuery to help us kind of paint things on the page, but it didn’t, it didn’t help us organize the data. And then things like Angular and React came along as solutions to this problem. And then for a little while we had the massive front end framework wars, and these days React kind of stood out as the leading way to do this. But basically, its job is to manage the data that’s coming into the application — the global state of the application — and allow you to represent that using UI. So, it’ll figure out which parts of the page is need updating, which parts don’t, and you don’t have to worry about making sure that everything’s in sync. You just make sure that your data’s correct and that you have a good mapping to the UI.
Philip Winston 00:04:42 One React concept is components. Are these visual components you can point to on the screen? Are they more generic than that? What is a React component?
Timothy Beamish 00:04:52 By and large, yeah, they would be visual components, although not always. You can have components that are hidden in the background that don’t actually render anything, but generally speaking, a component is a little piece of UI that gets some data sent to it or it holds onto some internal data. When it gets data sent to it, that’s called a prop or a property. And when it holds onto internal data, that’s called its own internal state. And it’ll render something to represent the props and state that it has. So maybe if the component is meant to represent a button and the button needs to be red in some cases and green in other cases based on some information coming into it, the component’s job is to say, hey, I take a property that tells me whether I’m on or off, and if I’m on, I’m red and if I’m off, I’m green.
Timothy Beamish 00:05:36 And the thing that talks to me only has to know whether I’m on or off. It doesn’t enough to know anything about red or green. That’s my job to handle that on means red and off means green. And I will take care of that. You just make sure that the data flowing into me tells me whether or not I should be on or off. And that’s kind of what a component does. Now you can have — that’s probably like the simplest form of component. There’s a few different types of components and the more, I guess, trickier ones would be things that have to fetch asynchronous data to update the application. So, some components, their job is to say, hey, I need to get some data to render myself property. We don’t have it. I need to go ask a server for some data. So, it has to do,= some sort of asynchronous operation. Then it becomes more complicated. It has to handle things like, hey I’m asking for data now, but I still need to render something so I need to render the fact that I’m asking for data. So, I’ll typically put like a loading spinner on the page.
Philip Winston 00:06:35 So when we say render, in this case with web browsers, we’re talking about emitting like a chunk of HTML that’s going to fit into the page somewhere and then the browser renders it perhaps at some rate.
Timothy Beamish 00:06:49 Yeah, rendering is painting pixels onto the screen, and unfortunately it tends to be a rather expensive operation. So, one of React’s superpowers is that it holds onto a copy of what the UI looks like in the background, and it’s called the shadow DOM. And React is able to update the shadow DOM. The shadow DOM is like a giant tree, so it’ll update a piece of that tree and it knows that the stuff that’s on the screen is one tree; my shadow DOM is another tree. They should be in sync. And I know that because I’ve just updated my shadow DOM, there’s a little piece of it that’s now out of sync with what’s on the screen. I need to repaint just that little piece. So I don’t need to repaint every pixel on the screen. I just repaint that one little piece. That’s a big benefit because, as I said, painting the screen is expensive. So, if I only have to paint a little portion of the screen, I’m doing much better. And that’s kind of where React superpower comes in. So, it uses the shadow DOM to figure out what parts of the real screen do I need to paint? And then it does the process of rendering, which is like you said, putting pixels in the right color in the right place on a screen.
Philip Winston 00:07:54 And the DOM is the Document Object Model that’s kind of like the HTML tree?
Timothy Beamish 00:08:01 Yes, exactly. So, if you crack open any webpage in Chrome, you can hit F12 and go into your developer tools and look at how a webpage is built. It is in a document, which is structured in what’s called a DOM, which stands for Document Object Model. And it’s basically a tree. So, it starts at the very root and the very first tag will be your HTML tag. And then that tag will have, that’s the root of your tree. It’ll have some children. Typically, the first two children that it has is a head child and a body child, although it doesn’t necessarily have to have that. You could have some more things. And then in your head child, there’s information that represents kind of metadata about the document. And then in your body child is kind of the representation of what you see on the screen.
timothy Beamish 00:08:43 And so, that’s all split up into HTML tags and those tags organize the structure of the document. And then CSS is — cascading style sheets — is the technology that decides what that structure is meant to look like in terms of what colors and formatting and things. And React organizes all of that for you. So instead of thinking about it in terms of HTML elements and CSS tags on things and CSS attributes on things, you think of it in terms of components, which are building blocks that still live in a hierarchical tree-like structure and they have styling glued into them, and they get represented in the ethereal world of React. They get updated in the shadow DOM and then React says, hey, my shadow DOM’s changed, I now have to go repaint different HTML on the front end.
Philip Winston 00:10:56 Since React came out of Facebook, I think, I would imagine one of the benefits of components is being able to have a bigger team work on a single application, but maybe have components that are reused. Is that part of the benefit of React that it can enable this kind of reuse?
Timothy Beamish 00:11:13 Yes, absolutely. You’ve hit on a great point. So, in the old days of HTML, if you had a button on the top of the screen and a button on the bottom of the screen and they looked the same but they did different things, you’d have to write that code out twice. With React, you would make a single button component, which then you can use all over your site, and that single button component — I mean you want it to do different things in different situations. So, you would pass in some property to an instance of that button to say, I want you to execute this function when you get clicked. And this other one I’m going to create you look exactly like the other one. I’m going to reuse this component, but I’m going to pass in another function to you to say that, when you get clicked you do something slightly different. So yeah, it allows for tremendous reuse, which helps you organize your code base, makes you far more efficient, and helps teams work together much more efficiently.
Philip Winston 00:12:07 Let’s move on to Next.js and we can still reference React features to compare them to Next.js or just sort of go between the two, but let’s talk about Next.js and what it adds. I’m going to start with this data fetching and client-side rendering and server-side rendering. Just kind of comparing those and which does Next.js favor or which does it encourage, or which styles are common?
Timothy Beamish 00:12:34 Okay, so Next.js doesn’t necessarily solve the problem of data fetching for you. So, this is a pretty big problem in the React ecosystem. Like I said, React is good at you give it some data and it renders it to the screen. But how do you get the data in the first place? React doesn’t have an opinion on how you do that, there’s lots of different ways to do that. Next.js doesn’t necessarily solve that problem either. So, there’s lots of different ways to do data fetching. And typically, data fetching is when your software needs to know something so it reaches out to an external server — usually an API server — asks it a question via an HTTPS request and gets back some data. And this is kind of the principle of Ajax. This is what kind of turned Web 1.0 into Web 2.0. And there’s lots of different ways to do it in React.
Timothy Beamish 00:13:19 Probably the most popular way is a library called Axios, which it’s an older library, but it’s bulletproof. It works in more browsers, it has tons of features. The new way — the future — is using an API called Fetch, the Fetch API, which is built into the browsers now. But more people that I’ve found are still using Axios because it still is a little more feature rich and works in the weirdest of places. It’s been around a little longer. The old school way of doing data fetching is actually making XHR requests, which is old and convoluted and we now have lots of frameworks built on top of that. So, you don’t have to deal with actually like the guts of Ajax, which is building XHR stuff. So, the newest coolest way of making data fetching requests, outside of Axios and Fetch, is a framework that kind of encompasses both of those.
Timothy Beamish 00:14:11 At BenchSci we use something called React Query. React Query is a library built for React that is meant to handle your data fetching stuff in a React context. So it uses this concept of hooks, which is a newer React mechanism, where a component will call a series of hooks — one or more or zero — before it does its rendering. Those hooks are meant to perform procedures, and in one of those procedures can invoke React Query, which has a clever caching system built into it and knows if I can’t find something in my cache then I need to reach out to an API server to get the information I need, and then I can send that back. So, the output of the hook when you always call the hook when you render the component, the output of the hook is the contents of React Query, which is either going to be go away, I’m loading, or here is some data for you, or there’s been an error happening. And so, your component then either has the data it needs, it knows what to do or it can render a loading state or it can render an error state. So, I should mention React Query sits on top of the Axios system or the Fetch system, or if you really are crazy you could do it with the old school XHR Ajax system, and most people use Axios as far as I’ve seen.
Philip Winston 00:15:33 So, let’s talk for a little bit just about where the need for another framework came from or how Next.js came to be — or how you came to discover Next.js — and kind of just from a high level, what sort of benefits does it provide? I think the data fetching is an example of, there’s a lot of choices there and a lot of options, but just from like a high level, if I were considering using Next.js, what aspects of my project would steer me in that direction?
Timothy Beamish 00:16:02 So as I’ve said, React only solves one part of the front-end problem scape, and it does a good job of that. Next.js comes along to offer you some opinions on how to set up a React application, and so it kind of makes things a little more rigid and concrete for you. So, one of the things that it provides out of the box, which is pretty nice, is a nice routing system that’s really easy to kind of understand and comprehend. Routing is the relationship between what is the address in your URL, like in your browser, in that URL bar, what is that address in relation to what is on the page? So, we’ve now moved to single page applications on the internet, which means when you load up a website and you traverse through that website page by page by page, you’re not actually, in a lot of cases, making full requests to get back full webpages and render those.
Timothy Beamish 00:16:53 What’s happening is you are just saying, hey I want to travel through this website and my state is just going to update and change as I’m traveling from place to place in this website. The route needs to reflect that I am now at this point in the website, now I’m at this point. I need to have a route that reflects that so that if I want to leave this website and then come back to this one part of it, I have a way of getting there, which is what the route is. I can bookmark that route, I can save it, and I can call up this website and get to that state again. So, routes have now changed because in a single-page application, you’re not necessarily downloading a whole new page, you are downloading the first page and then you are transitioning your state to be where it needs to be to be aligned with whatever the route is. Does that make sense?
Philip Winston 00:17:44 Yes, I think so. So, at one point the URL corresponded strictly to sub directories in a file system and if we’re running a single-page application, that’s no longer true, but the different components of the URL still need to point to something, but maybe they’re point to something within the application.
Timothy Beamish 00:18:02 You got it. Right on. So, what Next.js does is it helps solve the complexity of the routing issue. Previous to Next.js, I think the most popular library would be React Router, which enables you to build a routing system in your application, and it’s usually one of the first things you do. So yeah, I’m going to create a React App and the very first thing in your React App is like your top-level component, and in there you will stick your routing system which will say, okay, I’m the very first component that loads is my outer shell — it’s usually called the app component. And then somewhere pretty soon after that I’m going to define all my routes. So Next.js this works since NICE. Next.js says, hey we’re going to offer you something a little different. We’re going to, because you’re used to file folder structures being represented as routes, we’re going to offer that back to you.
Timothy Beamish 00:18:55 So Next.js gives you the ability to structure your Next.js app. In your file folder structure, you can have a folder for all of your different routes, and your routes follow whatever that file folder structure is. So that you end up in your file folder structure: if you have, I’m at sub directory A/B/C then that means that the page, my website/A/B/C will be an actual page, and you can stick a page component in that folder structure. Next.js will look at that file folder structure, it’ll extract out the hierarchy of it, and it’ll turn it into a set of React routes for you, and it’ll paste it into the top level of your app. So, you don’t have to worry about setting up routes programmatically anymore. You just do it using your file folder structure.
Philip Winston 00:19:40 So we’re talking about single-page applications, but then in this case, does that imply it’s multiple pages when you’re talking about the sub directory corresponding to the route? I guess just trying to understand, is single-page application the most common way, or is it a required way to do sort of Next.js applications?
Timothy Beamish 00:20:03 Okay, this is interesting because in a single page application, technically there is only one web page. But humans like to organize things as when I click on a button and I go somewhere else or I go somewhere else. In actuality what’s happening is your state in the application is changing. You’re not necessarily loading up a brand-new webpage, you’re still on the same webpage, but you’re seeing a completely different state. And to help humans kind of resolve this, we use the terminology, hey you’ve gone to a new page in the website, but that’s not actually true. You haven’t downloaded a brand-new page; you’ve just updated the state in that page. So, in the olden days, yeah sure you’d have to, if you were to go to A/B/C, you would be downloading the C page, and if you were to go to A/B, you’re downloading the B page. That’s not true here. So yes, we are always on the same page. A single-page application is only over one page. It has to update its state to get to the right state, which is aligned with the route that you’re on, the URL that you’re on, and all that is to help humans kind of resolve the fact that they think they’re moving around to different pages when in actuality they’re not. It’s just a state update happening in a single-page React app.
Philip Winston 00:21:21 Maybe this is related, but I came across a concept called dynamic routes. Is that building on this, or is that a different type of route?
Timothy Beamish 00:21:30 Sometimes let’s say I want to go to A/some ID/C. So, let’s say we’re looking at a shopping cart or some sort of shopping site, and you want to look at all the products in the winter section. So, maybe you want the URL to be products/winter/list and that will list all the products in the winter section of this department store that you’re looking at. But maybe they have a summer section as well. So maybe you want to go to products/summer/list and you’ll get all the summer products. Now the list of summer products and the list of winter products, you could abstract them to say that they’re going to be really similar, like it’s a list of things and some of them are summer in one case and some of them are winter. So the fact that that you could flip summer and winter, so summer and winter becomes this kind of dynamic piece. In a React App you could say that we’re actually going to render the exact same pages and exact same components, and all we need to do is change one little piece of data in the back to say, hey database get me my winter products and I want to list those or get me my summer products and I want to list those.
Timothy Beamish 00:22:35 And so, in the route up in the URL, the dynamic piece is the variable that represents winter and/or summer. In Next.js, you can do that by updating your file folder structure and using a special piece of syntax, just chucking some square brackets around a folder to indicate that that folder is going to have a dynamic property about it. And so, Next.js can easily say, oh this is a dynamic folder, I’m going to turn this keyword that you used inside those square brackets, the name of your folder, I’m going to turn that into a variable and I’m going to feed it into whatever component is downstream from this folder. So, when you load that component up, it’s going to get a variable — let’s call it season. So maybe, maybe you’ve named that folder square bracket season ending square bracket, then your component’s going to get a season variable coming in and you can assume the value of season is going to be either winter or summer. And if it’s not you can throw up an error saying you’ve picked the wrong season. And if it’s winter, then the React component downstream from that is going to ask the API, give me all the winter products; if not, give me all the summer products and then it’s going to generically do the same thing. All I’m doing is asking for a set of products and I want to then render them to the screen.
Philip Winston 00:23:47 So with season there’s only four seasons, but I think what you’re saying is that dynamic portion of the route could correspond to anything in the backend or in the database, and there may be hundreds of flavors to deliver. And so yeah, representing that as static pages would be impossible, I guess. So that seems like a big advantage.
Timothy Beamish 00:24:07 Yes, you’re right there. I tried to give you a trivial example with seasons and yeah there’s only four of them, but yeah you could, you could scale that up to names of people or something which could be theoretically infinite.
Philip Winston 00:24:19 Okay, so we talked about routing and how Next.js does that differently than React by itself. Another concept we’ve talked about rendering, but is the difference between client-side and server-side rendering and what is Next.js providing there, or what is its opinion on which is the better approach or which is which approach should I consider, in which case?
Timothy Beamish 00:24:41 There’s a lot of interesting things to touch on here. First, I think we should probably define client-side rendering versus server-side rendering. And then there’s a third one which is called static rendering, also called pre-rendering. So, there’s three kinds of ways to render an App. So, I’ll first just touch on what each of those are, and then I’ll go into the benefits and drawbacks of each of them. So, client-side rendering is kind of the very first in the evolution of single-page applications. Client-side rendering was the first way of doing things. By no means is it perfect — like, it presents some problems — and then server-side rendering came along to say, hey, then we think we can solve those. And then, static rendering came along later to say, hey, I think we can solve the problems of server-side rendering. And none of these things are kind of perfect.
Timothy Beamish 00:28:13 It doesn’t exist on the server, there’s no window object because there’s no browser on the server. When you’re, when you’re rendering, when you’re doing server-side rendering, you’re doing an inside of Node.js, there is no browser. So typically, you have to have a lot of if-else statements in your code base to say like, Hey, if I’m on the client and then I can access window and do this thing, else if I’m on the server, then I need to access the file system and do this thing to get the piece of data that I want. So, it ends up making your application kind of messy. And so, I should touch on the third thing, which is pre-rendering or static rendering. And that’s when you take every permutation of every piece of state that your application could come up with on the backend, and you render it before any requests show up on your web server at all.
Philip Winston 00:29:49 Let’s hear that, although I want to be clear of these three approaches, client-side, server-side, and I guess static, which is Next.js enabling. Can you do all three with React alone or is which is Next.js providing, I guess?
Timothy Beamish 00:33:15 So now these couple hundred thousand requests are waiting because there’s only one Node instance that can only do one thing at a time. This is a big scaling problem. So, to get around that, you can scale up Node: you can run one Node per core of CPU that you have on your web server. So, you can throw money at the problem and just increase the number of web servers you have, increase the number of cores on the web servers you have, but you end up getting into round-robin problems. So, to do that properly you have to have some sort of scheduler built into your web server system saying, okay, I need to find a Node instance that’s free and I’m going to send this request to that Node instance and then the next request I’m going to send to the next free Node instance that I have.
Timothy Beamish 00:34:37 We’re not Node experts, so we don’t know how to scale Node properly in order to fulfill the needs of a high-traffic website. At the end of the day we went with client site rendering. I’m happy about that decision and I kind of, anytime anyone suggests server-side rendering, I kind of go, ah, it’s maybe not such the greatest thing. I mean, I know it’s sold to you as like the savior of all the problems that client-site rendering, but it comes with a whole bunch more problems. So, I’m going to stick to client-side rendering. And I will do pre-rendering, but that only works for cases where you’re not going to have like a very interactive dynamic page. So, pre-rendering’s great for blogs — something like the New York Times or something — if they’re just printing up news articles, they just need to render that news article once, there’s nothing on that page it’s going to change. So, you just render it once using a React system and then you can serve that page statically. That’s a great solution for those sorts of things, but it won’t work in like a chat application.
Philip Winston 00:37:38 As long as we’re talking about optimizations related to performance, there’s another Next.js feature related to image optimization. Is that something you make use of?
Timothy Beamish 00:37:47 Yeah, that’s something. So, when we flipped our application over to the Next.js, one of the exciting things was making use of the Next.js image tag, which offers a lazy loading feature for you. So, images are generally also expensive things on webpages, and like the goal of a web developer is you’re trying to figure out how do I give you the biggest awesome experience for the least amount of bites sent over the wire. So, images give you a big bang for your buck, like they look great, but they do costs a lot. And so, you want to load them lazily, which basically means if I’ve got a whole webpage on the screen, the bits off the screen is known as below the fold. So, if there’s any images below the fold, I don’t need to actually download them unless I scroll them into view. Or if I get close to scrolling them into view, then maybe it’s a good idea to start downloading them.
Timothy Beamish 00:38:37 Let’s say I have a massive page that’s really, really long and most users never ever get to the very bottom of it. You don’t need to load that image that’s hidden on the very bottom. There’s only like 2% of your users that are ever going to get down there. And so lazy loading solves that problem. So those 2% of the users, they will scroll down as the browser approaches getting to that image, Next.js will say okay, we need to download this image now. So, it basically means you only download the stuff that you are going to look at and see, which is a big savings. Next.js also is able to take your images and size them up accordingly for the screen size that you’re on. So potentially you’ll have people coming in on mobile phones, people coming in on desktops, they’re going to want different sized images. Next.js will handle that for you and make sure that you are downloading the appropriate one so that it fits into the right place.
Philip Winston 00:39:28 Before we move on to your migration story, one last Next.js question. I’m just wondering if there are any developer quality of life features that Next.js, in particular, provides that just make sort of the day-to-day development easier? I guess one I have listed here is a fast refresh. Is that different from React itself?
Timothy Beamish 00:39:48 Oh yeah. Okay. , I could talk about that. So, in terms of the developer’s quality of life, I would say Next.js gives you a nice way to organize a React application — a nice standard way to organize a React application. So, if I have a Next.js background and I come to your Next.js app, I’m going to know where to find things. There’s kind of some opinions on where things are stored, like all the routes are stored in a pages directory, I know where to find configuration stuff, so how to configure the website itself. And one other really fantastic thing that Next.js does well is called hot module reloading, or hot swapping, I think it’s also called. But basically, when I’m developing a React application locally on my own computer, I generally have the code in my VS code IDE or WebStorm or whatever, and then I have the site running live in a browser. nd as I make changes to the code, I want to be able to re-render that in the browser really quickly to make sure that I’m coding properly.
Timothy Beamish 00:40:43 Hot Module reloading is a promise that React gave to the world when React initially came out, but it never worked well — or I mean it worked for a little while and then it broke, and it became really frustrating. And basically, what you want to happen is you want to be able to load up your React application in development mode so that it’s talking to a special little server in your development environment which is mapped to the code in your IDE, SO the second you change code in your IDE and you hit save, boom, the React App goes, hey, something’s changed, I’m going to re-render myself. And so, you get that instant gratification of oh I changed something in the code, I see it in the React App, hooray! That’s called hot module reloading. You don’t have to rebuild your whole App manually and relaunch the App to see the change. It just happens right there for you.
Timothy Beamish 00:41:24 This came with React when React first came out, and it always notoriously broke in the first few days of starting up a new React application. And people were always scratching their heads, why did my hot module reloading stop working? And it was usually because you installed some module that did something funky and it broke hot module reloading, and I spent so many hours chasing down why does hot modular reloading stop working in our application? Next.js does, I don’t have to think about it anymore. it’s just works, and I can install modules all day long. And I don’t know what secret sauce they’ve done in there, but they’ve done something magical that just makes hot modular reloading working. It’s a huge headache when it doesn’t work because it’s such a glorious feature for developers to have. So that’s definitely a bonus of Next.
Philip Winston 00:42:06 Yeah, in my career I’ve had the gamut between essentially hot module reloading even for a desktop application to the opposite where it’s extensive restart time each time you make a change. And it’s certainly one of those things you really enjoy when it’s a short edit-test loop. It’s really nice.
Timothy Beamish 00:42:27 Yeah.
Philip Winston 00:42:28 Okay, let’s move on to your migration story. You published a blog post — Moving House to Next.js in October 2022, and we’ll link to that in the show notes — but let’s talk first about what application was this in general terms? What was the scale of it, what did the application do as far as databases or images, or what was sort of the requirements? And then what limitations you were bumping up against when it was React only, and kind of what pushed you over the edge to make the move to Next.js?
Timothy Beamish 00:45:30 So that was the goal of the migration situation. Before we had done this, we basically had two apps. We had an Next.js app, and we had the Selector application. And we had to use an express server so that when you come hit our website, depending on which page you went to, you would get one of two single-page applications. You’d either get the Selector app or you’d get the Next.js App. And this of course is not optimal because as you’re moving through the site, you could stumble upon a brand-new single-page application. Loading up a single-page application. The worst part about it is the initial load of the application, the first three to five seconds of loading that application up. Now a second on the Internet’s an eternity. So, you want to minimize those. And so having to go, as you’re moving through a website and you hit a brand-new single page application and you have to load up a whole thing from scratch, that’s painful.
Timothy Beamish 00:46:23 So it’s not a great user experience. We had to use an express server to decide, you’re going to this route. Okay, well actually that means you no longer are in this single-page obligation. You now need to be over in this one over here. These are all pain points for our users. And we wanted to remove all of that. We wanted to make the site feel a lot snappier and faster, and we didn’t want to have a Frankenstein experience of some of the stylings and components and ways of doing things in one application don’t look a hundred percent the same as they do in the new applications. So, as you’re moving back and forth, you’ll notice, oh, this button has square corners, but this one has round corners, that’s odd, jarring. So, you want to be consistent across your application. So, there’s another reason to do the migration.
Philip Winston 00:47:04 I was going to ask about the express server. Is that essentially routing but at a higher level than within an application? Or what is the express server?
Philip Winston 00:49:00 So, this may or may not be related, but I think you mentioned LaunchDarkly in your post, which is a feature flag management system I guess. Did that play a role in this migration in terms of being able to selectively make these kinds of decisions or selectively turn on functionality?
Timothy Beamish 00:49:17 Yeah, LaunchDarkly is, it’s typically a split testing system so that you can actually launch code into your application without it being turned on for all of your users, so just some of your users get to see new code. And typically people will use it when they’re rolling out brand new features and they want to figure out, I want to present this feature to just a few users, not all of them. And I’m going to analyze how those users behave on this new feature before I decide to roll it out to even more users. You could also use LaunchDarkly as a feature flagging system. So, you could essentially turn it into something that says you are a user of this type of role, therefore you get these sets of features. We’re going to orchestrate all of that inside of LaunchDarkly and enable you to only see certain things on your website. You could use it in a blog if you have content that only certain paid users wanted access to, you could potentially use LaunchDarkly to determine do you have the feature flag that enables you to see this content or not.
Philip Winston 00:50:17 So in terms of things that changed, switching this flagship application to Next.js, did routing change or what was your method of doing routing before and what did you change to?
Timothy Beamish 00:50:27 Yeah, the old Legacy application had React router installed in it. So, the routes were programmatically inserted into the application. Next.js handles that piece for you. So, you don’t have to programmatically put them in your App, you just have to set up the right file folder structure. And so that’s what we had to do. We had to make sure that all of the routes represented in the React router — and they were a little convoluted because you can have routes inside of routes, and you can have components that are introducing new routes. And the old application kind of had a bit of a mess there where it wasn’t usually when you set up a React router, I’ve seen it done where it’s all kind of done in one file and all your routes are defined in one place. But our legacy application didn’t have that.
Timothy Beamish 00:51:06 I don’t know why. I wasn’t around when whoever decided to do that made that decision, but it’s a little tricky to figure out what are all the possible routes that this App accepts and can render. We had to do a little deep dive to figure out what they were. Once we figured them all out, we then had to transfer them over to Next.js and place them inside the pages folder. So, every Next.js app has a pages folder, and underneath there are all your subfolders that map perfectly to your routes. And one of the things that I like doing for a Next.js application is I like to just make sure that that pages folder only really contains routing logic. And so, when it finally gets to like, hey, you’re going to A/B/C, so C needs to render a component, I call that a scene, not a page.
Timothy Beamish 00:51:50 And the reason why I call it a scene is because I used to work on an application with a bunch of Android developers and their terminology is scenes, so they don’t talk about pages. Android apps don’t have pages, they have scenes. And because I’m also building an application but on the web I said okay, I’ll adopt that terminology. I’ll call it scenes. Granted though, in Next.js it’s under a pages folder. I’ve just realized that now as I’m looking at the folder structure of Next.js. So anyway, the point is, the thing I want to mention is that in that pages folder in Next.js, I feel like you should only really have routing logic in there. When you get down to rendering the actual C — the A/B/C rendering — the actual C, what you want to do, what I like doing is saying, hey, I want you to render the C scene — or I guess I should call it page now that I’m looking at how Next.js names things.
Timothy Beamish 00:52:35 I want you to just render the C scene, and I’m not going to actually place that page rendering logic inside the pages folder. It’s going to import that scene from our global components folder. So generally, a React application will have a whole system of components usually organized in a nice global components folder. And I would make a scenes folder up there or a pages folder up there so that all of my rendering logic is in one place and all of my routing logic is in another place. So that’s just a nice way of kind of separating the different folders and what they’re meant to do inside of Next.js.
Philip Winston 00:53:11 Getting into some logistics, how did you divide up the team to work on this transition versus maybe continual work on the new applications, and how did you make the actual transition? Presumably it had to be done in one step or at least one day.
Timothy Beamish 00:53:27 Yes. Yeah, that was, so we have to realize that we’re taking a legacy application and we’re merging it into a new application, and both of those applications are being worked on constantly by a whole bunch of developers all over the place. So, we carved out a team to do the migration, and there was like four or five of us I think, and that was our, our migration team. And the first thing we did was we set up a bash script, or maybe it was a Node script, that in our branch it performed the migration. It took — actually, we probably did it as a Node script now that I think about it because there was a lot of details that went into doing that. But we basically just did a like a copy-paste-move, move files around. Actually, we had to use get-move. You don’t just copy-paste files around, you actually have to get-move them.
Timothy Beamish 00:54:08 The reason why you need to do that is because when you’ve finally done the merge, like on the day that you’re actually going to do it and you finally do the merge and you push that back up to your repo, there’s going to be developers out there that are still working on old branches in the old world, and as soon as you make your merge, all their files aren’t going to be there anymore. They’re going to have moved over to somewhere else. So you have to use a get-move operation to let them know that when they re-base and when they pull down the comments of the main branch, aka the migration, when they merge that into their old long-running branch get is going to say, hey, that file you’re working on over there, I’m clever enough to know that it’s moved over to here. Here’s where you go to find it.
Timothy Beamish 00:54:47 So we’re not going to lose the development work from those people. So, we had to incorporate that into a Node script which we ran, and it was sort of like our dress rehearsal script. We would perform that, it would make the move and then we would go in and see okay, what’s working and what’s not working. If we’ve taken all the code, we’ve moved it across, it’ll be amazing if it just loads up perfectly on the first day; it didn’t so, but it was pretty close like you’re taking React and you’re putting it inside Next.js which is also React. So, there’s a good chance that everything’s going to line up. Some of the things that you, we needed to work on. Weíre transferring our image tags over to the Next.js images, transferring the way we used to do linking in the old App to the Next.js links, which are a little more clever, and then some configuration stuff.
Timothy Beamish 00:55:30 So sometimes links, like I said, would be shouting out to an Express server to figure out do I need to actually switch to a single-page application? We don’t need to do that anymore. So, we would have to add some stuff in this dress rehearsal script that would say, okay, I’m going to rip that out and put it somewhere else. Your runtime configuration sort of stuff. So how you inject configuration into the app depending on what environment you’re on. When we build our application, it runs in our local development environment, then it goes into what we call mainline, which is like a development environment where everyone’s changes meet each other for the first time, and we find out what happens; and then from there it moves to a QA environment, and then to a staging environment, and then finally to our production environment. And for each of those environments you need to inject a bunch of configurations into the application that was done differently for the two different Apps. And so, we had to migrate those and merge them together so that they’re working properly.
Philip Winston 00:56:20 I think the post might have mentioned public runtime config. Is that a type of configuration for Next.js or for?
Timothy Beamish 00:56:27 Yeah, that’s a mechanism for adding configuration to a Next.js app. There’s a couple different types of configurations. There’s configuration that you statically build into the application at build time, but you want to build your React application once and then deploy it to different environments and have it behave differently on those environments because it’s in that environment. And to do that you need to inject public runtime configuration into it. So, it’s saying, I’ve built the code, I’m not going to change the contents of the code anymore. But now that you’re on the QA environment, you need to talk to this server instead of that server. And you do that through public runtime config.
Philip Winston 00:57:04 One of the last sections in your blog post I think is called Moving Day. How did that go when the final transition was made?
Timothy Beamish 00:57:13 Yeah, it was great. And it was stunningly great because we took a ton of code — like thousands and thousands and thousands of lines of code that’s still being actively worked on — and we moved it from A to B, and there were no major blowups. Like, we’d gone through the dress rehearsal script so many times and we’d done that move, and we created a special QA environment just for this move. So, we would do the move, we’d take that branch that had everything merged together, we’d push it up to this new QA environment, and we went through it with a ton of manual QA folks and a lot of developers went through there like looking for different bugs. And then whenever we found one, we would incorporate that back into our migration script until we got to a point where we’re like, yep, this is working fine.
Timothy Beamish 00:57:58 The brand new, like the QA server that we have it running on, we can’t seem to find any bugs so let’s go for it. And so, we picked a date, we announced to the whole company, okay, we’re going to do the migration. It’s probably going to take us, you know, 10 minutes to like do the merge and then we need to deploy it, which takes another 30 minutes and then we’d like to do a sanity check for like an hour or two just to make sure that nothing’s crazy and then you can go back to work. So, we kind of picked a day, we told all the developers just take a couple hours and do something else for a second and we’re going to deploy this and see what happens. And it went great. Like there were no major bugs. I was really stunned. I was like, wow, this, we did such a good job at building this nice dress rehearsal script and the whole flow of putting this up on a QA server and testing it out before we unleashed it upon all our developers. And so yeah, it was a big success, I’d say.
Philip Winston 00:58:42 What were the concrete advantages that most surprised you or that you were just most impressed with in terms of the transition, as far as how the users perceived the application, maybe performance or otherwise, or as far as developer and continued development? Basically, what was the net result here for you guys?
Timothy Beamish 00:59:00 Well, clearly like one of the big advantages of doing this was that you don’t have this Frankenstein issue of jumping between two different apps. And so, you have this like loading problem that happens as you’re bouncing back and forth and in our suite of applications you would typically bounce back and forth between these two things a lot. And so, having that jarring experience was gone now. So, the app felt a lot faster. The other really big thing, and this I think was the major win, is our TTI went down by a lot. TTI is time-to-interactive. It’s what I think is one of the most important metrics in any single page web application that you’re building. And it’s the time that it takes for the first bite of that React application to touch your browser until the application becomes interactive and you can actually click and do things on it.
Timothy Beamish 00:59:42 You’re aiming for that to be between three and five seconds. And I think in the old days ours was up to like seven or eight seconds, or something. But anyway, when we finally got everything moved over to Next.js because of code splitting, automatic code splitting — that was the big thing: our legacy app didn’t have any code splitting in it because no one bothered to kind of figure out, it’s a hard problem to solve and no one bothered to figure it out. We got it for free in Next.js when we moved everything over to Next.js. We saw like a 40% improvement in our TTI, which is actually a decrease because you want TTI to be low. And that to me was like the major win for us in that your application loads a lot faster, which gives you just a tremendous boost to your usability of the application.
Philip Winston 01:00:22 And going forward, I’m guessing it’s an advantage to have one style of applications, so that’s going to make development easier.
Timothy Beamish 01:00:29 Yes, absolutely. So now we can be more consistent in our components. We can share components. So, the legacy application can now use components from the new application. It’s all just one application now. So, there’s a much better ability to be sharing things back and forth. We’re going to be a lot more consistent. Since this, we’ve now developed a nice design system that allows us to make sure that all the facets of the application are using components out of the design system so we remain nice and consistent. And so now there’s this nice uniform experience across everything that BenchSci is providing.
Philip Winston 01:01:00 Let’s start wrapping up. Can you speak a little bit about the Next.js community, the documentation, resources online, how you feel that is, especially for someone new to Next.js?
Timothy Beamish 01:01:13 Yeah, I think Next.js is a very successful and popular React framework, probably the most or one of the most successful React frameworks out there. And that’s definitely due to documentation. You don’t get to a good place in developer’s hearts without a great documentation system. There’s an amazing Next.js developer community. That’s also a big attraction to why you would want to use it. When you enter into a new framework, you want to be able to know at some point I’m going to get stuck. How do I get unstuck? What are the best ways of getting unstuck? And the community for Next.js, the documentation totally help with making you feel like, yeah, this is a safe move, let’s do it.
Philip Winston 01:01:48 How about for someone starting on a Greenfield brand-new application, are they taking on more to start with Next.js, or is that going to simplify it if they use it from the beginning?
Timothy Beamish 01:01:59 So in the olden days, it used to be this thing called Create React App, which nobody’s using anymore, but it used to be the standard like, oh, you want to make a React App, just download Create React App and get that running and that will get you a React App up and running nice and fast. I would never recommend that anymore, and I would totally recommend Next.js. So now you really just have to do like an MPM install Next and it’ll load up an application for you and everything just works and it’ll give you a nice front page, which of course you’re going to change to suit your own needs. But it’s all there, great documentation to show you how to put things together and move around. So yeah, Next.js is an awesome first stepping stone for someone who wants to make their first Next.js App. It’ll help remove a lot of the difficult, confusing parts.
Philip Winston 01:02:40 That sounds great. Thanks so much for sharing all this with us. Is there anything you’d like to say about where to follow you or learn more about any of this online and we’ll put any links in the show notes?
Timothy Beamish 01:02:52 Yeah, I mean you can find me on Twitter @timothybeamish, and of course the link to the blog post is in there. There’s some more details of how we did things. Feel free to reach out to me on LinkedIn or on Twitter. If you have any questions about doing something like this, I’m happy to help.
Timothy Beamish 01:03:07
Philip Winston 01:03:08 Thanks so much for talking to me today. Yeah.
Timothy Beamish 01:03:08 Thanks. It’s been great.
Philip Winston 01:03:09 This is Philip Winston for Software Engineering Radio. Thanks for listening. [End of Audio]