Vivek Yadav, an engineering manager at the payment company Stripe, speaks with host Adi Narayan about building regression testing for microservices. Drawing on his work estimating network costs for card payments, Vivek explains why unit and integration tests fall short when a single transaction’s cost can depend on 100+ parameters across card networks, countries, and merchant types. He describes Stripe’s approach: structuring services so that core logic is stateless and separated from I/O, then wrapping that logic in an Apache Spark job that replays historical transactions from cold storage to compare old versus new outputs at a granular, per-transaction level. This allows Stripe to validate major migrations with far more confidence and speed than a traditional shadow-rollout, and it now doubles as a tool for assessing network rule changes, debugging incidents, and even showing customers how a pricing change will affect their costs. The episode closes by noting the same pattern applies wherever AI-driven code changes need fast, reliable evals.
Brought to you by IEEE Computer Society and IEEE Software magazine.
Show Notes
Related Episodes
- SE Radio 595: Llewelyn Falco on Approval Testing
- SE Radio 685: Will Wilson on Deterministic Simulation Testing
- SE Radio 632: Goran Petrovic on Mutation Testing at Google
- SE Radio 164: Agile Testing with Lisa Crispin
- SE Radio 366: Test Automation with Arnon Axelrod
- SE Radio 498: James Socol on Continuous Integration and Continuous Delivery (CICD)
- SE Radio 370: Chris Richardson on Microservice Patterns
- SE Radio 213: James Lewis on Microservices
- SE Radio 495: Vaughn Vernon on Strategic Monoliths and Microservices
- SE Radio 351: Bernd Rücker on Orchestrating Microservices with Workflow Management
Articles and Resources
- Stripe blog: Scaling up your microservice testing with Apache Spark – part 1
- Stripe blog: Scaling up your microservice testing with Apache Spark – part 2
- Apache Spark
- List of popular Apache spark packages
Transcript
Transcript brought to you by IEEE Software magazine.
This transcript was automatically generated. To suggest improvements in the text, please contact [email protected] and include the episode number and URL.
Adi Narayan 00:00:18 Welcome to Software Engineering Radio, I’m your host Adi Narayan and I’m here today with Vivek Yadav and we’re talking about building regression test harnesses for microservices at scale. Vivek is an engineering manager at the payments company Stripe, where he spent nearly a decade building payment, cost estimation, building and pricing infrastructure. In short systems that estimate what it costs to move money on the internet. Vivek, welcome to Software Engineering Radio.
Vivek Yadav 00:00:44 Thanks for having me.
Adi Narayan 00:00:45 Is there anything about your background that I missed or mention that you’d to add?
Vivek Yadav 00:00:49 Just one more thing to add there that yes, before Stripe I was working on a bunch of startups and the last startup I worked on was also payments related and it still is doing pretty good.
Adi Narayan 00:00:59 Wonderful. I’m really looking forward to this conversation. There’s a lot of interesting stuff that I’m looking forward to cover. So, let’s get started with just the basics. What are the different kinds of tests that engineers often write for microservices?
Vivek Yadav 00:01:10 Yeah, so they come in a full suite of tests. So, first one would be basically your unit tests, which are really just trying to test your basic functionality. A function level functionality essentially then comes the functional test which go a bit broader and just are trying to test some functionality at a class level or a couple of classes level and so on. And from there on you have integration tests which try to test across more complicated boundaries across different modules or across different services and so on. And beyond that it is much less formalized but different companies or different teams might come up with unique novel testing strategies that suit their unique need. And we will talk about one of those here today.
Adi Narayan 00:01:56 So in the context of microservices, you run an engineering team. When a team writes or migrates a microservice, what does that testing strategy look ? Talk me through the process of figuring out what they should test for and how they should sort of think about the problem.
Vivek Yadav 00:02:10 Yeah, so especially in the context of migration, usually there’s a goal of a migration, right? So, you’re generally trying to move from one system to another system. Most of the time you want to preserve the behavior and want to extend the capabilities so that you can do new stuff with the same code while making sure that the old stuff continues working as it was working. So, most of the migrations in the beginning are for migrations that unlock a future capability and in a for migration, you need to make sure that your past behavior is not changed. You are able to support that kind of traffic and capabilities exactly as it was before. It is very important to make sure that your service behavior is preserved.
Adi Narayan 00:02:55 Got it. So, when you mentioned for testing and migrations, this is essentially let’s say if you’re migrating Python for instance or if you’re going from one database to another, you want the behavior of the microservice to be exactly the same, the same input, same outputs nothing and you want to test it for all possible scenarios. Is that accurate?
Vivek Yadav 00:03:15 Exactly. Someone basically living outside the system may not even notice that a change happened that would be gold standard. So just to take an example, let’s say you have an old payment stack, hypothetically speaking, let’s say you have some stuff written in mainframes and you can write that in Rust today. Your system has been running for 40 years, it is doing critical payments and you want to make sure that all the historical stuff keeps working as it was working before while you change things under the hood. That would be a for migration.
Adi Narayan 00:03:47 And when you’re thinking about building these test suites, is it, does it happen in sequence? Do you sort of, first do the unit test, then the integration test and then eventually get to the regression test? Or do they all sort of happen in parallel?
Vivek Yadav 00:04:01 I think the code writing does happen sequentially, but the planning should happen thinking the end goal upfront because some of these tests scenarios which is unit tests, functional tests or integration tests, it is easy to do them layer by layer and that works naturally. But if you want to have some more esoteric testing as we had in our case, we had to think about that testing upfront while designing the code base and the layout of the code base because we have to structure the code to enable that testing upfront. And so, it depends on the scenarios. Code writing can happen sequentially, but planning should think through all things upfront.
Adi Narayan 00:04:46 Got it, got it. So, I think most engineers can relate to this. You migrate a service, it could be a for situation. You write through a test suite, switch suite, everything passes and then something breaks in production that no one predicted. From your experience, what do people often miss?
Vivek Yadav 00:05:03 From my experience, the things people miss are really tail end scenarios which happen very infrequently, team does not have collective memory of them. Those are not really recorded in your existing test suit. It depends on how many different scenarios your service have to work through. But in many practical, production scenarios, these test cases are in millions and so you cannot really enumerate them one by one in your test files, whether unit test or functional tests. In those scenarios we, uncover some corner cases which no one could keep in memory essentially.
Adi Narayan 00:05:47 Understood. Now when somebody’s, when a new developer is learning to code, they of course learn about tests and they start writing tests as test driven development, but you’re building a service or building a function and you’re writing tests around it. I think what you’re talking about is orders of magnitude more complex and you’ve written two blog posts on Stripes website, which walk us through these scenarios to lay the context for people. Can you give us a sort of an idea of what the system looks ? What are these sorts of big systems that you’re working on where testing, especially regression testing plays a huge role? Let’s lay out the context as to what this example looks .
Vivek Yadav 00:06:23 Absolutely. I’m going to take maybe two, three minutes to just set it up maybe a little longer, but it’ll really help us along the way. So as an example, let’s take an example of payment costs. So, payment costs are, think of them as costs of moving money on the internet. So, to give you an example, let’s say that you’re buying some shoes on Amazon and you are paying $150 for those shoes, Amazon is going to get about 145, 146 out of that 150. And the two three dollars differences are essentially going to different parties in the payment stack. So, at most relatable example is you have these different credit cards, and these credit cards will say if you use this credit card we will have 3% cash back or some other rewards and so on. All these rewards are funded by something known as interchange.
Vivek Yadav 00:07:20 So essentially when you’re doing this $150 transaction on the internet, about $2-3 of that is going as an interchange fee to the card issuer, the bank who issued you the card. So that’s one party in the payment stack. Then there is card networks, Visa, MasterCard or other card networks worldwide and these card networks have their own fees in different forms. Those fees are known as scheme fees and so each transaction would have more than one scheme fee and these are fractional fees, sometimes they’re fraction of ascent as well and so on. So, there could be many parties in the payment stack which are enabling that payment to happen, and they are taking their fee. So now if you from one transaction, if you scale to the internet economy, so let’s say in the case of Stripe, Stripe last year processed about $2 trillion and assuming let’s say on an average worldwide there is 2% network cost that comes out to be about $40 billion of network costs.
Vivek Yadav 00:08:20 So we built systems at Stripe which are estimating these network costs and trying to sort of make sure that we are correctly sending right data to the network so that we are not overpaying and so on. And these network costs depend on a lot of parameters. So, to give you an example, just in US Visa there are more than 60 parameters that impact it cut a single transactions cost and now you can scale that to different networks. So, let’s say there will be MasterCard, there’ll be Amex and others and you can now scale it to different countries as well. Canada will have slightly different things. Europe will have slightly different things. APEC has different things. So, a card transactions cost is impacted by more than a hundred parameters worldwide and that forms a very fat tail of scenarios which are very hard to enumerate in a normal unit test or functional test scenarios.
Vivek Yadav 00:09:19 The best thing we have for these scenarios is we have our historical data. So, networks do report some of these costs at a transaction level back at some delay. And so based on that historical data, we are able to sort of have a good sense of what does correct cost looks and based on that we can then form a test harness that can test the changes on our historical data. So, let’s say if I change some code or configuration, we can run that on the historical data to see what would have been the impact of this. We might have some expectations that they should not have any impact and we can detect if there are any changes and we go in the other direction as well that we expect some changes and we just want to figure out the magnitude of changes.
Adi Narayan 00:10:08 Got it. I think there’s a lot to unpack there. So let me sort of summarize this for our listeners and tell me if I’ve got this right. When you go to a website, we go to Amazon or any other place and buy something, I pay a certain amount, but then there’s a lot of middlemen, middle parties that are involved in this transaction and everyone takes their cut. Naively I used to think that, it would just be five a 1% transaction cost or something that. But it turns out that for I think Visa alone, there’s 60 plus parameters that are involved in estimating all these middle parties, little cuts and it’s very important to keep those synchronized, keep them updated so that as a payment company, you’re able to estimate all of these costs instantly and make sure that the right entity is getting the right amount. Is that correct?
Vivek Yadav 00:10:56 So, there are not that many entities. The number of entities is fairly limited, but there are different fees and a single entity might have different fees. So, the 60 parameters you talked about may impact on which fee will apply and to what degree they will apply, but the number of fees is relatively smaller per transaction.
Adi Narayan 00:11:17 Got it. Can you give us an example of these 60 or so parameters, can you take a few examples? What are these parameters? what is one of those parameters?
Vivek Yadav 00:11:25 Yeah, so let me categorize that into three buckets. So, one bucket will be just the characteristics of the transaction itself. So where is the transaction happening? Is this a card present transaction or is it an online transaction? Are you doing tap to pay or are you entering your manual card number, right? Did you provide your zip code or not with the card transaction? So those are the sort of transaction related properties. Then there are some properties which are merchant related. Where is merchant located? What type of merchant is this? Is this a fuel transaction or is it a telecom transaction or is this a retail transaction? Factors that are merchant dependent. Then comes some other factors which are about how, and whatís the level of data you are providing for the network. So, there are things called level two and level three data, which is level two data would be are you telling the network how much of is the stacks versus not in the total amount? Is the tip involved or not? Level three data will be full itemized receipt of this thing. And so, all of these parameters impact network cost. Another big thing is, is this a cross-border transaction? Are there effects happening during this transaction? So, there are many, many of these things I just named probably 10 of these.
Adi Narayan 00:12:43 Got it. Yeah again, I used to think that it’s a 0.5% service charge but clearly there’s a lot more going on behind the scenes. Thank you for that example. Given that, there are all of these things going on, as I understand it, some of these parameters could be yes, no types, is it a card, present card not present but, or it could also be a whole bunch of metadata linked to that transaction itself. So, is it right to think of these parameters as metadata linked to a transaction?
Vivek Yadav 00:13:11 Effectively yes, all of these will be metadata to a transaction and yes, some of them are yes/no. Ultimately all of them boil down to, some sort of an enum and an enum have a variety of values.
Adi Narayan 00:13:23 Understood. Yeah, that’s just useful to that example sort of useful to think about this and these parameters can keep changing, right? I’m guessing that say if I were, if I had a Chase Sapphire card versus a Chase Sapphire reserve card, would information that also be relevant from the context of these interchange prices?
Vivek Yadav 00:13:42 Yes, very relevant. So that particular thing is known as card type. What type of card you have and some whether it is debit or credit or prepaid and within credit there are many other levels and yes, all of that is very relevant.
Adi Narayan 00:13:57 Got it. Okay. So, from an engineering site, you’ve given an example of what this looks in the real world in terms of the different fees and different entities. From the engineering side, what does this look ? What do all of these parameters, these billing types, these card types, what does it look from under the hood?
Vivek Yadav 00:14:15 Yeah, so I can try to distill that to a very relatively simple form. See ultimately all this boils down to there is some information that is going to the network during an authorization or a settlement call and networks are estimate, actually are calculating the real network costs based on this data and based on certain rules. Now as long as for a given transaction all the rules are known and the data that is being sent known, then it is then it is possible to actually estimate the cost of the transaction effectively by reverse engineering the behavior. Does that answer your question?
Adi Narayan 00:14:54 So let’s try to think about this in a different way. When a transaction comes in, when a person buys something on the internet, there is the amount that they paid and then there’s all of these metadata or chain of information linked to this. Now when it comes to a payment provider Stripe, what happens under the hood?
Vivek Yadav 00:15:11 Only the transaction. So again, taking an example of let’s say you’re buying shoes on the internet. When the first time transaction is happening, there’s an authorization call that is going to the network. So, network could be network will be your card holder. So, if it is a Visa card, the call will go to Visa. If it is a MasterCard, the call will go to that, but the first call is an authorization call only and that is the only real time call. There’s no money movement happening in that call. So, we’ll call Visa, Visa will call the issuer bank, issuer bank will check if this is, does the user even have the funds or not to spend on this transaction and they’ll say yes or no. So ultimately in the response of authorization call, all you get is a promise that the bank will pay the money whoever brings the promise to the bank.
Vivek Yadav 00:15:57 So that’s step one. Now Amazon is a good example here. The second step of this transaction will be an authorized, sorry, a settlement or a capture that happens when Amazon actually ships you the shoe. So, you bought it today, maybe shipping happens two days later. That’s when Amazon will say, hey bank, give me the money now. I have sent the merchandise to the user and that step is known as settlement. There are two separate calls, the first call is real time, the second call is actually not real time, it happens as a batch file. A lot of transactions get batched together and sent to the network together as a response to that batch file, the banks and the networks will essentially move the money just one more thing. During that moment of money, they will take their fee out of that. So, you asked for 150, they’ll say here’s 146, we have taken our fee out of it.
Adi Narayan 00:16:51 As a payment provider, does Stripe figure out what these interchange fees are at the first lip stage or at the second stage?
Vivek Yadav 00:16:59 We do at both stages. The first stage also costs some money; much little compared to the second stage. We figure out all these costs all along, so we don’t wait on the network. Now it may happen that our estimate might differ from the network and that’s an accuracy metric we care about.
Adi Narayan 00:17:20 Understood. Okay. So, this is great. Thank you for explaining how these things work and letting us peer under the hood. Now from a testing point of view, you have all of those parameters and there’s some sort of a decision making rules based engine that is working on these parameters. What do unit tests cover? What does integration test cover and what gets missed?
Vivek Yadav 00:17:39 Yes, so unit tests will cover for basic functionalities. So as an example, just to really dumb it down, a transaction or a specific fee might have a fixed component and a variable component. So, when I say fixed component, it’s this fee means 1 cent every time this fee is charged, a variable component will be it charges 0.1% every time this fee is charged and we will have some logic return to calculate these things based on certain amount on the transaction, right? Is that logic working correctly? So that kind of small, small logics that overall combined together chain together to form the total experience. That’s where sort of the unit tests come in. Another example would be just translating the Stripe API request to sort of let’s say network language and that translation has a bunch of logic. Is that logic correct or not? So, every small, small pieces of logic is where unit tests come in but they cannot really cover for all the potential varieties in which different logic pieces can be combined to actually have a full transaction end to end.
Adi Narayan 00:18:52 Understood. So let me unpack that. So, for unit tests it could be things card present versus card not present, what the parameters look across Amex, Visa, whatever, different kind of entities. And you can do this for different kinds of parameters for each parameter and all its possible permutations. But when it comes to all of these 60 parameters and all the possible ways in which these things could be different and interact with each other, that’s not something that any unit is going to cover. You need something far faster than that, is that correct?
Vivek Yadav 00:19:24 Yes, exactly. Far faster than that. And also, it is about getting confidence on changes a stable system is good, it’ll always have changes. Can you be confident that the changes are correct and as expected and it is very hard in this domain to get that confidence just out of unit tests or functional tests for that matter.
Adi Narayan 00:19:46 Great, thank you for that setup. So now tell us how does one becomes confident about these kinds of changes?
Vivek Yadav 00:19:52 So the regression testing problem comes from a desire to be very confident about the changes being made on a service. So, in our case we are responsible for estimating network costs or Stripe. It’s a very sensitive area and the cost of error is pretty high. So, any changes we make, especially the bigger changes, we want to make sure that those changes have exactly the right impact that we want. Again, at the Stripe scale previous year our network cost as an estimate would have been about $40 billion. If you really drive that sort of letís, say to a normal number, just to think about on a day-to-day basis, if our costs are wrong by even 0.1%, that can add up to $5 million in error on a daily basis. So, this is where we have to be sort of extremely careful about these systems and sort of measuring our changes correctly.
Vivek Yadav 00:20:47 So sometimes let’s say we are doing a code refactoring and we want to make sure that it has no impact on the outcomes because the number of permutations on transaction cost is so high, we are not confident that a unit test framework will give us the right level of confidence. And so, what we built is a test harness that can run on historical data, whether one month data or one year worth of data, leveraging Spark to run this historical data through the core logic. So, we can run through the older logic versus newer logic, produce two different outputs and compare the outputs. And in a code refactoring we expect the output to be the same and if output is not the same, we will get a very granular transaction level diff as to where it is differing. That difference helps us figure out the bug in our logic and if there’s no difference then we get the confidence that the change is safe and we can roll it out without worrying about losing money.
Adi Narayan 00:21:49 That is great, yeah, thank you for explaining it that way. For someone who’s not worked with Apache Spark, what does it look and what does it do and what does it when you say build this logic on Spark or run this in a Spark harness, what you are talking about?
Vivek Yadav 00:22:02 In our case we have a microservice which generally does normal microservice stuff, receives a request, does some logic, sends some response, does some DB rights. We also have a need to test the same logic on historical data as I just I was talking about. So, we have arranged our service in a way that IO layers are completely separate from the core logic. So, our core logic behaves just a library that receives a function call effectively and gives a response and this is built on top of Java. So, we are able to wrap that same code library with a Spark wrapper as well. Now Spark wrapper, when I say that, let me just set up Spark for a second. Spark is an offline processing data processing engine effectively. Spark scales linearly; it can read from cold storage so think of S3 and discs and so on.
Vivek Yadav 00:23:01 And it is massively parallel. So, the way it works is it does bulk reads and bulk writes and the compute happens on different nodes. So, data travels to these nodes and travels back, but all this RI read reads and writes are happening on normal files. They’re not happening in a DB setup. So, they are much, much faster and it is way easier to spin up and scale down a Spark cluster compared to spinning up a sort of a MongoDB SQL cluster. Now in our case we have historical records of our request responses available in S3 and that enables us to run sort of a Spark job on them, read them, invoke the cost estimation on this library I was talking about and record the response and once we have that we can then do further analysis.
Adi Narayan 00:23:57 That’s great. So am I right in thinking about this as you have historical data, so effectively cold storage of transactions, all of those 60 parameters for each transaction and the outcome of the transaction, which is what the cost was for the interchange cost, was going back say , 2020 and even earlier and all of that is stored some way in some S3 dataset and Spark is able to, for each transaction, each time you change something, it could be a change in the code for some small microservice part, it could pull up those five years of data, run the same transactions again and see if the output is the same or not and do this really fast. Is that accurate?
Vivek Yadav 00:24:35 Absolutely, yes.
Adi Narayan 00:24:36 So we use the word cold storage, and I think not everyone may have the same idea of what that means in the context of a payment company. What is cold storage and I’m, I take it that’s a big project by itself. So, what does your cold storage look ?
Vivek Yadav 00:24:49 I would say companies of Stripe scale and even bigger companies, they do have a bunch of data analysis needs and data analysis needs are kind of different from your real time transaction processing needs. So, most of the companies I know about do have a bunch of their sort of request response logs or some other data available in cold storage. It’s just a normal hygiene stuff you do as you scale up. There are very standard ETL shaped, so ETL is Export Transform and Load I think type shaped work that happens, which for any sort of key historical data sometimes, because of regulation as well that these companies store, I do want to call out one thing there, there. Stripe and companies in similar domain are extremely careful about what is stored in this cold storages. So, none of the sort of personally identifiable information and things of that shape are never stored. In these cases, we are just storing transactional data, transactional properties, they don’t identify a person, they just identify type of a transaction effectively.
Adi Narayan 00:26:02 Got it. So, if I go back to the example of buying something on Amazon, it won’t include the fact that who I was, what my age was or even what the shoe was, but it’s more of this is what the transaction looked , this is where it was made and the card parameters perhaps and then the outcome, correct?
Vivek Yadav 00:26:19 Yes, yes.
Adi Narayan 00:26:20 Got it. You said a few other things a little while ago, so I think we should think and unpack first you run this whole suite of tests, suite of regression tests on a Spark cluster and then it runs the whole thing and it tells you hey, look either the same or it’s different? What scale are we talking about? Is it tens of tests, is it hundreds of tests? What does this kind of, to go back to the idea of I feel confident that this change is okay, how many tests should be done?
Vivek Yadav 00:26:47 In this scenario what we are really doing is, we’re not running a test, we are essentially running all a bunch of historical data through the new set of things. So, let’s say you had old code and you have new code, right? So, you’re trying to run old transactions through new code. So, to give you a sense of scale, hypothetically speaking, let’s say Stripe has a hundred million transactions a day, each of these transactions will have correspondingly about 500 million different cost items daily. And so, if I say one year worth of data, we are talking about, I don’t know, north of 30 billion transaction input and out about five times of that on the output side. If you talk about a week, it’ll be much smaller. So, we are running all these transactions through the Spark system essentially and getting a response back and then comparing the output with the historical data. So, it’s less about number of tests, it’s more about number of transactions within through.
Adi Narayan 00:27:51 And how do you decide that, is this a small change, I should run it on one day’s worth of transactions or this is a big change, I should run it on a week’s worth of transactions?
Vivek Yadav 00:28:02 This is, I would say just the SOPs of the system depending on the sensitivity and depending on the use case. So, change is one type of use case. The other type of use cases we should talk about where it is useful, but it really depends on the need. So, most of the scenarios we essentially run a week worth of transactions and if there’s no change, great if there’s some change run for a longer time, get more data and then help with debugging.
Adi Narayan 00:28:29 Understood. Okay. Let’s dig into what the design of a regression test system looks . Let’s try this out. if you were to draw this as a diagram, what would be the different components, for a system of this type? So, you have the cold storage which has all of the information, all of the transaction records going back X number of years, you have the microservices living in someplace and then each time a change is made, this job takes over and it runs it through a week’s worth of transactions. When that is finished, do you get a report? Do you get just the delta of the things that failed and who gets the report? Is it the person who’s built the test? Is it someone else? Is it sort of a team that gets it? What do you do with that sort of result?
Vivek Yadav 00:29:07 So let me set that up and then I’ll talk about a bunch of scenarios around that. So first of all, by definition this is about regression testing. So, we have to have some historical data against which regression is being done essentially. So, the first requirement for this thing is to work , hey, yes, your request responses, historical request responses are available to read and compare. That’s one. The second thing is your overall code base should be arranged in a way, that it is working in sort of real-time service environment as well as it is wrapped in a Spark wrapper and the wrapper is ready all the time. So essentially if you’re making core code change that changes the core logic. The logic essentially gets wrapped as a library and then library is a dependency of essentially a Spark wrapper. So, they essentially change together by definition that’s a part of your build system.
Vivek Yadav 00:30:07 So that’s one part. Now let’s say that I make a code change right now and I raise a PR. When I raise a PR, I can comment on that PR with a specific tag and by doing that specific tag I am essentially telling a system that we have pre-set up to invoke specific Spark job with the input of one week worth of data. So that one week worth of data is pre-identified, it gets refreshed every week, but that’s identified on a location. And so, with that input we invoke the Spark job. The Spark job runs through airflow in a cluster and when it is done, the second Spark job that will run, which compares again the corresponding one-week output with the new output and it has a predefined format. So, it basically prepares a report, the report, anyone can read that report who has access to it, but it attaches that to the PR.
Vivek Yadav 00:30:58 And so with that, the PR reviewer and the writer both have access to the output. So that’s one way to invoke it, when I’m making code changes. There’s another way which is these networks keep from time to time changing the rules of how these fees are to be calculated. So that kind of thing happens every quarter and every six months there are relatively bigger changes and so on. So, in this whole system, either code could change or the configuration could change. The rules are the configuration and so we could run. So, when configurations change, especially when networks change, we’ll try to run about year worth of transactions to estimate the impact of those changes. And in that case, there’s a separate sort of the same swag job is basically pre-packaged to run on a year worth of data with sort of a new input, which are the new rules, and it’ll run that and produce a report. In that report changes are expected, and we are trying to understand the changes in the impact of those changes.
Adi Narayan 00:32:01 Got it. A lot of stuff there. Let me try to summarize a couple of things. One thing that you mentioned was I think something around the code is working on a real time level and as a Spark object. What I meant took that to mean is you write something, you build a tool which should work right now for whatever transaction processing, but it should also be wrappable within the sub Spark job in order to do this historical analysis. And in order to do that it needs to conform to certain standard design patterns. Correct?
Adi Narayan 00:32:33 Yes. And those design patterns that you mentioned earlier where, classic ones separate the core logic from the side effects, if you’re doing a calculation, just do the calculation, don’t mix it with writing to a database and so forth. Is that what you’re thinking about?
Vivek Yadav 00:32:48 That’s most of it really.
Adi Narayan 00:32:49 Got it. When you run these huge jobs, and I think the distinction that you made was very good, which was, you could either change the code that Stripe has to process or you could change the characteristics of the parameters, configurations, each of those things could change. Sometimes both could change, but you want to be able to isolate one thing and do the regression test for that and then isolate the other thing. Is that a good way to think about it?
Vivek Yadav 00:33:16 Yes, correct.
Adi Narayan 00:33:17 I get the part where you’re changing the code. So you say I’m migrating and I’m doing a standard. There is some particular data database that the company uses, we’re switching to something else, we’re migrating all of those things. My internal code changes, so then I run this whole flow of transactions again with the new code. When it comes to configurations, what am I running again? Is it because the transaction is no longer the same, right? The configuration itself means transaction is different. Do you, see what I’m trying to say? what gets tested again? What gets run again?
Vivek Yadav 00:33:45 Yeah, so configuration changes are essentially the rules that determine the network costs. So effectively what’s happening is a transaction has a proper, the its properties. So those are not the changes I’m talking about. I’m talking about the network rules. So for example, networks will say if you pass a VV on a transaction, then we will not charge you a specific fee. It’s known as a VS fee, but if you don’t pass it, then I will charge you a VVís fees.
Adi Narayan 00:34:16 Am I right in if I simplify this, it could be oversimplifying, there is the metadata for a transaction and there is the interpretation of the metadata. The metadata might mean something, it might not mean something and those things are what can change certain things which before didn’t cost anything, suddenly could cost something.
Vivek Yadav 00:34:32 Exactly. So, the rules of the game might change. The transaction properties are the same, but the rules might change.
Adi Narayan 00:34:37 And if the rules change, what are you comparing it to? Because say I bought the same shoe in 2024, the interchange fee was something, some minor thing was 0.01%, right now it’s become 0.05, the output is going to be different, right? So, what are you comparing?
Vivek Yadav 00:34:53 Exactly. Yes. That’s a very good question. So, let’s look at the second side of this equation now. So, Stripe is a platform a lot of merchants are processing on this platform and Stripe does have a product which is an interchange plus billing. That basically means that all the big companies are saying, hey Stripe, why don’t you pass the cost of processing directly to me and charge some markup on top of that. That is the predominant way of billing payments essentially. Think of it as the cost plus way of billing. So, when networks change the rules, Stripe gets impacted. Yes, but also our users get impacted. So again, let’s say that you’re Amazon and Amazon is processing on Stripe. Amazon wants to figure out, so networks will say, starting on April 1st, the rules that determine network costs are changing and now Amazon needs to figure out what will be my impact of this.
Vivek Yadav 00:35:51 Now Stripe could run Amazon’s transaction, Stripe could encode the new rules in our system and run historical Amazon transactions through the new rules and then produce the output and compare that with the old fact and produce a different, can give a guidance to Amazon then, hey Amazon, based on our understanding, the new rules are ly going to decrease your cost in these, these, these areas and increase your cost in these, these, these areas. And so Stripe will produce a guidance for our platforms and that guidance then helps them determine how to sort of figure out their unit economics.
Adi Narayan 00:36:29 That is so interesting. Because when you think about any software company, any engineering organization, you’re thinking about testing and unit testing and all of that to make your code better catch bugs, things that. It’s sort of an internal use case. What you are saying is the regression test, yes, it’s an internal use case. We don’t want to make mistakes and lose money, but the same regression test is also quite a strong benefit. It provides value to a customer as well. For a big customer Amazon, they can use that regression test output, I guess packaged in a certain way to say, hey, this new card that credit Visa has launched will be, X percent cheaper or more X percent more under certain circumstances. So that, and that kind of data is useful. It’s a useful business input that they can use for decision making.
Vivek Yadav 00:37:18 Exactly, yes.
Adi Narayan 00:37:19 That’s fascinating. I never thought about that. It’s a great way to sort of think about it in terms of what value regression testing can offer from this whole system that you have right now. There are many parts to it. There’s the cold storage, there’s the ability, this part that’s able to spin up, take the code that you have microservices, wrap it and run it for weeks, worth of data and even more, what was the hardest part to build it?
Vivek Yadav 00:37:42 The hardest part was to start the overall code base with this mindset that we are going to adhere to these principles and arrange the whole code in a way that it is amenable to the same code. Is amenable to both Realtime path as well as a Spark batch path. That took us, we tried a bunch of other things. That was not the first obvious decision. We wanted the regression testing, but we were not sure Spark would be the right thing or not. But ultimately, once we landed on that, it took few months to organize the initial code base, but after that has been fairly easy.
Adi Narayan 00:38:21 Why was that difficult? Because at some level, it seems logical, right? Separate core logic from side effects and things that. The design patterns seem fairly standard. What was the hard part there?
Vivek Yadav 00:38:31 I don’t think anyone had done a Spark based regression thing. So, I’m simplifying a little bit, but ultimately any service when it starts, it needs to load some context. And the usual pattern on services is that load the context from DB. And so even small things, let’s say FX rates, which is relatively small configuration effectively are, yeah, you read from DB when you need them, but you have to still arrange your sort of for a single request, arrange for the processing in a way that all the reads that will be needed later in the pipeline gets done upfront and IT context is prepared for that transaction. And then that context is given to the library to execute just reorienting the whole thing from that perspective was a bit of a mindset change.
Adi Narayan 00:39:22 So don’t load the dependencies for each transaction but load them en mass at the beginning for a bunch of transactions?
Vivek Yadav 00:39:30 No. So that one is needed for Spark, but even for a transaction you may load things incrementally, oh now I need FX rates, let me load this. Okay, now I need that set of rules, let me load them and now I need that, right? You might do sort of the whole thing on a piecemeal basis. Another way to do this thing will be to load all the things upfront, prepare the context packet and then give it to the library to execute.
Adi Narayan 00:39:53 Got it. Got it. Okay. The context back and as someone has worked on migrations and I think a lot of our listeners would’ve worked on some migration to the other, they’re already very stressful. There’s a lot of timelines that engineers are trying to protect and get the migration done. How did you make the case for building a harness during migration rather than sort of treating it as a separate project after the fact?
Vivek Yadav 00:40:14 Yes, in fact because we were doing a migration, this was the easy part. As I said, it’s a very sensitive area and we are directly dealing with money. So, migration had to be really, really confident that the new changes or the new set of code is the right thing and how do we prove that? The only way to prove that to us as well was to make sure that we run the historical data on it and as if it comes out with the same, it’s great. Now, one-way people do this is they will run a new system in the old system in parallel for, I don’t know, depending on the use case, sometimes many months. And that way they generate confidence and slowly roll out. And running, we wanted to shorten that timeline basically rather than doing a month’s worth of rollout, we wanted to have confidence upfront and also we didn’t want to do this incrementally because if you are doing the shadow rollout situation, you’re just dependent on new transactions coming and so on. So, you don’t get too many iterations on it. But with a Spark based wrapper, we could run sort of a month worth of transaction, year worth of transactions multiple times a week. Every developer could have their own Spark job running separately while they’re testing different phases of it. So, it was actually a speed bump for us in terms of once this initial setup is done in terms of testing and so once we lay out that story, it becomes very easy to buy in for everyone.
Adi Narayan 00:41:47 You mean a speed boost, right? Not a speed bump.
Vivek Yadav 00:41:49 Yes. Speed boost, sorry. Correct, correct.
Adi Narayan 00:41:50 Got it. So essentially, I think one of the things also you mentioned was you might say that the feature is being developed at a slow time in the markets, I think towards the end of the year after Christmas, but if you want to really test it, you want to test it during a busy shopping season for instance. So, and you can do that, you can sort of pick, you can, by using a representative week, you can get a lot of data, which you may not get if you were just sort of let it be, let it sort of run in parallel or something.
Vivek Yadav 00:42:17 Absolutely.
Adi Narayan 00:42:18 Got it. A couple of quick questions. You’re running a week’s worth of data to Spark that looks, that sounds like a lot of compute and that can cost a lot. And given that you mentioned developers can use it, if I was built having a PR and I submit a PR and I put appropriate comment, it’ll go and trigger a week’s worth of Spark testing. Do those costs add up? Is Spark mean compute on Spark? Are we talking about very large amounts to do these kinds of tests?
Vivek Yadav 00:42:44 So I think we should think about two types of costs here. One is yes, the compute cost, another cost is the storage cost and then the time cost. So, once we establish that this kind of testing is needed for our needs, then the only two options are either you do it with a DB or you do with Spark. Now with Spark the storage is actually that cheap compared to DB and the time cost is also way lower. So, the only cost that increases, the compute cost, which is not that high, especially if you’re doing one week, one month worth of thing at a time. Spark costs increase a lot if you’re doing same amount of compute every day. So let’s say if you’re doing multi-year worth of compute every day, that’s actually pretty costly, but otherwise not.
Adi Narayan 00:43:32 So if you’re testing a whole year’s worth of data every day, then it’s a lot. But if you’re only doing a small amount say a week or something each time you release a new feature or make a bug fix, then it’s not much. Correct?
Vivek Yadav 00:43:44 Yes, exactly.
Adi Narayan 00:43:45 Got it. Let’s dig into a few things that you mentioned that I found interesting. Realtime microservices, you’re querying live data, right? You’re hitting up databases, you’re looking at feature flags, you’re looking at configuration tables and all of that in order to make a decision. How do you handle, take all of these dependencies and store it in your cold storage you because these are Realtime dependencies that you’re pulling from somewhere. is it all rows in a table? How does that look like? how do you take, take all those and put them in some place that can then be pulled out in the right, in the right sort of format for running the hardest while running the harness?
Vivek Yadav 00:44:19 Yeah, so that’s a real consideration and that’s why this whole thing I’m describing may not fit for a different use case. In our use case, the biggest thing is the transactional data that’s the highest sort of, let’s say number of records, but other things which is, let’s say what determines cost of a transaction, that kind of configuration is few thousand rules effectively worldwide. So that’s not that much. And similarly, the other things like FX cost or some flags context, all of those things are way lower in sort of num different number of rules. So, the way we do this is think of converting every lookup you do during a real time call into a join in Spark. And so we do all those joins upfront and so let’s say we’ll read transactions on the Spark side and okay, this transaction contains Australian currency and USD, okay, let’s just load, FX rates and the pick the FX corresponding to that, so join on that. So, let’s just one example, but there could be other joins based on that context, but we load all of that upfront to do the joins upfront and then we have a context prepared and then we send it to the library. So, it comes at a additive cost every time sort of we have to add a new dependency in this chain, we have to update the Spark join path as well. But those things happen relatively infrequently.
Adi Narayan 00:45:46 Correct me if I’m wrong, when you take the example of, If a transaction required, like you said the Australian effects, would the cold storage capture the exchange rate at the time of transaction and then save it somewhere? Or is it a case of, I have the timestamp and I can go, go look up the Australian exchange time series and then pull that number and connect it at the time of running the job?
Vivek Yadav 00:46:10 Yeah, we’ll have it pre stored.
Adi Narayan 00:46:12 Okay, so a snapshot at the time or everything that’s relevant to transaction at the time is stored into the cold storage?
Vivek Yadav 00:46:18 Yes.
Adi Narayan 00:46:19 Got it. And since the cold storage is just a standard S3 bucket, it’ll be several, maybe hundreds or thousands of terabytes, but it’s still cheap. It’s just an S3 bucket.
Vivek Yadav 00:46:28 Very cheap. Yes.
Adi Narayan 00:46:29 Understood. Okay, so now I think a lot of you, what you spoke about is relevant for migrations. Outside of migrations, where else has this regression test system, proved its value?
Vivek Yadav 00:46:41 Yeah, so I say migrations is where it pays off immediately and it’s where we, that’s the best place to build this system. But after migrations, a few things. One is code changes continuously keep happening and so migration help the system put in place. But once the system is in place, you can keep using the system for any code change to detect if the code change is having the relevant impact or not. Sometimes you expect no impact, sometimes you expect some impact and so you can detect that. So that’s one example. Another example I gave was sometimes networks will change the rules. So let me try to actually give a more fashionable example. So, let’s say that you are working with one of these AI companies and you have some token costs and you want to figure out, and let’s assume that you had your historical request responses stored and you want to figure out what would be my cost if I changed my model from A to B.
Vivek Yadav 00:47:40 And if you could replay your data, not exact data, not exact sort of inference, but whatever matters for billing, you could just figure out if I changed from model A to model B, my cost will reduce by 10X. It’s the same thing. So, you can use these systems to basically figure out if the rules of the system changed, in my case, if the network cost rules changed, what would be the impact of those changes on my bottom line on my customer’s bottom line. So that’s two. The third example is sometimes incidents will happen, sometimes you will write a wrong code or wrong configuration and estimate wrong things, maybe send wrong cost to your customers as well. And then later when you detect the incident you need to figure out what was the impact of that mistake. The way to do that is you correct the mistake and you compare the new data with the old wrong data and you get a difference and you can attribute the difference to every transaction. And based on that figure out, the impact on every customer. These are just few cases. The couple more that we can go into if you’re interested.
Adi Narayan 00:48:49 So, one of the examples that you mentioned, the token example, which leads into another question that I had, which is outside of payment, outside of a company like Stripe, who else would benefit from a system like this? Because it is a lot of investment to build it.
Vivek Yadav 00:49:03 So I think this system really works. If your area or your service is not dependent on too many dependencies, that’s a requirement. If you have too many dependencies, I don’t think it makes sense. Then if you don’t have that many dependencies, I think anyone who is doing a migration should just set this up. It requires little upfront investment and pays up within the migration timeframe and then keeps giving dividends. Especially in the current age where, let’s say cost of making changes to the code is reducing the cost of knowing whether the change is safe or not has not reduced. Everyone will need some scalable systems that can actually very reliably tell you if your change is safe or not and help you. The output actually helps you debug as well.
Adi Narayan 00:49:53 You mentioned they should set it up and it’s not that much work to set up. What would it entail? So first I’m guessing you need to have a cold storage, which means everything that’s coming in and going out, put it somewhere, even if you can’t model it too, well just keep it somewhere. Second is creating, writing, designing your microservices in a way that they can be wrapped in a Spark wrapper, right? For somebody who, for folks who’ve never done it, how hard is it to sort of shift into that role into doing that?
Vivek Yadav 00:50:20 It’s a learning curve. It’s equivalent of saying, let’s say someone has not done unit tests before. How hard is it to set up unit tests? It’s equivalent amount of effort. Does it help?
Adi Narayan 00:50:30 Got it. So, it’s sort of a mindset shift effectively
Vivek Yadav 00:50:33 It’s a big mindset shift. Yes. And but it pays, it forces discipline on you that really pays up.
Adi Narayan 00:50:41 Understood. Okay. In real world systems, what you’re saying makes a lot of sense. When you have everything stored and in order to run an old transaction, you pull up that transaction, you get all the parameters linked to it, and then you run it and boom, you see the output. What if some dependencies are missing? What if cold storage has gaps that you didn’t know they exist, but then does the test just fail and sort of skip it? Is that transaction ignored? What happens in situations like that?
Vivek Yadav 00:51:07 Yeah, it depends on how critical was the dependency. So, if the dependency was absolutely critical for the functionality to happen, so let’s say in cases of foreign exchange situations in our case actually, that things will fail and we’ll get an error output. And so that’s a thing to look at, but if the dependency was not super critical, then it might fail in a way that you don’t notice upfront and it might show up as a difference.
Adi Narayan 00:51:32 If there’s a gap in the cold storage. What happened? You said if it’s important then it fails, then make noise. If not, let it go. This was my understanding.
Vivek Yadav 00:51:40 Yeah. So, I think the upfront advice I will have is you should set up in a way that there’s no gap in the cold storage. You should treat cold storage as a first-class citizen to make this successful. But if you have not done that work, then either your sort of work will fail or it’ll produce some difference. So, let’s say your Realtime service had some data, but your cold storage did not have that data. The impact of that will be visible in your test situation versus your past golden dataset. There will be a difference and you’ll debug that difference and you’ll realize it is coming because there was a gap in the cold storage.
Adi Narayan 00:52:21 Got it. And if you’re in a big company in most big companies, you are getting data from some other part of the company and in order to make these decisions with the microservice, you are pulling from different sources. What I’m hearing you say is when if you are building a micro regression harness for your neck of the woods, whatever data you pull from other sources, even if they’re internal sources, you capture a snapshot and you keep it with you. So, you don’t go and query those other sources at testing time.
Vivek Yadav 00:52:49 Yes, correct. So that is the cardinal rule of doing Spark jobs basically, that you don’t want to query when the Spark job is running. You want to load all the data upfront and then use joins.
Adi Narayan 00:53:02 Load the data upfront and use joins. Got it. Okay. Understood. Letís look at, oh, we have to get to the AI part, which is on everybody’s mind, given that code is so much cheaper to produce, features are faster to shift. Does that mean that having such a digression harness increases the confidence that each new feature that’s implemented could be vibe coded even if it’s vibe coded, the level of testing and sign off required before it’s shipped is still high?
Vivek Yadav 00:53:32 Yes, exactly. So, think of this as your own local eval on your service and your AI agents could be making all the code changes as long as they pass this eval, the changes are safe, you get the confidence immediately.
Adi Narayan 00:53:46 Got it. So, if it’s, regardless of whether it’s a small change or a big change, you’re able to run it through a weeksí worth of heavy transaction data and if it, there are no diffs or if the diffs are all minor ones, you know that it’s mostly safe.
Vivek Yadav 00:54:00 Exactly, yes. You get that confidence with this. Yes.
Adi Narayan 00:54:04 Understood. One other thing that I had as someone who has ran aggression tests and I would, a CI would run and it would say, hey, something’s failed. It can be quite a frustrating experience because something failed. You got an error log, you don’t know if the mistake was in your code or if some sort of a dependency in a library that got missed. Or if it’s something upstream that happened. Does the testing harness make that triage process less painful or because from what you said, it’s running so many times, how do you make that less painful for developers?
Vivek Yadav 00:54:39 Yeah, so this whole thing really helps in that because most of the times output is not an error message. The output is a different difference log. So, it’ll say for transaction X, I expected output to be two dollars, but it was two point one. So, you essentially get a transaction comma fee level difference or in our case it was transition comma fee level. But in any case, the way this whole thing works is you get a difference at a very, very granular level and that really helps you pinpoint, okay, what is common about this set of, granular transactions, oh, all of these are UK to US transactions. So, there must be something off in that area of the code or the rules. So, you get a lot of debug hints upfront with this.
Adi Narayan 00:55:37 So effectively you’re able to replay that particular scenario again, or you’ll know what those scenarios, what the parameters impacting that scenario is, so you can pretty much quickly go to the issue. Okay. Understood. Yes. Does AI help with summarizing and making that clearer to interpret?
Vivek Yadav 00:55:56 I have not tried that, but that’s a very good suggestion and I’ll pass that on to the team.
Adi Narayan 00:56:01 Understood. Yeah, because I’m guessing people already, they must already be using it. You may not be aware because, because when you see these logs, when we see logs coming in, tell me what the error is. I can you it’s especially the standard ones, it’s easy to use it for that. Okay.
Vivek Yadav 00:56:14 Yeah. Once you say it, I feel I haven’t thought of this before
Adi Narayan 00:56:19 Enough. Cool. I think we’ve covered quite a lot of ground. Anything that you’d to add that I may have missed out, or do you want to rephrase something?
Vivek Yadav 00:56:27 Yeah, just one sort of high-level thing I want to call out is the cold storage apart feels fairly sort of, let’s say risky. And we take absolute care of making sure that no privacy related or PII related data gets stored to the cold storage. So, that work is sort of taken care of at a very fundamental infra level at Stripe.
Adi Narayan 00:56:53 So PII meaning Personally Identifiable Information, right?
Vivek Yadav 00:56:57 Absolutely, yes.
Adi Narayan 00:56:58 Got it. So, none of that is, and I take it the sort of standard processes around how do you scrub the data before you put it into cold storage.
Vivek Yadav 00:57:04 Yes, exactly.
Adi Narayan 00:57:06 That covers most of what we spoke about. If people want to know more about what you do or learn about the systems you built at Stripe, where should they go look for it?
Vivek Yadav 00:57:14 Stripe blog is one place where I have published about this one, but other than that my email is [email protected] so they can reach out to me.
Adi Narayan 00:57:24 Understood. Good. This was a really interesting conversation. I think a lot of us could benefit from regression testing harnesses, getting the management and engineering management to approve it, to build these kinds of systems, it requires a lot of foresight and to think about all the possible ways in which they’ll pay off. And the example that you pointed out about using a regression test harness to even explain to a customer like Amazon how a card change transaction would impact their bottom line is really clutch. I mean, it’s sort of something that I didn’t think about, but it can, it shows the value of having such a system, not just for internal use cases, but also for external use cases. So yeah, thank you for taking the time. And thank you for explaining all of these concepts in a pretty easy and lucid manner. For Software Engineering Radio, I’m Adi Narayan. For related episodes in a link to the blog. Please do check out the show notes on SC-radio.net and do check out our other episodes with similar topics. Thank you very much and thank you Vivek for taking the time with us.
Vivek Yadav 00:58:23 Thank you, Adi.
[End of Audio]
