Vinay Tripathi, a senior network engineer in Google Backbone Engineering and an 18-year network engineering veteran, discusses BGP optimization, a technique that’s critical in achieving top goals in distributed applications. Host Philip Winston speaks with Tripathi about BGP, autonomous systems, peer grouping, router hardware and software, software-defined networks, and shared network optimization and debugging stories. Brought to you by IEEE Computer Society and IEEE Software magazine.
Show Notes
Related Episodes
Other References
- Network Function Virtualization
- Software-Defined Networking
- What is VxLAN
- OpenFlow
- RPKI (Resource Public Key Infrastructure)
- IGP (Interior Gateway Protocol)
Transcript
Transcript brought to you by IEEE Software magazine and IEEE Computer Society. This transcript was automatically generated. To suggest improvements in the text, please contact [email protected] and include the episode number.
Philip Winston 00:00:18 Welcome to Software Engineering Radio. This is Philip Winston. My guest today is Vinay Tripathi. Vinay is a senior network engineer at Google Backbone Engineering. He has 18 years of experience in various networking technologies including BGP and SDN, which we’ll discuss in today’s episode. Vinay previously worked for Amazon and Juniper Networks where he held expert certification in both enterprise and service provider routing and switching. He is an inventor on four patented networking technologies and is a senior member of IEEE. Welcome Vinay.
Vinay Tripathi 00:00:57 Thank you Phil. I’m excited to be here and looking forward to contributing my knowledge.
Philip Winston 00:01:03 Today we’re going to discuss the BGP protocol and how network and software engineers can improve the performance and reliability of their applications by understanding and optimizing BGP. First I want to point to a past episode on BGP Episode 468 on Internet and Routing BGP. This is a great introduction to networking concepts and BGP. We’re going to review just a little bit about BGP to start. Firstly, what is BGP?
Vinay Tripathi 00:01:33 Okay, so before I go and talk about BGP, I’ll just take a step back and just briefly talk about the network in general. So if you look at the internet, internet is basically made out of bunch of nodes, thousands and thousands of nodes which are physically connected through the fiber and then you run protocols on top of them. So BGP is one of the protocols which makes internet happen. So what that means is when you look at the internet, it’s basically bunch of autonomous systems which are owned by certain organizations and then they talk to each other, they share information and they make communication happen end to end globally. So the BGP protocol is as such is called port gateway protocol and there are two type of BG protocols, internal and external. So the external protocol is the one which is used to connect different autonomous systems and the internal BGP is the one which is used to communicate those routes which were received from external entities within your autonomous system.
Vinay Tripathi 00:02:44 So if I have to explain someone who is not familiar with BGP or networking concepts, so the BGP is kind of a mailing service for the whole internet. So for example, in the United States, you have many postal offices and every city has its own postal office. Now if I want to send some letter from Virginia to let’s say California, I will drop that letter here and it’ll reach their going via multiple postal offices. So these cities or states you can call as an autonomous system in context of a BGP or routing and then it has to travel through multiple autonomous systems and then you receive your letter or the packet at the other end. So that is high level BGP. I am happy to go deeper if you want me to.
Philip Winston 00:03:34 So you use the phrase autonomous system. I’m curious why that is autonomous system and not autonomous network. I don’t know if there’s a history there or if it’s just a choice of words.
Vinay Tripathi 00:03:45 Yeah, I’ll try to attempt that but I don’t know how close it’ll be to the truth because mostly these information will be somewhere in the RFCs, which were written years ago in BGP. So if you look at the BGP protocol as such, what happens is every node, every router or node, whatever you call, they run their own instance. So they receive the routes from their peers, they run their algorithm on that and then they decide what is the best path to reach the destination. And it is done independently on all the net nodes within your autonomous system. So it’s basically, it’s not a network which is deciding, it is each and every node which is deciding within the autonomous system. The only difference here is no matter who is owning that autonomous system, their decision has to be consistent. That is how it can make work, internet end to end, otherwise there’ll be inconsistency. So I think that is the reason why it’s not called network, it’s rather than system because everyone has to make their own decision.
Philip Winston 00:04:50 So if I understand it, I think all of the networks in one autonomous system are under the same administrative control, whereas if you have two autonomous systems they might be sort of owned or run by different companies. Is that right?
Vinay Tripathi 00:05:05 Yeah, that’s right. So each autonomous system can be mapped to organization. For example, Microsoft can have its own autonomous system. Amazon can have its own autonomous system. Similarly, service providers, internet service providers or ISPs have their own autonomous system. And when they talk to each other they use EBGP and then within them they use IBGP.
Philip Winston 00:05:28 If an organization just has one connection to the internet, whether it’s a home ISP or a business connection, I take it they don’t have to worry about BGP. When is it that you do have to worry about BGP?
Vinay Tripathi 00:05:43 Yeah, that’s a good question. So if you have just one BGP or your requirement is to use internet connection just to connect to the outside world, that is where you can actually manage with the just static routes which you can negotiate work with your internet service provider, which kind of default route you can configure a default route towards the internet service provider. And similarly they can configure a route specific to your organization. So you can manage in that scenario. But when you want to have for some reason, let’s say you want to have redundancy in your network. If one ISP is down, you want to go through another ISP. That is where more than one BGP connections come into picture and that is where a lot of engineering goes into where you want to decide which one I want to use and when, because every service provider charges you differently. So you want to make sure just everyone, you want to be economic, and you want to decide which connection I should use. So for example, if you have service provider one which is cheap and service provider too, which is costly, mostly for your normal operation you’ll use IP one. But let’s say IP one is down, then you’ll go and use the IP two. So these are very simplified solutions. I’m talking about at the larger scale you deal with just similar problems at different scales in varieties.
Philip Winston 00:07:13 So you kind of mentioned this, but just to be clear, if I’m an organization and I’m adding my second connection to the internet, kind of let’s not go through the entire process but what are some of the main steps I have to take at that point? How do I make that transition to a multi network or a multi connection company?
Vinay Tripathi 00:07:32 So there will be various levels of work involved here. So one is you and your service provider will agree on what are the charges, what is the availability of the connection, it’s kind of agreement you are signing with the service provider and then you’ll agree on the how will the configuration be, how you will configure it, your side and if there is any a with your internet service provider, how to configure the BGP sessions between both of you. So these are very basic requirements to start with and then depends on it becomes complex let’s say. So let’s take the Cloud providers for example. So if you look at Microsoft or Facebook, they have to peer with different type of service providers. They have to peer with Google, they have to peer with Microsoft, they have to peer with some transit providers, and they have to also peer with different other entities.
Vinay Tripathi 00:08:30 And if they had to deal with them separately because everyone has different agreements in terms of how you are going to use that network. For example, Facebook and Microsoft might have mutual understanding of using connection because both of them benefit off the traffic for each of them. So they might not charge both of them. They’ll allow the traffic which is going for their services and the both of them allow it without any charge. But at the same time, let’s say you have a service provider, transit service provider whose job is to provide transit connectivity globally, that guy doesn’t have any mutual benefit with you. So that company is going to charge you for the traffic which is going to transit over their network. So these are some examples how BGP connections play in the real life when they are with the entities.
Philip Winston 00:09:25 I was going to ask about peer grouping, but it sounds like you just described that. So peer grouping is when you have this arrangement with another autonomous system to exchange data without cost, is that the difference?
Vinay Tripathi 00:09:38 So peer grouping is more than that. So what happens at the device level when you enable BGP protocol on any device, so when you are dealing with the internet scale, you are dealing with millions of routes. So routes are nothing you can think number of end hosts or networks advertise their own prefix and the those are reachability are what I’m calling route and network or hosts. So BGP routers in the real networks deal with millions of routes. Now any change happens in the network, it has to do a couple of things. For example, the moment it receives the change from its peer, it has to go through whole routing table, analyze the routing table and check if this is the best path or not. It’s called best path route selection, it’s algorithm which is vendors, it’s defined by the IT how should, what should be the outcome.
Vinay Tripathi 00:10:37 But every vendor has their own implementation in this and once they run their algorithm against that change, then they decide what is the best path to reach that particular host or network. And then they had to communicate these best routes to their peers in the upstream. Now the challenge is more number of routes you have, you can imagine how much compute it is going to take to go through this cycle of, and you can imagine in the internet every second there is a route change continuously. So your router is continuously processing. Now peer group is where you group multiple BGP peers in one group based on the treatment they are going to get. For example, if you have your inner network and there are certain devices who are going to get the similar treatment in terms of routing policy, then you’ll group them in one peer group, let’s call it peer group one.
Vinay Tripathi 00:11:34 And then you might have another set of devices which are going to receive different set kind of routes. That means they are going to have different kind of routing policies. So they’ll go and become policy peer group two. So what happens in the real networks in the devices is as you scale your network, your peer groups expand and you’ll have more and more peer groups, more peer groups add more cost in the computation of the device. So every time you are going to have the change, you do best path algorithm, you analyze all the policies for all the peer groups and then you advertise routes. So at some point of time you had to decide how much peer group you want to have based on the requirement of convergence in your network. So we’ll talk about this later probably you’ll have some questions down the line, but this is the peer group in the nutshell.
Philip Winston 00:12:27 I just want to flag one term you used there, convergence. I am guessing that’s when the routers agree on the routes.
Vinay Tripathi 00:12:35 That’s absolutely right. So when there is any change in the network, all the devices, so just I explained to the process of best path algorithm. So all the routers in the network have to do the same thing and they all have to agree that okay they are all done. The moment for example, the trigger comes at T zero and your computation happens at T20, T20 will be your conversions time.
Philip Winston 00:13:01 Great. So that was a little bit about BGP and again Episode 468 has some more details. Let’s move on to routers, which we’ve indirectly been talking about. How has router, hardware and software changed say in the last 10 years or five years? Sort of what are we looking at that’s going on in that part of the world?
Vinay Tripathi 00:13:23 Okay, so I can just go from my own experience where I’ll just go 20 years ago. So when I started my career, there was one of the biggest router launched at that time by Cisco. It was called Cisco CRS, Career Routing Service router. It was multiple chassis router. So one router will have multiple chassis to get more bandwidth out of one router. And that router was designed for service providers because you are at the central of location, you are serving a lot of customers. And the challenge was with those routers where they’re very costly and they will consume a lot of space, lot of power. And as we progressed, we started realizing especially when data center evolution started, the network growth was exponentially increasing. So when the growth was at that high rate, there was consistently a battle between the power and space requirements.
Vinay Tripathi 00:14:26 So that means the networking vendor had to innovate and bring down those routers to smaller sizes, higher throughput and less space and power requirements. So that was the time when everything was going smaller and efficient and less power consumption. And that is the trend even today it’s still everyone wants to have smaller box with more compute power and less energy requirement. And then in the same data center evolution thinking you would’ve heard the virtualization. Where at some time hardware were becoming costly for data center requirements because you had to deploy thousands of nodes in your network. Now typical routers which were made by Juniper and Cisco at that time, they were very costly. So to bring down the cost, it was the cheaper boxes where everyone started evaluating cheaper boxes. That is where this, so router was split.
Vinay Tripathi 00:15:30 So router had typically control plane management plan and data plan. So if you go back a little bit five years or even more actually, so this control plane and data plane separation started, and it became you can have your own server and then you can run your own operating system on that. And you are basically software control. Pen is just software. So you can develop your own software, you can use the cheapest hardware and make it work for your requirement. So then that came into picture where you have virtualized, it’s called virtual machine VMs. So you have VMs and you run it on servers and you get the same thing. Same thing in the sense of control plan. You don’t get the same performance for obvious reasons, but for the data center requirements, most of the cloud providers, they have their own way to develop these control plane data, plane suppression and make it work for their network. And then I think next to that came this STN, which we’ll probably talk later part of it, which further took these control planes away from the devices and took it to the further upper layer.
Philip Winston 00:16:43 Yeah, you’ve mentioned the difference maybe between an individual organization and a data center run by a cloud vendor. And we’re going to talk a little bit more about that. So let’s just talk a little bit more about routers and then we’ll jump to SDN which you mentioned and then we’ll talk about optimizing BGP and how software developers can get involved in this. So we’ve only really talked about BGP as a protocol, but I gather that routers might be running many different or several different protocols simultaneously. Is that true?
Vinay Tripathi 00:17:15 Yeah, that’s true. So depending on where those routers are located, if they’re in the service provider or they’re in data center or enterprise, there might be one or more protocols. Usually you’ll see in the service provider network service provider routers are more feature rich and they often run various routing protocols on the router. Because when you receive those BGP routes from the external peers, there has to be a way to make the protocol next top of BGP prefix to be reachable. So that is where all the supporting protocols IGP come into the picture.
Philip Winston 00:17:55 Okay. And you talked earlier about the control plane versus the data plane and I think you mentioned in a virtualized world the data plane is not maybe as efficient. So I gather for a physical router, the data plane is highly efficient. Is the idea that the control plane is when we’re changing the configuration of things and the data plane is we’re just cranking away on one configuration?
Vinay Tripathi 00:18:19 Yeah, so control plane is mostly uh, brain of router where all the calculations happen and all configurations are put in place. Data plan is where all final decisions are put in place. For example, what firewall filters you want to have, what is the best route to reach there? Those decisions are pushed from the control plane to the forwarding plane and forwarding planes have usually if you talk about the vendor devices, they have specialized chips in those line carts, they’re called line cards forwarding plane and those line carts will have specialized chips which make switching very faster with all the various features available.
Philip Winston 00:19:01 So just two more questions on routers and then we’ll talk about SDN. So you’re talking about VMs, is that the same as network function virtualization or what does NFV mean?
Vinay Tripathi 00:19:14 Yeah, it’s same thing. Basically you take out your control plan and you virtualize it. So you don’t have to be dependent on one vendor. So you can imagine a scenario where you have hypervisor hypervisor, you can use it as a underlying hardware hypervisor. And then on top of that you can run multiple VMs for example, you can have Juniper, Cisco or your own virtual machine VMs developed by you which you can use for control plan programming or management.
Philip Winston 00:19:49 Okay, let’s talk about SDN. So what is A SDN?
Vinay Tripathi 00:19:55 SDN is a Software Defined Network. So it’s, I think we are at the natural discussion and naturally going. So after VM it comes naturally. So what happens is if you look at the routers as I explained earlier, ? So nodes are routers, routers are made out of control plan, data plan and management plan. So when in typical router each and every router has to do independent calculations, independent computations and take decisions, they are not taking the whole network in account when they’re taking any decision. For example, when BGP calculates the best path, it doesn’t take into account how much bandwidth is available in the network, if there is any device in the network down, things that or any interface flapping. So the problem is everyone is making their own independent decision and in that situation, what happens, you might not be using your network as effectively as you can.
Vinay Tripathi 00:20:55 So that is where STN comes into picture. Now what you do is basically you take out the control plane, which is brain of the device to one layer above, which is one common brain for your whole network rather than every device having its own brain now, that brain or the controller SDN controller, it connects to all the devices, it gets the data from all the devices in your network and it becomes the best point to take the decision on behalf of things happening in your network. For example, let’s say your network is huge Microsoft, Amazon or any similar organizations which are globally available. And some event is happening in one of your metros which is impacting the network. Now if you have STN controller, your STN controller has a view of global network, it knows what’s going on at per certain metro or pop and it can take decision such as disable that device, disable that line card or take out that metro from the network as a whole. So what happens is basically you have more control over your network in terms of efficiently using all the resources and also it gives you more programming and management capabilities because when you use a tn, you take out the management plan and the control plane out of the device and you use all the nodes just as a forwarding plan. So you take the decision in the SDM controller, you push those changes on the devices and traffic flows based on those final decisions.
Philip Winston 00:22:32 So can you gimme a specific example of something you personally used to have to do related to network configuration and how that changed with SDN maybe something that was painful and is now easier?
Vinay Tripathi 00:22:47 Yeah, so some decisions are easier to make through SDN as I explained earlier, specifically mitigations network mitigations. So if some event is happening in your network, in the traditional network what will happen is if there is an incident happening in the network, there is an alert created for that and then that alert creates a bug or whatever system you have in your company and then it’ll be assigned to some human, that human will go look at the device, analyze it and take the action. Either it’ll disable the device or move the traffic from the device, whatever is the policy of the organization. Now when the SDN is used for the same purpose, the whole process, what I explained is automated now, so the incident happens, and that incident generates alarm. That alarm is fed in your TN controller pipelines and that alarm goes there.
Vinay Tripathi 00:23:42 Now there are different type of implementations. Some places organizations still want to have their control so they don’t want the SDN to take automatic decisions, they wanted people to review it and then take actions and other places you’ll see that SDN is capable enough or autonomous enough to go and drain that device or take out the device from the network and alert the team that okay I have taken out away this device. Now humans can go look at it and analyze if it is the false one, they can take action based on the event happen. So similar I have seen, and I have been part of similar things in the past. So this is one example, there are more examples this for example BGP. So BGP is running in the edge as I mentioned. So your edge networks connect to the different other autonomous systems, not those devices are running BGP.
Vinay Tripathi 00:24:39 So when you have STN controller, you have more control on those BGP sessions and BGP configurations compared to the manual one because your every second counts in the larger net networks. So look at this. So when you have a large network, you have multiple path to reach the whole internet. So even if the STN can go and disable one interface, it’s actually not going to have a large impact on the network because you have different, but if you go on the other side where there was no action taken and there was outage happening, that is more concerning than having one node or link removed from your network based on the event, if that makes sense.
Philip Winston 00:25:21 So how is this SDN controller interacting with the routers in your network and what if you have a variety of types of routers or even different brands, how does that work?
Vinay Tripathi 00:25:34 So there’s a protocol called open flow which is used communicate between the forwarding plane and the steam controllers, which is used for programming the flows. Now the challenge is with the various vendors because every vendor can have different way to allow these and there are multiple other ways to basically there is something called open config also used which is pretty standard networking now, I think most of the vendors are supporting that to allow SDNs to communicate the configuration changes directly to those devices, doesn’t matter who is the vendor. So there are certain protocols open config and OpenFlow which are helpful in that when you’re dealing with multiple vendor environments.
Philip Winston 00:26:21 Okay, open config and OpenFlow. How about the network operator that’s interfacing with the SDN? Are they using a GUI or are they writing scripts maybe that interact with an API and as an example of that, how would you deal with a version control or rolling back you’ve made a mistake in your configuration and you want to roll back? How does that work?
Vinay Tripathi 00:26:49 So the version control is pretty much standard just we do for any software related, itís centralized and it depends on the organizations, how they are going to use it, every organization as they’re on. So one thing in when it comes to STN, you’ll mostly see the larger organizations who have expertise in software engineering only they are the leaders there because to run the STN you need both expertise, network engineering expertise and you need software engineering expertise as well. And to the second part of your question, how it is configured. So that is also actually I have seen all options. there is a GUI, there is CLI also actually three options and even you can have command line so you can through command line, or you can use GUI to push the configuration changes and if you see the most sophisticated deployments of TNS, you actually don’t have to do that. They’re pretty much automated to take actions by themselves. So there are tools which can complement them in the scenarios where you want to take action in case something is going against the business rules, that is where you go and use those override CLIs or . But otherwise you will see they’re pretty autonomous.
Philip Winston 00:28:06 Okay, let’s talk about the rise of cloud computing and this move from individual organizations to cloud vendors that might be hosting software applications from many different companies for them many different companies. And then we’re going to talk about optimizing BGP and how that relates to software development. So for cloud computing, you’ve mentioned data centers a few times, can you just kind of summarize what’s different about a cloud computing vendor running BGP compared to an individual organization running BGP? What changes?
Vinay Tripathi 00:28:47 So BGP is used at multiple places in the network. So when BGP is used to talk to different autonomous systems Facebook talking to Microsoft in those situations it doesn’t matter, they’re same but the change in the cloud is mostly when the use of BGP within the network. So that is where you’ll see BGP is used at different places such as some cloud providers use BGP between the controllers and the routers themselves to get the information about the network topology and BGP is also used internally to communicate your internal routes to your SDN controller to feed those systems. So these are some places where BGP is used internally but externally I think that remains same as service providers.
Philip Winston 00:29:37 So as we said, a notable thing about cloud vendors is they’re running infrastructure for many different companies. I think this might be called multi-tenancy. What challenge does this present? I know there’s the noisy neighbor problem but as a cloud vendor what do I have to worry about GI given that I’m handling routing for many different companies?
Vinay Tripathi 00:29:58 So when you have multi-tenancy environment in the data center, especially complicates routing because you can imagine how many thousands of VMs running in the cloud and those clouds are mostly private IPS are given and each VM can have duplicate, each VM can use same IP address. Now the challenge is how you make sure they don’t conflict within the same data center or within the same network. So that is basically there are two types of routings underlay routing or layer routing. So the underlay routing is the one which connects all your physical infrastructure and all the nodesí routers, and the overlay routing is, which is used from the edge sometimes from the edge to the server directly, which every organization has their own way of encapsulating, and I haven’t seen much standards there. So people use VXLAN for example in their data center, but they customize it for their own use, and they use that for, so let’s for example, let’s say the traffic is coming from remote network, it lands to your edge network now your edge network has to send this traffic to your data center server. So now edge network will do all the encapsulations for you and that is where all the encapsulations are happening, and proprietary outer headers are added in the packet, so the packet reaches to the destination. So that is what I’m calling as an overall network. So they are pretty complex. We can go in detail, but I think that be out of scope for this discussion.
Philip Winston 00:31:33 How about just the scale of these cloud vendors? How do they scale up the amount of bandwidth, the total amount of bandwidth and the ability to route that much data? Is it simply a question of scaling horizontally and buying more and more routers or is there something else that’s a standard technique?
Vinay Tripathi 00:31:55 Yeah, so that’s very interesting question. So I think that that is a consistent battle in the Cloud network. So you can never have unlimited space and unlimited power and unlimited money also. It’s ultimately money. So there is one more angle into that. It’s called optimization. So everyone wants to have most reliable network? But then you hit the wall of scaling and that is where the optimization comes into picture. Now you want to get most out of your network. That is where you want to have, you want to optimize your network at the hardware level, you want to optimize your network at the configuration level, you want to optimize your control plane and you also have to want to optimize your operations because that is also very important part of it. So you basically, you optimize single device to get most out of it and then you do optimization at the, for example STN controller or your systems which are monitoring your network to do further optimization. So your net network and the parts are efficiently used end to end.
Philip Winston 00:33:05 Well I think you’ve introduced the next topic which is optimizing BGP. So this is where we want to talk to software developers that are out there who may need to be aware of BGP optimization depending on the type of application they’re working on. So let’s start there. Why do software engineers need to be aware of BGP or in what situations do they need to be aware of it?
Vinay Tripathi 00:33:32 Yeah, software engineers who mostly interact with the network, they have to be aware mostly so when in the cloud networks or the networks which have SGM controllers in picture, they often work very closely together to make networking happen. So I’ll give you an example. So in some part of network, let’s say you are deployed as TN, which is managing the device, and you have the software engineering capability in your network to do that, make it happen. So you can program it the device, it’s okay, it’s working fine, that’s fine, but if you don’t have the expertise of networking there, it’s hard to understand how I design the reliability. How do I design the failure scenarios and if I am doing it properly or not? So that is where I have spent a lot of time working with software engineers and vice versa in solving these problems which are mostly reliability and the scaling.
Philip Winston 00:34:37 So let’s flip it around, what can go wrong if we don’t pay attention to BGP? Can you give an example of a problem that you encountered that turned out to be the result of maybe a misconfigured or less than optimal BGP setup?
Vinay Tripathi 00:34:55 Yeah, so I think BGP by itself is very infamous for these types of incidents where you can have bad network outages which are attributed to configuration issues or some malformed packets. So I have seen one such incident where a malformed packet BGP update received in the network and that was propagated throughout the network and then that was causing high CPU utilization on certain devices. Now how was that found? So when we started getting a lot of incident alerts, we started looking at it, what’s going on? And we saw that certain devices in the network are having high CPU utilization. Went there and started troubleshooting. We found out the root cause was basically one route which was slipping through the routing policy, and it was making through our network which was causing this problem because the attributes that route was not understood by the routers operating systems.
Vinay Tripathi 00:35:56 Now that was a root cause. Now it came to how can we mitigate these problems? So that is where our interaction with STN teams or the software engineering starts, that can be mitigated by looking at the routes we are receiving from the external peers. So we understand what type of attributes they are carrying and if some peers are advertising that prefixes or routes with attributes which are invalid, we can shut down those peers. So that call can be taken by the TN controller or automation systems deployed in the organization and that is where the transaction between software engineering and network engineering happens.
Philip Winston 00:36:38 So if it’s afull on outage, apparently the users can’t even access the application at all. But what other end user experience problems can there be, I guess performance, would that be latency or maybe you can name a couple problems that could be the result of BGP configuration.
Vinay Tripathi 00:36:59 Yeah, so there might be as you said, latency and then the other one could be the total blackout. You cannot access the application anymore. Then you might even see depending on the application you are using, if it is video application, you might see the buffering at the end user if you’re using the text or wise total loss of the packets and you can experience it in your call. So I think if you attribute them it’ll go to the latency and black hauling traffic, black hauling.
Philip Winston 00:37:32 You may be mentioned this a little bit, but let’s go in deeper on how you would identify A BGP related performance problem or outage. I guess what tools or techniques would you use?
Vinay Tripathi 00:37:46 There are many techniques used for monitoring BGP and the network in general, but I would talk about one incidence of convergence, which we talked earlier because convergence is something which can impact the end users experiences and especially in the cloud networks where customers are more connected to your network, they have more access to your network, it’s more visible. So basically the longer your BGP convergence or the network convergence is longer impact will be seen by the customer. So there are different variations. So some convergence are acceptable in the network there are certain time of convergence which are, which are network provider and then customer agree that if the incident happens, this is the expected convergence time which is basically you can say the out time and if you have more than that agreed time, that is a bad time, bad conversions and that bad convergence is the one where you the loss for the service provider and even the end customer who is providing the service.
Philip Winston 00:38:56 So we talked about SDN, if we’re using a monitoring or analysis tool, is that just talking through the SDN or would it go directly out to the routers and and gain information that way?
Vinay Tripathi 00:39:10 Yeah, so it depends on how the tool is design designed. So there is telemetry is often used to collect the data from the devices and that similarly SN MPS and other protocol which is used to collect the data from the routers and then these data are fed to the steering controllers and then as TN controllers use that data to make the decision.
Philip Winston 00:39:32 I think you’ve mentioned overall network health beyond just BGP. Can you give, one or two examples of what an unhealthy network is from the inside? I know maybe as the end user we’re experiencing either an outage or delay, but what sorts of metrics or analysis would lead you to believe a network was not healthy?
Vinay Tripathi 00:39:57 There are a couple of things. One thing is we talked about the latency part. Other thing is asymmetry, user can measure from their end that the traffic going and coming back is taking different paths so you can experience the latency because of that. Also third part is partial outage, uh, partial loss. So you might be sending a hundred MBPS of traffic but only 90 MPS received. That might be because some policy or some policies they’re deployed in the network which is so basically the user is violating that policy and because of that the partial data is getting lost. So that is one more thing which comes into my mind. Other than that, another challenge user can experience especially in the cloud world is you want to scale your network or your services, but the scaling limit is getting hit because of the service provider not able to scale their physical infrastructure in the short term.
Philip Winston 00:40:59 Let’s talk a little bit about the boundary of responsibility between a software engineer who’s dealing with an application that’s sensitive to the network configuration and a network engineer and maybe a DevOps. I know this might vary a lot from company to company, but can you kind of explain where each of those is focused?
Vinay Tripathi 00:41:22 Yeah, so usually uh, it’s pretty clear responsibility what network engineers are owning and what is software engineer and then sometimes you’ll even hear something called DevOps which are in in the middle and in the cloud networks you will see that role also. So when it comes to network engineering, so the network engineers are mostly within the network engineering, there are multiple sections, uh, or responsibilities, but we’ll only focus on engineering. So network engineering is mostly related to the security scaling reliability. So you want to make sure that your network is secure and reliable and it’s scalable on demand. So as a network engineer, I would spend most time in optimizing in this direction at the device level and at the configuration level at the management level of those devices. But when it comes to the STN or software engineering who are owning the STN, they’ll be mostly dealing with the data collection from the network, how to measure you those decisions, so basically take the business logic from the business, what is the requirement from their end customers.
Vinay Tripathi 00:42:38 And then you have the data from the network that is basically what is the available bandwidth, what are the available resources, how much capacity we had in this region, things like that. And then you take decision based on that business logic and you pass on those decisions knowing that what is the capacity available to the networks. So that area is the basically the software engineer engineers part where they can own the configuration, sorry, provisioning monitoring and the mitigation. All these three-parts network engineering part will be very clear what I explained, but sometime there is a gray area where we both have to sit together and find out what is the real value for when we are monitoring something, what should we monitor. So you are not monitoring something incorrectly. So those are some areas where we both are kind of owning it but not owning a gray area. Kind of something
Philip Winston 00:43:37 You mentioned security, I think we might do a future episode on BGP security because I think there’s a lot of different facets, but can you just name one or two things specific to BGP that could become security issues? Just to give us a sense.
Vinay Tripathi 00:43:55 So there is multiple level of security which you can configure for BGP. So one is to BGP has authentication which you can enable for that particular BGP session, which is typically MD five is used or authentication. So unless that check authentication check is passing between the two peers, the session itself will not come and then the second and there are more security features available to validate the if the route particular route has been originated by the authenticated au autonomous system or not. Similarly there are wastes which is typically RPKA it’s called and then you also have a way in BGP to verify the whole path for the prefix is valid or not. So I think it’s used something called BGP SEC which is used for that purpose. So I’m aware of these three uh, security mechanism for BGP.
Philip Winston 00:44:51 Well how about summing up BGP optimization in terms of the business case, have you ever had to sort of sell the importance of optimizing BGP or had a discussion that was kind of at that business level and how do you frame that importance of it?
Vinay Tripathi 00:45:10 Yeah, so I have worked on various BGP related issues and something on top of mind is related to conversions which is, you if you BGP is always scaling and scaling is always conversions, they’re related. So I was working for convergence related issue for one of the customers and I was supposed to bring down that convergence by X percentage which was agreed upon by the customer. And as a part of that exercise I was supposed to basically do the research in the network, whole network, look at the configurations, the pairings, the path counts and then look at the platforms used in the networks and then come up with the plan to reduce the optimization by whatever percentage you agreed. So as a part of that exercise I did my ion was to basically start optimization from the device level and then till the STN level.
Vinay Tripathi 00:46:15 So we started from the device level in the analysis we analyzed, we figured out that device platform which we are running legacy device which we can upgrade and get the new hardware which has more computation power and memory which is required for BGP routers and then there are suggestions to improve the configurations of the devices so we can do faster processing of the updates. And then the third part of the session was to work with the software engineering team, and which was used for provisioning the BGP sessions with the routers and reduce the BGP sessions from whatever number was there, let’s say call five to one session. So that is where I was interfacing with the software engineers. The challenge was how do we reduce from five to one by not losing the redundancy of the session? So there I and the software union team, we had discussion with the way forward and we agreed on making changes at the protocol level and as well as the software level.
Vinay Tripathi 00:47:22 So the protocol level change was to basically enable a graceful restart at the STN controller and the change in the software was to instead of having all the five VGP sessions with the routers, create the application in such a way that the task can be running in the background as a backup which can spin when it fails and which can work together with a graceful restart so you don’t lose the routes during that time. So even when you establish the new session, you can still have those routes, so you don’t have to basically go through the conversions for that particular session. So that helped us brought down the number of paths on those devices. So the end result of the exercise of the optimization was to, we had less number of BGP parts on the device. We used the latest hardware which can do faster calculation and we had the better component design at the TN level which was helping in this basically reducing the number of BGP sessions. With that, those devices,
Philip Winston 00:48:27 I’m wondering about the amount of bandwidth involved in whether it’s a software virtual machine router or a physical one. Are we talking about gigabits per second or terabits per second? what sort of volume is a modern router capable of?
Vinay Tripathi 00:48:45 Yeah, they’re in terabytes actually and now they are even one line card is terabytes
Philip Winston 00:48:51 And one line card is what? One part of a router?
Vinay Tripathi 00:48:54 Yeah.
Philip Winston 00:48:56 I guess why is it called a line card?
Vinay Tripathi 00:49:00 Good question
Philip Winston 00:49:02 WellÖ
Vinay Tripathi 00:49:03 Don’t worry. I don’t know if I can answer that.
Philip Winston 00:49:06 So in that case you mentioned upgrading the hardware. Do you have an example from your own experience or otherwise where a network was underperforming due to BGP configuration, and you were able to speed it up without any hardware changes? I’m just wondering how badly could a network be underperforming if you can measure that or give a sense of how much you were able to increase the speed by?
Vinay Tripathi 00:49:34 Yeah, so I’ll take one of my examples where I was working with service provider customer. So they had a BGP conversions problem, and they were running legacy devices, and they could not upgrade the devices for next two years because of the cost of their budget for the uh, next two years. So the challenge was to how can we give, optimize their device to get most of it out of it. So for that first challenge is to how do you measure that? What is the current conversion? So there are multiple ways to measure that. One of them is if you have access to the customer device, you can configure and their BGP session which can receive all the routes and you can monitor that particular session for the BGP updates. And when the BGPQ is empty and when it becomes busy, that is a time called conversions.
Vinay Tripathi 00:50:30 Now once you have that data, you know the convergence time, now you can go and do the optimizations. So in this case where customer was using legacy devices, my recommendation to them was to go start with the configuration optimizations and adding the new features. So one config optimization was something which we talked earlier peer group. So peer group you can optimize, you can reduce the peer groups in your BGP configurations and club more number of sessions under one peer that will reduce the number of PGP uh peer group in your device. That means device has to spend less time processing when it is sending the routes. So that gives pretty good outcome when you are implementing it. So in this case we saw the conversions was improved by 40% just by reducing the peer groups from 50 to 25. So that was pretty decent, convergence improvement for that customer.
Philip Winston 00:51:31 So it is possible to say double your performance just through configuration changes.
Vinay Tripathi 00:51:36 Yeah, and there are scenarios where you can actually, so this was not the old platform I was working with one of the customers when I was in Juniper and they had a new hardware and a new feature related to BGP and that time that feature, if you enable that feature on the BGP, it’ll basically split your routing tables in multiple routing tables and to make the update process faster. And it was actually improving the PGP performance by 90% with the same configuration, no change in configuration. So uh, combination of new platform and new features can have a very positive impact on the device level and the network.
Philip Winston 00:52:17 So we’re going to start wrapping up, but I guess I’ll ask for one more story. Do you have an example of a purely organizational problem where it really was just communication between people or some type of problem that was solved strictly by talking to people and working it out and not really a technical problem.
Vinay Tripathi 00:52:39 So I worked in that type of role when I was in Juniper as resident engineer where most of the time I would spend, so I was the resident consultant for Microsoft at that time and we used to work on situations where customer comes with the complaints and those complaints are actually not the technical issue. Sometimes it is customer’s own issue but because they don’t have the experts in their network which are capable to handle those platforms. So they always want to reach out to you to consult with if this is a false alarm or is it a true issue. So it happens quite often that in those roles you’ll have to deal with situations where you’re dealing with the customer management rather than the technical problems. So one such problems happen in my resident consultant role, I would often have to deal with customers where I had promised the release, a software release at certain dates, but because of last minute bugs arrived on that release, we couldn’t ship it.
Vinay Tripathi 00:53:48 Now that is where you have to basically work with customers and that is where your relationship with customer comes into play, how much trust you have developed with the customer. So in that situation, basically I had to manage customer and to explain to them, be transparent that okay, this was the reason, and we want to give you a good quality of release and we don’t want to shape something which has the defects which can impact your network. And based on the conversation with that and the last past relationship with the customer, they agreed to delay that software release and managed with the current release for another three months.
Philip Winston 00:54:26 Okay. Let’s wrap up. I wanted to ask about what’s next for SDN? That seems a big step forward compared to individually configuring routers, but what problem is left to solve there? What’s still a pain point that you’d like to see addressed?
Vinay Tripathi 00:54:46 SDN has, if you compare it with routing world, where we have protocols governed by ITF SDN has pretty less standardization in in that sense. So everyone uses their own ways to implement except the talk talking to the devices, which involves the different vendors. So I think maybe that evolution has to happen over the period of time and probably less software engineering expertise required where you can have a controller which can work across the different platforms. And also something is on top of SDN is called Self-Driving Network and it’s called sometimes intent-based networks, which is many organizations have it already, but it’s not easy to implement by all the customers. So I hope sometimes it’ll be accessible, and everyone can be able to benefit out of that.
Philip Winston 00:55:43 I was going to ask about intent-based networking. Can you explain that a little bit more? Is that a technology or is it just an approach to using an SDN? What is intent based networking?
Vinay Tripathi 00:55:55 Yeah, so intent based networking is just another layer on top of SDN. So SDN is a controller, which can be called by itself or by another API. You can trigger events based on that which can be pushed to the devices. But it is still not completely autonomous by design. So when you add the intent based network, what you do is basically you define your organization policies. For example, for this customer or this type of customers, this type of SLA has to be there. So that SLA could be no, latency acceptance of this range and bandwidth committed of this range. Things that you can define these are the business logics which you can feed to the intent based network. And then intent-based network can itself call STE controllers to make the changes in the network. So basically you, the whole network is driven by itself. So sometimes it’s called self-driven network also.
Philip Winston 00:56:56 Okay. How about a message to leave with software engineers who want to become more networking aware. Maybe they’re not going to go get a networking certification, but they just want to learn more about it. What would you point them to for resources or just an approach to, to learn more about networking?
Vinay Tripathi 00:57:16 So I think yeah, software engineers don’t have to be as fluent as network engineers for sure. So there are so many resources available on the internet to understand basic TCP IP level and once you understand the BC uh, TCPI IP layer, then it’s easier to develop the further expertise in the protocols depending on what you are dealing with. Mostly I have seen BGP is pretty common one and then IGPs are something which is organization dependent. Every organization has their own choice of IGP. So my recommendation will be to teach, to have sound knowledge of TCP IP and basic knowledge of BGP, which is enough to survive in cloud networks at least.
Philip Winston 00:58:06 I’m going to try to put a lot of these acronyms into the show notes so people can look them up. Sure. So I think we’re just about done. Where can listeners follow you or get in touch or what would you to point them to?
Vinay Tripathi 00:58:22 I am on LinkedIn, so anyone could connect with me. I would be more than happy.
Philip Winston 00:58:28 Okay. I’ll put your LinkedIn in the show notes. Thanks for talking with me today.
Vinay Tripathi 00:58:33 Thank you Phil for hosting me. I appreciate it.
Philip Winston 00:58:36 This is Philip Winston for Software Engineering Radio. Thanks for listening.
[End of Audio]
