Search
Search
Chris Love - SE Radio Guest
July 23, 2025

SE Radio 678: Chris Love on Kubernetes Security

Chris Love, co-author of the book Core Kubernetes, joins host Robert Blumen for a conversation about kubernetes security. Chris identifies the node layer, secrets management, the network layer, contains, and pods as the most critical areas to be addressed. The conversation explores a range of topics, including when to accept defaults and when to override; differences between self-managed clusters and cloud-service provider-managed clusters; and what can go wrong at each layer — and how to address these issues. They further discuss managing the node layer; network security best practices; kubernetes secrets and integration with cloud-service provider secrets; container security; pod security, and Chris offers his views on policy-as-code frameworks and scanners.

Brought to you by IEEE Computer Society and IEEE Software magazine.

Show Notes

References

Related Episodes

Transcript

Transcript brought to you by IEEE Software magazine.
This transcript was automatically generated. To suggest improvements in the text, please contact [email protected] and include the episode number and URL.

Robert Blumen 00:00:19 For Software Engineering Radio, this is Robert Blumen. I have with me Chris Love. Chris works at Modernize as a distinguished engineer. Chris and Jay Vyas are co-authors of the book Core Kubernetes, and Chris is a Google Cloud certified fellow. Chris, welcome to Software Engineering Radio.

Chris Love 00:00:40 Thank you Robert. Really appreciate you having me come and speak with you folks today.

Robert Blumen 00:00:45 Happy to have you. We’re going to take advantage of your being here to talk about Kubernetes security. Now, Kubernetes is a distributed system except external traffic. It uses compute resources. Is there such a thing as Kubernetes security or is it just about following best practices that are well known for these types of systems?

Chris Love 00:01:09 I think it’s both. I think of course it’s best practices like update your software dependencies, update your dependencies from your operating system. It’s that, but now you’re running a containerized system, so you have to remember to update the dependencies in the container as well as update your host version of Bash. So of course there’s intricacies to it. We’re running a nice distributed system that allows us to do complicated stuff like scaling and we have failover, but because of that, we’ve got a little bit more complicated networking that can cause some challenges from a security standpoint. But we have other distributed systems that we’ve been using for a while. Most of them are based around containers, but there’s definitely some wrinkles. But like you said, at the end of the day, it’s an API layer. You’ve got a bunch of compute nodes, which are either your servers or your EC2 instances or your GKE instances and you’ve got a bunch of containers running around. So I would say it’s non-trivial, but it’s not rocket science. It’s not as challenging as catching a rocket with a couple chopsticks.

Robert Blumen 00:02:15 The overall umbrella of Kubernetes security, it has many subtopics more than we can cover in an hour. If you were to pick a few that are most important to focus on, what’s your short list?

Chris Love 00:02:28 I would try to group them into, I kind of go from a large world internal, so we could talk about what to expect when setting up a Kubernetes cluster. So overall security, then you can go down to a no level security from there, network security, from there pod security, and from there container security. Container security is well documented, but I think some folks either don’t have the time and money to put those in place. Operating system security, I’m not going to talk to you about. There’s lots of other references that folks can go to that I always look at myself as a Lego engineer, right? We’ve got building blocks. Some are unique to Kubernetes, and like you said, some operating system security is operating system security, but typically on an operating system you aren’t running two different network layers and that’s what you get within Kubernetes.

Robert Blumen 00:03:20 So it’d be a good time for me to let the listeners know we did an entire episode on Kubernetes Networking number 619. We will return to that a little bit later. Let’s go down your list, Chris, and hit these things in order. Starting with setting up Kubernetes overall security, what are some of the main points that should be addressed?

Chris Love 00:03:41 Right. You want to think about from a network layer, from a node setup layer, and from an overall like account permissions layer. Again, if you’re running in a data center, this is a little bit different, right? But I would say majority of people that are running in Kubernetes are running within AWS or GKE or Azure or pick your Cloud provider. So there’s always some gotchas around those Cloud environments. For instance, you want to make sure that the role that you’re setting up your cluster with and the role that the cluster’s running with are the correct roles. You don’t want to set your cluster up with a role that is an account level admin. You want to give your Kubernetes cluster and your Kubernetes nodes the right level of permissions. So that means setting up a user before you set up Kubernetes. From there also, look at a private network.

Chris Love 00:04:33 Don’t expose your nodes to the public. In other words, Port22 on node A should not be accessible via an external network. You’re going to need to VPN into your nodes, and realistically, developers and admins shouldn’t necessarily have to at a node level. Also, your API layer or web API should also be behind a firewall that should be networked in. It should be on a private network where folks aren’t able to access it. And people, we’ve had bugs in Kubernetes where authentication for very short amount of time was broken at the API layer. And fortunately, the folks that maintained Kubernetes fixed it pretty fast. But it was overnight where if you had a publicly exposed API and you had X, Y, Z version of Kubernetes, people could just do a coup control command right to it. So doing some basic setup before you are thinking through your security model and your security setup before you set up a cluster is really important. IP spacing, for instance, that gets folks in trouble as well. So you want to make sure the subnet you’re running on is private.

Robert Blumen 00:05:40 Main points there would be use the Cloud service providers access management. Use a private VPC and put your whole cluster behind some kind of firewall or proxy to cut it off from the web. Did I miss anything?

Chris Love 00:05:55 No. And then you’re looking at some form of public ingress, right? You’re looking at having to maintain some form of ingress controller that’ll allow external traffic to go in internally into your cluster, which makes it a little bit more complicated, but I’d rather not have nodes exposed to the internet personally.

Robert Blumen 00:06:11 Would any of the last set of answers change? If you are running the cluster on the company’s own computers,

Chris Love 00:06:19 You can think about it in the same way, right? You would have some sort of router set up where you again, have internal IP addresses and external IP addresses. It’s just more work. You have to set up your air quote, VPC network or network that’s internal by hand as and maintain it. And you can’t just make a terraform call to your Cloud provider’s API to set it up for you. You have to have yourself or somebody else set those routers up. But same type of model exists. Networking is networking, is networking. We have an IP addresses here and there.

Robert Blumen 00:06:53 Do you have any stories from a time you were setting up a cluster, something you missed or from auditing another setup where you found something that needed to be locked down?

Chris Love 00:07:04 Yeah, many of those I’ve found, pretty much everything that I’ve talked about set up incorrectly. A lot of the times people set up their cluster and expose the API server and/or the nodes or both publicly. It’s more complicated to set up a VPN type mechanism to get into your nodes. It’s much easier not to have it that way, right? As well as I’ve seen many clusters that were created using, like I have a Cloud access, a Cloud admin account, and I incorrectly set up my cluster and it is using Chris Love’s Cloud admin account. That’s not good. Being able to have a Kubernetes cluster creative VPC or create other components isn’t a great way to do security within Kubernetes. I am both authorization and both off and off Z are complicated enough. You don’t want to add a layer where you’re running your cluster as a Chris Love user.

Robert Blumen 00:08:04 I’ve set up a cluster, I went through my checklist, done all the things we talked about. Are there any either open-source frameworks or testing tools that would run through and verify that the things that need access have and the things which should not have access cannot?

Chris Love 00:08:21 I believe the CNCF has a testing tool that goes through your cluster soup to nuts, kind of gives it a once over. Also, frankly, a documentation on Kubernetes. Security on the Kubernetes sites, tremendous. So I’m going through it almost drills down in the same order that I’ve been using in a long time, and I can’t say I’ve written any of the security documentation there, but it has a checklist of what to do. You’ve got tools that are owned or been donated and maintained by other companies that CNCF have, and there’s plenty of companies there outside that can either help you set them up or test them after the fact.

Robert Blumen 00:08:58 Sounds great. I think we can move on to the next layer that you address, which is node security. What are some of the common attacks at the node level?

Chris Love 00:09:09 Again, it’s instance or server level security. It’s SSH attacks, right? Fortunately, our modern-day Cloud providers give us good authentication methods for SSH. That is not a username and password, but again, it goes back to your maintaining nodes. I would say this, unless you’re on data center, you’re not looking at in-place upgrades. And if you’re running in your own data center, you could be running virtualization as well. And if you’re virtualized, you have your own Cloud. We’re looking at doing rollovers of an operating system to upgrade your OS. We’re also looking at running immutable operating systems, which is another thing that I highly recommend. Running operating systems that have read only components. It allows you to not have bad guys overwrite binaries. That’s just a bad thing. But again, in terms of how Kubernetes upgrades, rather than upgrading your OS, you need to upgrade a node.

Chris Love 00:10:11 So upgrade your nodes through a rolling upgrade, and then you update your operating system. If you’re running on bare metal without virtualization, then most likely you are looking at in place upgrades. But then you can shift around your workloads to do that. As we mentioned at the top of the show, OS level security is OS it’s really the, I would say that’s probably the best-known security posture that we know of, right? Because system admins have been maintaining OS level security since the beginning of time, at least the beginning of Unix in the beginning of computers. So that’s probably the best-known security layer or the best-known security practices that we have. Now as we move into the network security and we move into the pod security and we move into container security, I’d say those are all the newer technologies.

Robert Blumen 00:11:01 If I’m running on a Cloud service provider, then they manage the nodes, they will autoscale nodes in or out of the cluster as needed. Can I count on the Cloud service provider to manage the node image and to refresh it if vulnerabilities are discovered? Or is that something that as a cluster operator, I need to push a button or do something when I want a refresh of the node image?

Chris Love 00:11:26 It really depends on how you set up your cluster. First off, you can run your own control plane for Kubernetes. Lots of companies that are a little bit more sophisticated do that. They’ll still run their own master instances with ETCD, et cetera, et cetera. So sometimes the control plane is maintained, sometimes it’s not. I would say the majority of companies that are running on top of EKS and GCP or AKS, they are using a managed control plane. Nodes on the other hand, you often have options that the Cloud provider automatically upgrades for you or you have to upgrade them yourselves. What companies are finding that it’s often necessary for you to upgrade your nodes. Workloads are not typically like you still have complicated workloads that are not, I would say Cloud native friendly necessarily. And because of that, the upgrade process can have a couple bumps here and there.

Chris Love 00:12:22 There’s many companies that still have outages when they do upgrades and you can’t have a Cloud provider automatically upgrade for you necessarily. Nice theoretical world, we’d be able to have that. There’s other packages within Cloud providers where the Cloud providers are maintaining your nodes for you. And then of course upgrades do happen in the background. You’re able to set up, here’s the window, I want my upgrades to happen between 2:00 and 4:00 AM eastern time on Saturdays for instance, or on Tuesdays because that’s the lowest traffic day of the year or a week that you have. So again, there’s options going back to what you’re talking about originally, Robert, those are decisions you need to make before you set up your cluster. Do you want to maintain your own control plane? Please don’t. Please don’t. Unless you really know what you’re doing. Do you want to maintain your own node pools? 50/50 on that depends the amount of staff you have. Or do you want your Cloud provider to completely maintain everything for you? And on top of that, then you’re looking at less work, but you lose control. There are trade-offs there. Definitely trade-offs.

Robert Blumen 00:13:23 Give some more details of what can go wrong during an upgrade. Theoretically, Kubernetes will reschedule traffic off of the nodes you want to remove and onto the new ones. But when does that not work as intended?

Chris Love 00:13:37 Long running jobs, for instance, that don’t restart. So stateful applications that aren’t truly Cloud native. You can run into problems with more complicated stateful applications during an upgrade process. You need to do something to the database, for instance, put it into a certain type of mode before you upgrade, before you evict the pod. So sometimes you’ve got to do three steps before you upgrade and then three steps after you upgrade. So that makes it a little bit more complicated. Older database systems that have been made more Cloud friendly still have some challenges. So stateful applications, long running applications that don’t necessarily restart themselves or you have a 60-minute job you go through an upgrade shouldn’t be upgrading at that time because those jobs are running. Now you’ve got to rerun a 60-minute job and if it’s a critical process, it’s kind of challenging and you can run not only in upgrades, but you can run the same thing when you’re running or run into the same type of problem when you’re running auto scaling.

Chris Love 00:14:37 So if you have a long running job such as a CI build, your autoscaler sizes down the node pool, which in effect is the same type of behavior you do when you upgrade a Kubernetes cluster, you’ll have that job kicked off. Hopefully you want that job to restart, but sometimes they don’t. From security perspective, the upgrade process is important that it goes quickly so that you can remediate security issues quickly. If you’re looking at redeploying workloads, you’re looking at pushing out new workloads. It’s a combination of the processes that you have in-house to run upgrades, how complicated your workload is, as well as if you need to roll out new workload. Say you’ve got a spring dependency issue where there’s a CV in spring and you need to roll out an entirely new application because of that or 40 applications, again, your CI needs to be able to handle that type of rollout.

Robert Blumen 00:15:29 I’ve been involved in database migrations some years ago. This typically involves a lot of runbooks manual steps like you described. Were the three things you do before and the three things you do after. I am aware that there are some offerings in the database world where they’ve created Kubernetes operators that to some extent can take over the role of a human system admin. Are most things in Kubernetes world now aiming to be pretty automated in face of all kinds of disruptions? Or is it still kind of old school where database is going to be migrated and we’re going to have downtime and we’re going to babysit it?

Chris Love 00:16:08 Well, it depends on what you’re running, right? It’s application and operator dependent, not necessarily Kubernetes cluster dependent is the way I’d put it. If the workload can restart itself nicely, you flip a button ham fault tolerance mode and that your fault tolerance works as expected, you should be fine. It’s running the upgrade in dev first to make sure that you’re fine rather than having a 40-step runbook and playbook. Are we there yet? No. If you can afford it, have somebody else maintain your databases for you. Have people like I would say most companies aren’t Google, aren’t Apple, aren’t Facebook, right? They don’t have hundreds of engineers that maintain their databases for them necessarily. If you are that company, then yeah, you might be able to get away with it. The thing with running databases in other stateful applications is you have to understand how that stateful application actually runs on top of Kubernetes really well.

Chris Love 00:17:04 It’s not just I’m running an application and it runs great, it’s, I’m running an application on top of Kubernetes because of the way it fails over. And it basically if it says, okay, you got 60 seconds to leave application ABC or replica ABC is a better way to put it. You have 60 seconds and you’re out of here. I don’t care what’s going on, this node’s going down and your pit is going to get terminated. So your application has to handle that. And realistically though, Robert, your application needs to handle that anyways because it’s the same pattern that happens during outages. Again, it goes back, if your company can afford RDS, why not use RDS? If your company can afford running SQL server on Azure, let the experts handle it, it turns over some control. You have to run their version of Postgres rather than you might have three versions of Postgres that you can run.

Chris Love 00:17:58 So if you need a much older version, then you might be running it yourself. But that’s a whole another story where let’s have Postgres updated in your company. It goes back to time and money, right? The story I tell about security is, you’re going to get hacked. Most likely you’re big, you’re going to get hacked and or people are going to be knocking on your door a lot. But it goes back to business risk and time and money. As much as we talk about these security offerings and what to do, it all goes back to how many hours a day we have to engineer it, how many hands we have on keyboards. And I would say now we’ve definitely have some help from AI. I don’t think it’s quite there yet, but supposedly we’re going to be out of a job here pretty soon, Robert. So I don’t believe that. But that’s a whole another topic that we don’t need to go into. But like I said, it’s all time and money we can do this. It’s not rocket science. There is a broad range of things you need to look at, but it goes back to pretty much what we’ve been saying over its chat.

Robert Blumen 00:18:51 Would you accept one of the standard node images that’s recommended by the Cloud service vendor or would you start from uh, long term or stable release of your favorite operating system and then batten down the hatches to get it even more secure than the off the shelf?

Chris Love 00:19:10 I would say use off the shelf. And the reason being is that they have a staff of a hundred people or more that are maintaining that image. What most companies, they’re lucky if they have two. Most companies don’t have people that write kernel level code and the Cloud providers do. A good example, this happened a long time ago, was the issue with the Intel processors where we ran into some processor issues, CVEs within the operating system itself in the kernel. So the CVE was announced the day after Google released new images that contained the fixes. So we’re talking virtually immediate, it took Debian and Ubuntu a couple weeks to catch up if not longer. So you’re looking at that type of lag with CVEs where AWS, the big folks are talking about the CVEs before you even know about them and they’re coming out really fast with patches where us the same way.

Chris Love 00:20:08 And I’m not saying there’s anything wrong with Debian or Ubuntu or XYZ operating system. There’s definitely folks that even offer operating systems that are built for Kubernetes nowadays. So do you need one? I don’t know. Do you have a use case and is there a business value for your company to have one? Great. Well I run on top of like I do work with images and I get frustrated that there are some CVEs running around in a couple distributions and they say, no, those aren’t CVEs and we’re not going to fix them. A lot of security people don’t like that. But I go to Amazon Linux for instance, and I know if I’m going to put something on Amazon Linux and get it screened, I would say about 90% of the time, if it’s Amazon Linux maintain packages, it’s going to pass screening. Same thing with, I forget the name of the operating system that GCP runs, which is completely immutable by the way, which is pretty cool.

Chris Love 00:21:05 And again, I can’t even screen that in a lot of ways. But again, it’s leveraging you pay enough for a Cloud provider as it is, use their toys, leverage their knowledge as much as you can in a wise way. If you have use cases where you have XYZ software that has to run at the node level and you have to have this, then I understand you got to run that specific operating system. But again, it’s kind of like running your own email server nowadays, Robert, running your own image registry. Use the services that do it really well. Like I don’t want to run my own logging. I want to use somebody to ship my logs off to and or a solution that’s inhouse that I ship my own logs off to. I don’t want to do it myself. I have an admin and email server and I don’t know how long we want to use the experts.

Robert Blumen 00:21:57 Let’s switch topics now and move on to the next layer of your stack of the top security issues. That would be the network. What are the main issues in the network security area?

Chris Love 00:22:10 It’s access. So within a Kubernetes cluster, any pod can access any other pod unless you set up network restrictions. You either want to use network policies or a service mesh like Istio. This is probably one of the areas that I see the least maturity in within the folks that I work with. I would say recently I’ve seen far more maturity where folks are using service mesh that they want to use or they’re using network security policies where you are limited to call outside traffic. In other words, your pod cannot get to Google as well as your replica A cannot get to a different namespace with replica B. And that’s definitely something that you need to look at and control and it’s a lot of work to be honest. Within a namespace most pods will be able to talk to other pods as well as, so there’s two different ways that you can set up a pod in terms of which network it runs on.

Chris Love 00:23:08 Because we have our host network, right? And it’s how all the other hosts talk to each other. It’s the typical networking pattern that we look at. Then we have some sort of pod network, whether it’s virtual or it’s IPed a different way. They’ve done it about 15 different ways sideways. But if you run your pod on the host network, it can access everything in the host. Then depending on the permissions you give it. So definitely keep your pods on the pod network. There are some Damon sets and controllers that do host level operations. For instance, you need Nvidia drivers installed, right? There’s Damon sets that will run on your host. So when your host starts Nvidia drivers get installed. So you have to run them on the host network, but if you can, and majority of your workloads should be on your pod network as well as network traffic.

Chris Love 00:23:58 I would talk about, you mentioned operators. Operators need to talk to the control plane, the API server. And because of that you’re looking at RBAC security by default don’t mount the service account token, which allows for you to authenticate or at times if you misconfigure it allows you to authenticate with the API server. So it’s controlling access there. Also, you’re looking at DNS as well. That’s another consideration at a network layer. Everything basically within Kubernetes is all in DNS when you’re talking to the servers, that’s DNS, when you’re talking to another pod that’s DNS and that’s really important with in a Kubernetes cluster itself. But yeah, one of the key things you can look at with intrusions is you go from pod to either a secret you shouldn’t be able to get at or you go from pod to your API server. Secrets is another thing we should discuss, Robert. If you want to throw that in the list, we can talk about that after we talk about networks, but there have been some really good improvements in keeping Secrets secret as it would be.

Robert Blumen 00:25:01 I want to go back through, you made a number of points there and to delve into them a bit more detail. Network security policy controls, which namespace communicate with other namespace and service mesh is more granular. They are similar but different. Would you use one or both? And for what purpose each one?

Chris Love 00:25:23 Sure. Network service policies control both inbound and outbound traffic, right? Ingress and egress from a pod that the network security policy is bound to. So you can restrict talking to docker.io from a pod level, or you can restrict docker.io talking to the pod, or you can restrict pod A talking to replica B. You can do also all of that within Istio or whatever service mesh of the month you pick up. I’m not a service mess expert. There are very valid use cases for service mesh. And there’s one very good security use cases where you have all your traffic within your clusters encrypted. That’s definitely a use case when you’re looking at sensitive information that’s traveling in between pods. If you’re not using encrypted trans, like your application doesn’t support encrypted transmission across network layers, you’re looking at all your network traffic’s unencrypted. So you have a PHB web application that makes authentication to a job engine within your cluster and it’s password protected.

Chris Love 00:26:30 That password’s going over the wire. But if you have Istio installed right, that password’s now encrypted or another service mesh of course. So it’s using, that’s one of the things, this is drop-in, right? That’s one of the huge benefits you get. I’m not a service mesh expert, I have other good friends that are really good with that stuff. There’s definitely use cases, but once again, you’re maintaining or you end up paying for somebody to maintain a service mesh for you, which is yet another application, you’re looking at more load on your system. The CNI provider that is installed, provides your network security policies. So you’re really not looking as much overhead compared to running a service mesh. And there’s actually quite a bit of overhead nowadays. Running service meshes still is, it’s getting a lot better than it was initially and upgrade challenges as well with service mesh. And you’re running another application Robert, right? You get CVS in it. So try to limit the number of applications you’re running so you’re limiting your, the number of entry points you have from a security standard.

Robert Blumen 00:27:33 I’ll let our listeners know that we did Episode 600 on Service Mesh. Chris, now back to our conversation. You mentioned that the operator needs to be able to access CAPI and so it needs our back, whereas most applications you’re running do not need the API then would you default to the average pod should not have any access at all to the API and therefore should not have any credentials available? I think I’m just at this point restating what you said. Do you have anything to add to that?

Chris Love 00:28:06 Well, it’s on by default is what I have to add. And I’ve had conversations with security folks before about that, that have coded it and that look your dog gives you when it kind of turns your head and goes, huh? That’s the kind of look I often have where the service account token is automatically mounted so that that’s not something you want. Now that is most like also a pod by default. We’ll use the service account for the namespace. So again, create a service account like most Helm charts or you create your own deployments by hand will have service accounts. Like you’ll want to create your own service account. So there’s really two things, right? Does it have a service account token? What group and or permissions does that service account token have? And are you restricting, like even if you mount the service account token, for instance, if you don’t allow a pod to talk to the API server, you remove that path using network security policy.

Chris Love 00:29:05 Doesn’t matter if that security token, but yeah, don’t mount it. It’s a bad thing if you don’t need it. And let’s look back to operators to give a little bit more color to that. Typically operators create pods, maintain them, need to be able to access pods, do other cool things with pods. So they require access to the API server in order to do that. In essence, they’re making coup control calls if you want to think about it in that way. And that’s how they maintain, say you’re running cockroach database. Worked with them on their operator actually, funny enough.

Robert Blumen 00:29:35 We did another entire episode. So number 634 about Kyverno, which is a policy as code layer that enables you to create policies for things like every pod must have a non-default service account or many of these things that you are recommending could be turned into a policy. Do you all have a view about Kyverno or tools like that and where they have their place in your overall security profile?

Chris Love 00:30:04 Sure. And I assume that Kyverno is an admission controller?

Robert Blumen 00:30:08 So you have a policy as code, which is just a text file and it goes in a config map and integrates with the controllers to enforce the policies on, I believe on every single API call.

Chris Love 00:30:22 Sure. And that’s actually, so I mentioned PSPs, right? Pod Security Policies, they now have a pod security admission controller and that’s most likely the pattern that they’re using for that component as well. Yes, I do recommend that because for instance, you, I never recommend running a docker image out of Docker hub, right? You want to, if you’re a big enough company, you want to be able to screen the Bash image that you need to run within your system. So what admission controller can say is, you need your own unique service account for this deployment. You need to use the unique service account for this namespace. You cannot launch a pod that contains an image from docker.io without an admission controller. Like it’s kind of two ways, right? You should be screening your YAML before it’s installed in Kubernetes and then you should be screening your YAML once it’s running in Kubernetes. Because somebody can come in, bad actor can come in, get coup control, access, edit the YAML on a deployment and then it fires up their own image.

Chris Love 00:31:31 Well, if your network policy doesn’t allow you to download an image except for an image inside your system from ECR or Docker whatever docker registry of the month you’re using or container registry, then you prevent these things. But again, a mission controller is really good. It’s a lot of maturity though. I’ve worked with a lot of a larger companies that don’t have security and mission controllers. I would say that is probably one of the most mature, like if you’re looking at people that are crawling, walking or running, that’s definitely with the people that are running with security, they’re running their mission controllers. The pod security and mission controller now is part of Kubernetes. And that, it goes back to the statement you’ve made. What have they tried to make simpler? It gives you three stock out of the box profiles you could run with your workloads that enforces these types of things that you mentioned, Robert. So, and I’m sure the tool that you mentioned earlier as well provides some out of the box configurations and some best practices. Again, it goes back to leveraging folks that know what they’re doing. I’m not going to write my own container screener, I’m going to use somebody else’s, whether it’s open-source or closed source I’ve got enough going on in my life. Most engineers got enough going on their life. We don’t want to maintain some one-off awesome tool.

Robert Blumen 00:32:48 Do you have any stories come to mind of a network security issue that you either debugged or found doing an audit of somebody else’s system?

Chris Love 00:32:59 Oh no, this happened to one of our systems. One a client that I was working with, and this goes back to admission controllers, which is don’t allow external services to be launched if it doesn’t need to be launched. Right? Only route your traffic through ingress. An engineer and he was one of the DevOps engineers for some reason launched a Juniper pod, Juniper Networks, big data science stuff, right? Launched one of those, a deployment, had it on an external IP address. I think 30 minutes later a Bitcoin miner was installed on it because people are screening, are going through IP addresses on the Cloud provider that we’re on. They look for, okay, this is Juniper notebook, right? They hit the path where you get the login screen, then they say, okay, I’m going to try these 15 exploits on the 15 different versions. We had one of the versions that had an exploit in it.

Chris Love 00:33:54 30 minutes later we had Bitcoin miners installed. Yeah, our CPU usage went up just a little bit. So again, like how could that have been avoided? One, use a mission controller or network security policies where a pod cannot download software, right? If you don’t allow a pod to talk to external networks, why should a pod be able to talk to Docker hub? Why should a pod be able to talk to Google? Why should a pod be able to talk to XYZ FTP site? Shouldn’t. So if you don’t allow that traffic, if you don’t allow a pod to start up with an image that is a foreign image, pod won’t even start, right? So you won’t run into that problem if you don’t have a set of base images that you’re constantly screening for. In other words, you can only run company ACME’s Python image when you’re running a Python application. If you allow any Python image, you’re going to get CVEs in in-house. And it’s really tempting to do that. It’s really tempting to go to docker hub and download, use Python slim 312. Great product by the way. I’m not knocking them at all, but I know for instance it won’t pass CV screening.

Robert Blumen 00:35:06 Love that story, Chris. Let’s move on to the next layer in the stack, which we inserted in the middle of the show, secrets. What are your main points within the secrets security and what’s gotten better in that area?

Chris Love 00:35:20 So back in the day, secrets weren’t secrets and they’re still not, if you use default secrets, it’s base 64 encoded or base I forget exactly which encoding they use on it. But it’s plain text. But now most Cloud providers and some folks within on-prem allow you to mount a, so you have a Cloud provider secret. Kubernetes allows you to mount that secret as a volume. So you’re on the operating system, it’s actually a mount point on your container. And that’s how the secret is either injected or becomes a file store. It’s typically a file level secret that you can then access with your application. That’s probably the slickest and it’s one of, it’s the same type of spec where we have different, like CSI CNI, all the different file providers. Well now we have a container file provider reference. Like I said, Amazon provides it, Google provides it, I’m sure Azure provides it as well.

Chris Love 00:36:20 So all the big providers are using that pattern now. And still your secrets unclear text, you crack into the pod level, you’ll have access to it, right? But pod replica A cannot talk to secret C that it shouldn’t be talking to and access it, right? You’re not able to start a pod and mount any secret now. It’s this pod has to have this profile and be able to access this secret. And it goes back to pod identity within the Cloud. So for instance, say your pod is accessing the Cloud components, right? Often CI needs to download images, it needs to push images into registries. So again, it goes back to secrets management. You don’t want to put a Cloud level admin. I can push into ECR password and a regular password. You want to use pod identity management, which binds a Cloud role to a pod itself.

Chris Love 00:37:23 You then authenticate with that pod role. So it’s a couple of different things. You need to have your GitHub token or your token for XYZ SaaS that your application talks to. You put that into your Cloud provider secret, you can then easily roll it. There are sessions, it can be refreshed. Then you put it in through a file system mounted secret as well as if you have, you’re accessing your local Cloud provider. You want to use a rule on the pod and authenticate in that manner rather than accessing a secret out of a plain text secret. Everybody is using encryption at REST. So within etcd and within control plane, all the secrets are now encrypted. Especially if you’re using which what I recommend is the Cloud providers maintaining your control plane. You’re not running etcd yourself. That’s the big gotcha, I’m glad that improved. But, with the improvements it became a little bit trickier and getting pod level identities running correctly is it’s a little bit more fun. So you not only have to understand Kubernetes, it’s kind of like wiring, it’s Russian doll, right? You’ve got one egg that needs to fit inside another egg that needs to fit inside another egg. Then you have your secret mounted correctly.

Robert Blumen 00:38:40 Yeah, that rush into all things, that’s totally how I feel about Kubernetes. Now, this last point, I wanted to go through it because it’s a little bit complicated to make sure I got all the pieces. Say I’m trying to access a service through the Cloud provider, which might be for example ECR. There may be an option to have password authentication and you could put the password in a secret and mount it onto the pod. What you’re saying is don’t do that. Create an identity within the Cloud service provider and assign it a role that has a policy that granted access to ECR and then bind that identity to your pod. So it’s happening. Now there’s still some token, but it’s outside of the pod. So somebody gets into the pod, they can’t really easily get the credential. Did I get that right?

Chris Love 00:39:26 You got most of it right? Okay, so let’s go there. It’s actually not, it still is a token within the pod, but it’s a role that exists in your Cloud provider. You don’t have the username and password inside of a secret. It’s not clear text, right? There’s a token that is mounted again inside the pod and that’s where your credentials exist and that’s controlled by your Cloud provider. So you don’t have to worry about 15 different points accessing that same password. Pod A has service account A, that service account has an annotation on it that includes the role binding because that exists. And because there’s an operator or a controller that runs inside your Cloud instance of EKS say, it realizes okay, this annotation exists on this service account. This service account exists to this pod. I mount this token, this AWS authentication token within the pod when it starts, you can now use AWS CLI and you have the role.

Chris Love 00:40:29 So I create FU role in AIM, I give it this grant, which is list ECR images in this repo, in this account only. I then have fine green control over where it’s talking. It’s the same thing, right? I have to maintain those IM roles somewhere, but now it actually exists. The source of truth is the Cloud rather than the source of truth being it’s a Cloud. Oh yeah, I got to have a password too that I got to go change. For instance. You can automatically set it up that it has to roll, like it will reset itself. You don’t have to go through and reset stuff.

Robert Blumen 00:41:06 Sure, we would like credentials to be short-lived. Does this mean we all need to be coding our services or using libraries which either get some kind of a notification if a secret changes underneath and refresh or reread it from the file system every few minutes to ensure that we’re always using the latest?

Chris Love 00:41:29 And that’s the DNS controller that runs with, actually, it’s not the DNS controller. I forget which one it is. But it has a config map mounted to it and it deliberately refreshes that config map. So that’s the type of pattern that you’re talking about where, hey, did my meta to get data get updated? Yes. But the thing is with most Cloud libraries, at least they do that refresh automatically, right? If you’re using a web token in AWS with their API, they understand it refreshes. But yes, it is a short lived token. It is session based. There is a duration on it. So yes, you do have to refresh it, but again, I’ve even seen libraries that allow authentication from a database into a Cloud database that uniquely binds into the pod identity. So yes, you have to be aware of it as a Cloud native application engineer, this is the same type of thing that you have to think about as the fact that you’ve got to understand you’re going to restart. You’re going to leave your server within 60 seconds. That’s the type of application that you want to design. You want to be fairly stateless. And it goes back to that. If you’re using a, you’re running a long job for CI. You’ve got 60-minute job, you’re session token is for 30 minutes, guess what? Halfway through you better check to make sure your session token is active before you try again, right?

Robert Blumen 00:42:55 Absolutely. The last major area, we’re going to hit possibly two areas, container and pod security. Is that one or two distinct topics, Chris?

Chris Love 00:43:04 There really two different distinct topics. Pod security, and we’ve already been covering a bit of pod security because you brought up the service account token. We talked about RBAC. That’s directly integrated into whether you run a stateful set, a deployment, a job, a pod, whether you’re mounting that. And we talked about pod security when we’re talking about choosing which network the pod runs on top of. That’s another yet another configuration, whether it’s a host network or pod network, it’s in actually in a deployment or a pod YAML. So we’ve already been talking about a bunch of that. What we haven’t talked about are specialized pods. Some pods you want to be able to run. For instance, system level Linux commands. If you’re installing Nvidia drivers with XYZ pod, that pod is going to have to have run the host network.

Chris Love 00:43:53 It’s also going to have to have specific capabilities through your container engine that you’re running. Doc run C whatever you’re running to run your images. You actually define those Unix level permissions within the pod. Also you can look at stuff like SE Linux and you can look at App Armor. All of that can be derived through the newer pod security emission controller. So now that kind of stuff, you want to talk about again, people that are crawling, walking, running, that’s like sprinting, right? I know very few people, at least I’ve run into very few people that are running app armor profiles for instance. But if you know that your workloads are going to be attacked, your Nginx servers are going to be attacked on a fairly strong basis. Might not be a bad idea to look up how to run App Armor for Nginx. That way your workload is isolated better.

Robert Blumen 00:44:48 I some reading on this area, which discussed a technique where pods are granted temporarily the ability to perform operations at route level and even then only to a subset of system calls? Could you go into more what’s that about and how to use it?

Chris Love 00:45:05 So that’s actually just low-level Linux stuff. It’s been around forever. What you’re talking about now is we’re granting, so at a Linux kernel level, a process is you brought up route, right? You should never run your pod as route. So within, and that’s another configuration within Kubernetes is whether you run your process within your container as which user. You should always use a different user. If you run as route, you also want to see what permissions, like sometimes I’ve known controllers that restart nodes. Kind of crazy, but there’s use cases for it, right? Back in the day, I know folks over Comcast were mounting two different video cards to a node and it would require restart at times. Because this is actually a physical video card. When they’re streaming and encrypting movies, you’d be talking to an actual video card that would be streaming you that movie.

Chris Love 00:46:03 I’m sure it’s changed a bit over the years. But, restarting, there are use cases. That’s actually one of the Linux level permissions, whether you have permission to restart Linux. So again, you’re able to do that fine grain type specification within a pod on what type of permissions it has. And it goes back to don’t run as route within a container, but that’s both pod configurations as well as container configurations. So don’t and there’s a lot of people that are a lot better experts than I am when it comes to container level security because that’s such a broad topic now. But there’s just some basics as well, with container level security. But yeah, you’re able to go through, do App Armor Pro profiles, SE Linux profiles. There’s a lot of fine tuning you can do to isolate your workloads and specify the workloads that are given. Elevated permissions are controlled with only the elevated permissions they need.

Robert Blumen 00:47:02 In the short amount that we have left. I’d like to do a couple quick hits on container security. We did already discuss some issues around containers, such as whether they contain known vulnerabilities, where you get them from, who can pull them down. One other issue I had in my outline is the idea of container escape into the host. How big of a problem is that and what can you do about it?

Chris Love 00:47:27 Don’t run a host network. So if you’re not on the host network, the capability to escape to the host reduces tremendously because of how the networking and the virtualization almost quasi virtualization works. The routing isn’t there. So that’s it. Also, there’s some container systems provide the capability of isolating the workloads as well as if you have workloads. You can isolate workloads to run on specific nodes. So if you do get to the host, you don’t like, say you have workload A that cannot talk to workload B, right? It just can’t happen, right? So don’t put workload A on the same nodes that workload B is on. Pretty simple. I knew of a bank that every application had its own Kubernetes cluster. Really interesting pattern. Not a bad thing. If you need to isolate your workloads at such a low level, then possibly look at that.

Chris Love 00:48:21 There’s also Kubernetes virtualization within Kubernetes now, where you can run a Kubernetes cluster inside of Kubernetes, because that’s multi-homing. But I’ve digressed. Let’s go back to image level security and container security upgrade. Have the capability where you run CI and CD often, have a set of base images if your shop works in Python, works in Go, works in Java, have base images for your developers. Have your golden Java image. Have your golden Python image. Have those screened on a regular basis. Have those upgraded on a regular basis. It’s really important. People, I go into large companies and people are still not screening their images. They screen their operating systems. So goodness gracious, their nodes are being screened, but Bash inside of an image isn’t being screened. Also, don’t install Bash if you don’t need it. Don’t install, who is if you don’t need it, right?

Chris Love 00:49:17 For instance, Golang is one of the neat languages where you can compile and have it run on virtually nothing. The less binaries that are in an image use it. Use multilayer images for your builds, right? So you have a build image where you have your libraries that your shared build libraries that you need for Python dependencies. But in the next image that you’re actually deploying into production and using, don’t have those libraries in it. Don’t have Git installed, don’t have Curl installed, don’t have W Git installed. Just don’t, right? But sometimes you have images CI’s probably the worst, right? Because CI images need to, for instance, talk to GitHub. They need to do XYZ. They need to do all these different things. So be rigorous about maintaining your own images and upgrades. Don’t run images as route, as we mentioned.

Chris Love 00:50:09 Don’t allow your images to do a lot. It should be run application. That’s it. If you run Python 312 when you have time and make sure that you have a backlog to upgrade to Python 313. There’s a reason that you want to keep upgrading. Also have it simple as possible to redeploy your entire stack. It’s good for DR but it’s also good when there’s a huge bug that comes out that allows to do remote execution and you run that library everywhere. Log4J comes to mind. You’ve got to be able to push a button and as quickly as possible, redeploy your entire stack. Kubernetes is really good for allowing to you to do that, don’t get me wrong. But it’s still down to other principles, which is your CI and CD system. Make sure that that’s tuned in so you can do it. And the way you get to do that more often is, is upgrading often, right? So if you’re upgrading regularly you know you can redeploy your workloads regularly as well.

Robert Blumen 00:51:10 We’ve hit on some pretty big mountains in the landscape of Kubernetes security, Chris. In the time that we have left to wrap up, is there any parting words you’d like the listeners to have of top three things to think about in securing your Kubernetes cluster?

Chris Love 00:51:26 Upgrades, protect your control plane and step back and make sure like you do what you can when you set your cluster initially, those are probably the three things. Upgrade your nodes, keep up to date. I’ve walked into environments where they’re running a control plane that isn’t even supported anymore. It’s so old. It’s over a year old and gosh, that is not that old, right? But over a year-old Kubernetes is, might not be supported anymore depending on what version you rolled out at that time. Upgrade your Python, upgrade. If you got Bash in your image, upgrade it. And as well as and we really haven’t talked about this, but there’s other systems where you’ve got Apple level security, you got to be concerned about. You’ve got Kubernetes level security, you’ve got intrusion detection. All of that matters if you’re at the type, at a level of organization where you’ve got the time, staffing, and money to do that type of work.

Chris Love 00:52:21 It’s a pretty broad topic. And leverage experts, right? There’s a reason I run Kubernetes. It’s written by people that are a lot smarter than me and I realize that I’m not the one that’s going to go write the latest AI tool. Because that’s just not my expertise. I go back to, I’m a building kind of a construction engineer for computers. I’ll build a cool place that cool applications can run on top of. But in order to do that, I’ve got to have some groundwork. I’ve got to have some base layers. I’ve got to have some base systems that’ll support me. And it kind of goes back to a DevSecOps or DevOps principle where we needed to automate stuff. It keeps me out of trouble.

Robert Blumen 00:52:58 Where can listeners find your book, Core Kubernetes ?

Chris Love 00:53:01 It’s available on the Manning’s website. Just go to Manning.com and type in Core Kubernetes. Really want to thank Jay for dragging me in to write a book. It was quite an experience. It would take a lot for me to write another book, as well as, they can find me on Chris Love, CNM on all the different socials.

Robert Blumen 00:53:20 Any other place on the internet you’d like to point people to?

Chris Love 00:53:23 Sure. Wonderful company I work for, which is Modernize.io. Love working there. It’s been a real blessing. We’re a consulting company. Think of us as a boutique version of Deloitte, as you would say. And happy, really happy to join you here today, Robert. Appreciate the time that and the great questions you’ve asked me today.

Robert Blumen 00:53:44 Thank you, Chris. It’s been a pleasure. Thank you for speaking to Software Engineering Radio and for Software Engineering Radio, this has been Robert Blumen. Thank you for listening. [End of Audio]

Tagged as:
Join the discussion

More from this show

Menu

Recent posts