Assess the Security of Kubernetes with InSpec

In this session we explore how InSpec can be used to validate security and compliance settings on Kubernetes clusters. To scan all relevant security settings on a cluster that is managed in the Cloud this typically involves running checks against the Cloud Provider's API, the Kubernetes API as well as each node's local filesystem. We are going see a demonstration of how to run InSpec for security and compliance validation of a Kubernetes cluster with the example of the CIS benchmark for Google Kubernetes Engine (GKE).

The agenda for today is starting off with how to assess the security of Kubernetes. So we're going to see the challenges, and what layers we actually have to scan using any kind of tooling such as InSpec. Then we're going to see how we actually do it with InSpec and what the resource packs are to scan Kubernetes holistically.

Then we're going to see a specific implementation example that is the CIS benchmark for Google Kubernetes Engine, or GKE. That followed with a live demo of the CIS benchmark for GKE. And finally, we're taking the output of that demo, seeing what kind of reports are being generated by InSpec, and we're going to see Heimdall by the MITRE Corporation for visualization of the scan results.

So let's start with the first item on how to assess the security of Kubernetes. So Kubernetes allows us to provide a platform for application hosting that is very resilient and very robust. However, there is also an added layer of complexity due to its architecture. We need to look at different layers if we want to test the security of Kubernetes holistically on the platform.

Those three layers are shown here and visualized in this graphic. The first one is the Cloud or Virtualization Infrastructure. So for instance, your Cloud platform or the platform on which you're running your workloads on on-prem. Typical attributes that you're scanning here is access control to the platform, encryption settings, as well as metadata of resources, of course, not limited to those.

The second layer beyond the managed services that you often get on a Cloud platform is the Kubernetes API that you're interacting with to deploy your workloads. In the Kubernetes API you can define objects in Kubernetes areas such as port security policies, network policies in Kubernetes, as well as also metadata items that you might want to set.

And finally there are node configuration files on the Kubernetes nodes that you can access via SSH. Two example configuration files that have an impact on the security posture of your Kubernetes cluster are kube-proxy.yaml or kubelet.yaml. And for many enterprises, there is an additional complexity because they are running their Kubernetes workloads in a multi-Cloud or hybrid Cloud setup.

So that means that we need a unified testing framework to test the compliance of Kubernetes clusters across the enterprise and across all platforms for a unified declaration of secure deployment pipelines and vulnerability scans.

And how to assess each one of those layers we're going to look at next. So let's look at the InSpec resource packs for Kubernetes validation. The first layer is the Cloud Infrastructure or Virtualization Infrastructure. And Cloud vendors have open source resource packs for InSpec to validate infrastructure resources.

Three examples of those InSpec resource packs can be seen here for Cloud platforms but not limited to-- that is InSpec GCP, and their link to GitHub InSpec AWS or InSpec Azure. And using those resource packs you are free to create any kinds of tests that you find applicable to the security of your platform.

However, there are also benchmark profiles, agreed upon security postures, for instance by the Center for Internet Security, CIS, that have been published to test the Cloud platform against its API. And an example can be seen here and its link to GitHub. That is the inspec-gcp CIS benchmark that can be cloned and you can use it immediately to test the security of your platform.

An example of a very simple implementation can be seen at the bottom. That is testing any kind of attributes of the Google Kubernetes Engine cluster object. So if you describe the object Google container cluster with specific parameters to your environment, such as a project and your geographic location, you can test the value using InSpec of any kinds of attributes that are relevant for the security of Kubernetes.

Going next to the second layer, and that is the Kubernetes API. And the way we can approach to test the security of the Kubernetes API is we can use the InSpec Kubernetes resource pack that has been developed and published by Brad Geesaman-- and the GitHub link can be shown here-- that provides us a method to run checks against the Kubernetes API of any cluster, as long as we have access to the API.

The beautiful thing about this resource pack is that it does not define any kinds of objects in the resource pack that are statically defined, but rather we can test any kind of object that the API today, or tomorrow, or a month from now has. It means we can still use the same resource pack. But at a later point, if Kubernetes changes objects internally, we can just update our tests because now we're testing a new object. But the resource pack will still work.

And an example can be seen here. Very simple example, checking if a part exists with a specific name where you would do that you would check the Kubernetes object with parameters such as the API version, the type, in this case pod, the name space, and the name of the pod. And check any kinds of related attributes to this pod. This case, it should exist.

And the second example shows you that you are pretty much free to test any kinds of objects, such as pod security policies, fetch all the items that you get from the API and store it into a variable for post-processing and testing of its values.

A more complex example can be seen here for the test of pod security policies. Just as an example that shows you the power of this resource pack. And we have selected here an example from the CIS benchmark of a specific test that says that there should be at least one pod security policy that has a non-privileged policy. That means that using this policy containers cannot run in privileged mode, and do not have access to certain kernel features on the host.

To go through the code, what you would do first is you would fetch the list of all pod security policy names. There might not be any pod security policies, so this test might not be applicable. So you'd skip it in that case. Let's assume we have any, so we go to step number two.

You would iterate over all pod security policies, and fetch each object for each specific pod security policy that you find. You might have one, you might have several. And third, if you find a privilege policy, set a Boolean variable that you would later check, in this case has non-privilege policy, to true.

And fourth, once you have done that, you would test the value of the Boolean variable that you have just used in your algorithm to test whether there is at least one non-privileged policy. That shows you in total the power of the InSpec Kubernetes research pack with a specific example of testing attributes of pod security policies.

Moving on to the third layer. And that is scanning the configuration files on nodes using SSH. So for that you actually don't even need a specific resource pack because testing files via SSH is a native functionality of InSpec. But it is worth noting that there are very interesting challenges in the Cloud, and that those are some that we ran into in the past.

And that is if you want to run InSpec from Cloud shell for instance, or a CICD pipeline that is a managed servers in the Cloud. And the specific challenge is that the host from which you're running InSpec, such as Cloud shell, is not necessarily in the same private network as your Kubernetes cluster nodes. So you need to find a solution on how you access privately to your Kubernetes nodes.

And one way to do this is shown here in the example one in the case of Google Cloud, there is a technology called the Identity-aware Proxy that allows you to connect from your desktop or from Cloud Shell without the need of a VPN connection to instances without the need of a public IP. You create a SSH tunnel using the identity-aware proxy that grants you access based on your user context.

So whether that is a service account, or your personal user account and you're given access via Cloud IAM, and that allows you to access privately without any public IP exposure to private nodes. Once you have solved that challenge, testing files, its contents, and their permissions is actually fairly straightforward within InSpec. So you can see that in example two, testing local file permissions with InSpec, if you're testing a specific file using the file resource, you can test its owner, or its group, and in this case, you compare the values to route.

So now having looked at all the layers, let's look at how we can put all these layers together to one unified InSpec. Profile to be scanned. The center for Internet Security, or CIS, has published a benchmark for Kubernetes security recommendations. Those are agreed upon standards that are used by governments, businesses, in industry, as well as in academia.

And the example implementation here that we want to look at is the InSpec GKE CIS benchmark. Its link to GitHub is shown here. And it is worth noting that because of the three resource packs that we just talked about, this benchmark is separated into three sub profiles.

So the way you would run this benchmark is not using a single InSpec exec statement, but we have provided a wrapper script called Run Profile Status H, and that will create and store InSpec reports locally that you can use then for post-processing in CI pipelines, or in static analysis, such as the tool that we are going to look at in a moment that is Heimdall by the MITRE Corp.

So let's look at the demo for the InSpec GKE CIS benchmark. I have opened my Cloud Shell, and I have a Cluster available in this environment called InSpec Demo. It's a fully functional GKE cluster that I have deployed previously. One thing that I'm going to do to start with is I'm going to clone the repository of the GKE CIS benchmark. As you can see here it is stored in this folder.

Second, I'm going to copy an input file as it is commonly used for InSpec runs that lists a couple of very specific environment variables for my project. That is the project ID, the locations of my GKE cluster, in this case, it's a single zone, and also other parameters that are tested in the benchmark. I'm going to copy this input file into the repository folder that we just downloaded. it's going to have the same inputs.yaml.

I'm going to navigate into that folder. And then I will run the wrapper script called runprofiles.sh with the parameters that are required. In this case, the parameters that are required are the cluster name, my username that I'm using to connect to my Google Cloud environment, a key file that is required for SSH access, the zone or the region-- it would be the dash-r parameter of my cluster-- and the inputs file.

I'm going to run that. And there are some warnings on the Shell, but let's disregard them for now. We are seeing the output of the InSpec run. And we can already see that for the first profiles that are being run we get some successful controls and some failed controls. Now next, we would like to analyze the output of the entire run. So let's give this a few more seconds until the wrapper script has run InSpec for all the three layers that we just discussed previously.

I think there's one more node for SSH access. And we are done. There we go. So our reference script has successfully concluded. And the reports that are being generated by that script and by InSpec itself are stored in the folder reports. And you can see here that for each one of the subprofiles for each layer-- this one is the infrastructure layer, one the Kubernetes API layer, as well as the node configuration file layer, they are stored in that folder.

As a next step, we're going to look at how we can visualize those reports in Heimdall, and to get there, let's pause here for a moment and I will hand it over to Aaron who is going to talk in more depth about Heimdall and the MITRE security automation framework.

Wonderful. So you'll just go to the next slide. I'll let you drive, I'll talk. So thank you so much for having me. This has been a great opportunity to work with you. I first want to say that Google has been a great partner for us, and was one of our best strategic partners. And we've been able to accomplish a lot.

So the MITRE SAF, or Security Automation Framework, is an open source project by the MITRE Corporation and the FFRDC community, federally funded research and development centers, to provide a mechanism and a sharing in the open source community and the security community to bring together our own expertise and help grow the capabilities as a whole.

We were also one of the primary developers, or I was one of the primary contributors to the InSpec language itself, even before it came into the CHEF family when it was originally over at Deutsche Telekom, so this has been a great journey, helping grow the InSpec community, the security automation community. And if you want to know more about SAF or the tool that you're going to see here today, you can go to saf.mitre.org or heimdall-lite.saf.mitre.org.

So the next slide, please. So in general, the conversation that we've been having today follows the general SAF model for the workflow for security automation where we took the standard body documentation such as a CIS benchmark, or a STIG, or any other benchmark that you might have, converted it into InSpec content as its general compliance framework, created a validation profile as we call it, or a security profile, and then we're able to run that against the target and produce an output report, a security artifact.

And that security artifact is in a standardized JSON format, which we call each HDF, Heimdall data format, which was based off our original JSON schemas from the InSpec community and then enhanced a bit. And then we created a body of tools that can either be used in your CICD pipeline, and that's the Heimdall tools, or the InSpec tools, or the actual Heimdall gear itself, which is free to use for all the open community, to convert all types of security data to include InSpec reports, SonarCube, Burp Suite, OWASP ZAP, all kinds of different tools can all be converted into the standardized data format and visualized in the way you'll see here.

So that's basically the overview of what the staff tries to bring together, and Google has been a great participant in working with us and helping make that a reality for Kubernetes and the Kubernetes platforms. All right, next slide. So this is a preview of what you'll see. So this is what Heimdall looks like. Next slide. And let's take a look at them. So I'll pass it back over to you, Konrad. Thank you.

Thanks, Aaron. So let's have a look at how Heimdall is going to show the report. Let me just share the different-- share Heimdall. So I am going to drag and drop the five files in JSON format that were generated by the wrapper script that I executed before into Heimdall. So let's do that. Now it is loading. And from the previous run we can now see that in the GKE CIS benchmark, 53 controls actually passed, and 26 failed.

So our cluster is not fully compliant against the CIS benchmark that is open sourced, and we can drill down what specific checks have failed. If we look at this, we can then see in more detail every single control that has failed, and we can validate the exact value for each one of those checks and why they were failed.

So this is a great way for us to validate the security posture of our cluster by checking why each check has failed. And if we want to drill down for each one of the failed checks, we will see what test has been run, and what the expected results is compared to what was actually found.

So that concludes our demo. Finally, I would like to re-emphasize the most important statements from this session. So keep in mind that when assessing the security posture of Kubernetes, it requires you to run tests against the three layers, the Cloud or Virtualization Infrastructure, the Kubernetes API, and the node file system.

We have published research packs that help you implement those tests for each layer for Kubernetes. Now we would like you to try that out yourself in your own environment. So run the InSpec CS GKE benchmark in your own environment, clone the repository, and test it, and provide feedback on GitHub.

You can provide feedback by creating issues, or contribute to the repository by creating Pull Requests. Also get involved in the InSpec community and join our Slack Channel so we can discuss any kinds of topic around Kubernetes, InSpec, and GKE. Then see you at ChefConf 22 and on the InSpec Community Slack.

Well, thank you so much, Konrad.

No, go ahead. My same invitation is going out to the community as well for any of you that utilize Heimdall, or any of the SAF tools, or want to look at the library validation profiles to include the Google profiles. They're all listed on saf.mitre.org under Validation. So if need a quick link to get there fast to get to the GitHubs, then interact and use the power of open source to help us improve. Looking forward to it, and thanks again for listening.

Yeah great point guys. I do have a question though. With all of the open source options out there in terms of testing and validation, and as you were evaluating or starting this project early on, what are some of the choices that you had out there, and why did you choose to use InSpec for this project?

I can maybe start first. I was going to let you talk about it, Aaron, in just one moment. But there are other choices of how you can scan the security posture of your environment. One, for instance, is the Cloud Security Command center that has a slightly different approach to scanning your security posture of your environment.

However, InSpec is a perfect tool for defining all of your tests in a language that allows you to scan multi-Cloud, so no matter where you're running your workloads, you can use a single framework to define all of your CICD pipelines to deploy, and test your application deployments across GCP, AWS, AZURE, and so on. So that is one big thing.

The second one that I would like to mention is the flexibility of testing any kinds of platforms. So we have seen that we need something to test any kind of API, and any Cloud platform, as well as a way to interact with a REST API and the Kubernetes API in a very flexible and dynamic form. And also the native functionality of InSpec was also just very convenient for us to test the file configurations on the nodes.

Yeah, I totally agree with that. Some of the reasons that we started working with the InSpec community also included the ability to increase the metadata of our tests and give it better situational awareness about how those tests align to things like the CIS control that you were meeting, or the A53 missed control that you were meeting as we built out our benchmark so we could enrich that testing data so that it could be part of a business process.

Some of the other things that the InSpec community had talked about, because it is really a second generation compliance language that takes all the lessons learned from things like SCAP, and XCDDF, and Nesses Audit and all that, and really tried to grow those capabilities to say how could this really be a real-time tool that meets those requirements?

So the resource extraction layer and the ability to connect on multiple platforms as Konrad mentioned, but also the ability to grow our resources as we need to. So when we need to inspect something new, we can easily do that. We can tag and provide that metadata to say how does this align to our business process with the benchmark that we're doing, or the RMF process. All those types of things.

InSpec gave us the ability to do that because we literally built on it. We were solving these specific problems. The right ones run anywhere? Well, maybe we got that right in the compliance world, and now we can really do that with InSpec. Maybe not programming language, but we did it here.

So those are some of the things that we've seen. And even people being able to build out new resource packs, I find new train transports floating around on GitHub. I never would have thought about doing it over a serial port, but really cool. I can do it over Bluetooth? That kind of evolution from the open community is really helping us solve what used to be impossible problems, are now within our grasp. So it's been a wonderful community. And those are some of the--

Yeah, that's what I was going to say. It's the power of community and working together and really having that inside or that foresight that some of the community members had. Wait, I was tossing and turning because I had this problem and I found a way to solve it, and here, hear it is, enjoy. Make it better, please, kind of thing.

And one thing that I think is very powerful too is that, like you mentioned, the CIS, and [INAUDIBLE] for that matter, they are recognizing these profiles and reviewing, and certifying, and making sure-- level one, level two certifications, that are allowing for organizations to feel comfortable in a way.

To adopt this in-- it's not just the open source of old, just some guy in the back the back room just scribbling some code, and really having that recognition by governing bodies. I mean, that's a powerful thing. What are your thoughts on that?

The evolution of the community at large, and the community of interest, stakeholders, we've seen such a force multiply in the smaller investment from not only governments like DISA, like CDC, like FISMA, FEMA, the intelligence community, and even industry.

So some of our other primary partners in this community are VMware, and Refactor, and Tenable themselves. Even ones that we would normally consider a competitor from the viewpoint of the security automation community, they're coming together in a unified way to say let's solve a hard central core problem and work together.

And we're doing that across the industry, academia, and the government to say how can we unify, and then we each have our own value add, but how do we unify a solution such that we can operate on normalized standardized data? And agreed upon workflow process? And agreed upon content? Things like that.

And then each solve a very cool part of the problem but have a standardization that can help us move forward these capabilities in a way that really helps solve real world problems. If you look at all the breaches and everything, the easier we can make adoption of these types of capabilities, the safer we make the world.

And I'm sure that's part of Google's methodology and thought behind wanting to open source all of this, and that's why we the MITRE corporation open sourced all this as well. We all shared in the burden, we all share in the success.

Yeah, we talk also a lot about the codification and the ability to use, why code? Because as you saw in the examples, InSpec is a very human readable language which can essentially be used by more than one team. You don't have to have the code experience to read what's there and understand, and like you mentioned, Aaron, the tagging is impressive because the compliance officer can see that what you're validating against and tag it to the CIS benchmark that you're referring to the, the NIST 800 that you're referring to, HIPAA or whatever tag you have it that you're following, so you use one tool to cover your entire need for that particular aspect or control that you're trying to use to validate.

And surprisingly enough, it's the NSA that's actually building a lot of those HIPAA profiles.

Right. Yeah.

And those types of things. So it's from parts of the community that you'd never expect, but all of those things are starting to come together and that was our intention with this. And I think it's great. The last part that I've really seen a lot of growth in adoption in is, as we start to unify on some of these standard normalized data formats, I've seen an evolution in how this information is being used to further other processes that we would have never expected.

Concepts like adaptive capabilities testing, or streamlining of accreditation events, or documentation generation like NIST is doing with OSCAL, and automated documentation generation to do FedRAMP, all these types of things are being brought together.

And as we look at a multi-Cloud type of solution as Konrad alluded to, when you want to use Google Cloud platform and AWS and AZURE all together and you need a unified way to say, I built all those correctly, I've hardened them the right way, they're all validated and I can look at it, I need to be able to communicate that across all the teams, and production, and dev, and test, and they all need to be using the same answer sheet.

And that's what InSpec gives, and that's what the resource packs provide. And it gives you the ability to as almost a side effects do continuous and ongoing operations in a way that we thought was almost impossible 10 years ago.

Yeah, [INAUDIBLE] are very, very powerful. And on that note, if you're on the community, if you're interested in the community, there's tons of resources out there, we have LearnChef as part of our kind of beginners path into how to engage, and there's some content there for InSpec. There's tons of content here at ChefConf that we'll be covering that you can learn.

Join the various Slacks communities, get in GitHub, find the resources interact with the people, as you see two here very active community members that are engaging and sharing their stories. So please make take the time to research, to contact, to reach out. I've never seen a more engaged community. So please start the conversation if you can.

Any closing thoughts, Konrad, that you have on the topic, on community? What are your closing thoughts before we close out this session?

Yeah, in my closing thoughts just like to reiterate some of the things that you just said. Really, we strongly encourage everyone who is curious about testing the security of their environment, whether they run Kubernetes or they run different workloads, InSpec provides a very powerful way for us to scan our GCP environment, our AnyCloud environment, and we're more than happy to connect with you and talk about how we can use InSpec to test the security posture of your environment.