The Unofficially Official Guide for Config Management at Scale

As organizations embark on the well-established patterns of shifting to DevOps, a key driver of this is automation. Many organizations have found that configuration management and the CI/CD pipeline-driven automation that inherently comes with it is the first step along that path. But, initiatives such as this are not something that can be implemented overnight and are certainly not something that can be done without putting thought into the needs of your organization upfront.

Implementing Chef as a stepping stone into a fuller DevOps mentality shift comes with a list of training, underlying infrastructure, automation, and longer-term needs that if invested in upfront can significantly increase the success of your implementation. Even more, getting this right will help drive forward the required organizational shifts necessary to embark on wider modernization and standardization efforts driving significant value for your business.

This talk is the Unofficially Official Guide to Config Management at Scale. Hopefully this is where you intended to be, and if not, I'm glad you're here anyway. So I'm super excited to be speaking at ChefConf again this year. This is I don't know my fourth or fifth time and I hope that you find some useful information coming out of the session.

So the first thing I want to start by saying is the servers are hard. We all know it, that's it, that's the end, right? That's the tweet, we can all pack up and leave. Luckily my door is to the left so we can exit stage left. But in all seriousness systems are incredibly complex. They always have been more recently though the facts can't be truer. There's an infinite number of areas that need configured, that need kept compliant and patched, infinite tools to learn, you get the picture.

So just focusing on one of those areas like keeping a system compliant for example it's a daunting task, it's going to take an incredible amount of time. Now if you compound that time by 100 or 1,000 or 3,000, this is of course why many of the organizations across the globe have made automation a focus along with other initiatives to modernize their stack.

So this is old news, right? We've been doing this for a long time. It's not really, this is like ongoing news I guess. It's always going to be around in some capacity. So config management as an extension is something that's necessary if you have VMs in your organization, full stop. You want your systems to always look the way they should, always be compliant, you want your images to always look the way they should, to always be compliant. And so there's a whole conversation that we could have here about the merits for leveraging config management and image management processes for example, ephemeral environments, but we're here today to talk about how you implement Chef in the first place.

So if you're at a place where you just started this journey the first thing I want to say to you is welcome. You found a community of people that is all working or has worked to achieve the exact same things that you're doing. The Chef Community is arguably one of the most welcoming communities and helpful communities that I've been a part of. We're all passionate about helping each other and the trust that's been forged in this community is something that I'm not actually seeing anywhere else. We take our losses hard, we celebrate each other's wins big. And I'm excited that you're a part of this, whether you're new here or whether you've been here for a while.

And the second and probably most important thing I want to say is this, and that's that this journey is not going to happen overnight. Implementing Chef isn't just about writing cookbooks and applying them to servers. If you want to do this right there's going to be some unwritten steps that you should take first.

In many ways I often think of config management and its organizational implications as a jumping off point for DevOps enablement. And I think I've talked about that before in the past. Your organization may already be following DevOps principles in some capacity, but the level of automation and organizational cohesion that you'll build while building out this platform is going to really throw that into overdrive. It's really been my long winded way of saying that with all strategic initiatives this one's going to take some time.

So really quickly, who am I to talk about this at all? Before I dig into all of the good stuff that you should consider while you're doing config management in your org I want to tell you who I am. So my name's Brittany Woods, I'm a manager for the server automation team at H&R Block. Before that I was an automation engineer that had a focus on infrastructure automation in both on-prem and cloud environments.

So I've implemented Chef starting from the beginning more than once and in more than one place. I've also been the primary Chef administrator working to build out and administer the platform in some capacity over the last seven years. So during that time, I've learned a lot of lessons on what works and what doesn't, and I want to share that information to maybe help alleviate some pain.

I'm also passionate about DevOps, passionate about helping engineers get out of the grind of doing manual tedious work, and really helping them unlock their time and level up. And then I'm also passionate about helping organizations do the right hard thing. So really that's all I want to say about me, that's enough about me. Let's go ahead and move into the good stuff.

So as I mentioned, you're going to be going on a journey as you implement this platform. So when you're implementing Chef, a thing that I've learned is that there's really no instruction manual on how to implement Chef in your organization. I'm pretty sure that's by design, each organization is incredibly different and every organization is going to have their own needs. But after doing this several times I've realized there's some important areas you should hit on early to ensure that you don't get to this point of regret after you've already scaled this thing out.

At the end of the day, implementing Chef is something that's going to impact nearly every technical team in your organization in some form or fashion. It could entirely change their workflow, it's possible that it might give them new ways of achieving old processes, regardless of what it does though it's going to impact them in some way. So the question becomes, where do you start.

If you want to implement config management specifically Chef in your org, the first thing you're going to want to do is ask yourself what your goals are. So do you want to standardize your infrastructure? That might be one of the first things that you think of whenever you think of your goals. Are you looking to reduce drift on your systems and increase compliance? Maybe compliance is your driving motivational factor here. Maybe you want to empower your developers to own their infrastructure and its config or get away from situations where you have a single engineer that knows what's up about how a system is configured and run.

These are all examples of what's important to me and they're all good goals to have. They're all attainable, you don't have to pick between them, which is also a win, but really though they're also

examples. So the real thing here is that as an organization you really need to sit down and do some thinking.

What are some key outcomes that you hope to achieve by implementing Chef as a platform in your org? A word of caution when you're thinking about your goals, don't make one of them, Chef all the things in three months. Nothing's impossible. Sure you could probably do something in three months, but in my experience there's tons of pre-work items that you're going to need to get sorted out first.

So my recommendation is to take a product approach to your Chef implementation instead of looking at it as just putting a tool out there. Take this product approach because at the end of the day, you want to scale this thing out. And for best results, you want it to be used on every single server in your organization not just some of them.

So in order to do that, you can't do it alone. And as much as you're scaling configuration management as a practice you're also scaling your user base as well. So what you want to do is make this something that teams want to use, that they take pleasure in using. And make sure to really look at what you can do to drive the outcomes for your organization. Look at how implementing Chef is going to enable other parts of the organization and maybe their goals.

And then the key area is to look at what's going to give your end users the best and most consistent experience. You'll also want to make sure that you have full management buy in. So implementation of this platform is going to impact the entire organization and how it does its work. So you need buy in from the most senior levels in the organization to ensure that all impacted teams are able to dedicate at least some mental overhead to embracing this new way of working, otherwise it's never going to happen.

The next step you want to focus on is team formation. So actually there's two facets to this. What team is going to own the platform and then who do you want to start working on building this out for the organization. And believe it or not, those are two separate things here.

So the first one is who's going to be responsible for the platform. This is really who the long term governing team is. So Chef at some level is going to require this governing team to own and maintain its underlying infrastructure, do platform governance, all of that underlying automation in the environment like pipelines and documentation. You'll really need this team that's also keeping a pulse on the changes in the environment, driving upgrade, standards, enablement, building this out for other groups.

And I know what you're thinking, you don't need to hire a brand new team for this at all, that's definitely not what I'm saying, but you do need to upscale your engineers. The most natural fit for platform ownership of Chefs are those that are already responsible for ensuring the servers in your org need a certain standard. System engineers and administrators are that natural fit for this important role. They already know configuration requirements at the system level, they commonly have scripting backgrounds using Bash or PowerShell that they've been using for years. And all of that translates really well into the chef environment.

So upskilling these members of the team that already have a deep understanding of critical config on your organization's systems and then turning them into subject matter experts in the platform is a prime example of deriving value. You're upskilling and reskilling existing engineers without having to hire new experts. So you've not only upskilled and reskilled your talent that you already have, but at the same time you've given those engineers ownership of a tool that's arguably going to help them reduce repeatable tasks in their work lives.

So as I mentioned before, there's two facets to the team formation when it comes to Chef. And Chef is not a tool that's meant to be used by just a single team. I talked earlier about Chef being a jumping off point for DevOps because even if you're already following some DevOps principles Chef is actually going to amplify that. It stands to reason then that you wouldn't want a single team to be the ones chunking out all of the stuff for Chef indefinitely.

Not only does restricting the use of Chef to a single team bottleneck your implementation, but it hinders your long term ability to scale. So what you want to do is make your Northstar adoption. So this is using Chef at scale across your organization, that's really why we're all here.

So the question becomes then, who is going to help you build out this platform. All the automation pieces, understanding the structure of the platform, you absolutely want to involve the team responsible for a long term platform ownership. That team on the whole should be the one driving the implementation which we'll talk about more just in a minute. But also you want to include other groups that are going to need to you Chef in the longer term.

Get representation from various groups across the organization that are in the long term going to see benefits from a platform like this. By doing this you're achieving something really big actually. You're combating resistance to adoption by building out these other groups and giving them a sense of ownership in the success of the platform even though they won't be responsible for that long term oversight.

Most fear is driven from lack of knowledge in my opinion. And the easiest way to combat that is to ensure that teams have all of the information necessary, but also by giving ownership to those teams. But that other powerful thing that you're achieving is you're creating subject matter experts that are representatives from these other teams. They can then take that knowledge that they've gained and share it with their team thus increasing adoption of the new platform and then reducing training complexities for the owning team. So it really reduces that long term burden whenever you go to scale.

So now you have a vision for the platform and you have teams to help you get there, the next thing you want to look at is standards. So by this point you're likely going to be super early in your implementation, and it's important to remember that you're going to be revisiting these standards as you go along. But there are some initial standards that by defining them early you're going to save a ton of time so you don't have to go back and rework things later.

The heavy hitters are going to be cookbook profile and policy naming standards. So I recommend always prefixing this with an org acronym or name. This is to allow you to easily distinguish from public cookbooks. So ultimately this is going to help with dependency resolution as well as identification in your version control system.

Define where you're storing these. Always where possible use the exact Same version control system that the rest of your organization is using. If your development teams are using GitHub, use that. If they're using Azure DevOps, use that one. It's going to help with organizational adoption if you keep the sameness to your processes that they're used to following.

And that said, regardless of the group that the cookbook belongs to, the maintaining group, always remain adamant that all cookbooks need to live in the same place. This is going to help with governance and long term manageability of the platform. Having these strewn about and different systems is going to lead to chaos.

And then also consider your testing requirements. So how will you test your cookbooks. Cookstyle is a given in kitchen as well, but think about how you can standardize this. Think of ways that make your users experience better while still allowing flexibility to meet the needs across different teams. Think about what kind of images you'll offer, Vagrant in VirtualBox or will you offer Docker options.

Do your development teams commonly use one or the other? Again you're trying to develop similar processes and patterns. That way Chef is approachable by these new users, it's something they're used to seeing.

What process will you follow for reviewing cookbooks for release? Will you have a peer review for pull requests? Do you intend to have release Gits before it goes out to production? So make sure to define your production promotion process early. How do you intend to safely get to production or at least put enough rails in place that the process is as safe as it can be? That's part of getting that buy in.

And then while you're identifying standards you also want to identify areas for standardization which is slightly different. At this stage pretend that you're managing a Chef platform and you have 200 users let's say internally. How are you going to make their life easier? How do you reduce areas of institutional knowledge?

Remember, in taking this product approach one of the key considerations that you should help inform your decisions by is what will my users experience be like. My recommendation is certainly to standardize how cookbooks, policies, and profiles are generated. Make sure to include all of the required testing in managed locations, pipeline files, anything else that's going to increase a burden on them or something they would have to memorize whenever they're trying to learn Chef. Standardize your pipelines as well. Every single cookbook you have should have a pipeline that does the same thing as every other cookbook.

On that topic, you want to centralize those pipelines, nobody wants to be on a team governing standards and have to add a step to this Chef pipeline only to have to change it across more than 100 cookbooks. And you shouldn't pass that burden on to your users either. If you wouldn't want to do it, you shouldn't make your users do it either. And you also can't rely on whether they'll be able to do it.

So plan your Chef environment as if you will one day have hundreds of internally developed cookbooks because you likely will. And as I mentioned, try and standardize your testing as much as possible. You want to provide images for your users that match what your production environment looks like and get as close to that production environment as you can get. What this will do is make your testing valuable and telling for what's actually going to occur on that production system.

And then on the topic of pipelines, there's a minimum amount of automation that you're going to need. In my opinion, up front you want to ensure things run smoothly and what you want to be released to your Chef and for server or your supermarket is what is actually going out.

So for cookbooks you should be running the tests that the users are running locally. So cookstyle, for example. Everything runs cookstyle. Test kitchen, that's another one that you should be running. Each cookbook should run in test kitchen in some form or fashion. ChefSpec, it's cast are there. All of these are ultimately going to help you enforce the standards in your environment from the start. So your pipeline should also be the thing that is sending your cookbook to the Chef and for server or the supermarket or both.

And then policy files, those are going to be slightly less complicated but more complicated in some ways. So as always, you're going to want to run cookstyle. This is a good practice for any development item in

Chef like I said. The lockfile generated by Chef is what governs cookbooks allowed to run at that level. So make sure that you're running Chef update to generate the lockfile, but you want to only do that in your lowest environment. That's going to become this artifact that you want to promote upward.

Then you'll run Chef, push to get the lockfile to the Chef server. You're also at this point going to want to make sure to decide if you want to keep your lockfiles in version control. So my recommendation is yes, just so you have some visibility into the contents of it for troubleshooting, but so you also have it available if things get weird so you can reupload it.

So then the least complicated pipeline is going to be your profile pipeline. As always run cookstyle and then use zip up the profile and upload it to Chef Automate. But as an aside, you're going to want to make sure that you always ship your compliance profiles to a service account. That way it's namespaced in Chef Automate under that account. It makes sure that you're not tied to an individual user, but it also ensures all profiles are in the same place for easier use with your audit cookbook or whatever audit policies you're running.

So pipelines are going to be in part where many governance items are enforced. So as a result, they're going to be iterative. You're going to find other things that you want to add to your pipeline to meet your use case, I can guarantee you. These are really just the minimum things your pipeline should be doing. It's expected to iterate on them and update them.

In terms of prioritization, you're going to need a cookbook pipeline first and foremost. You're going to want to follow that quickly by the policy for your pipeline and then your profile pipeline can come last. This is going to allow you to start developing core cookbooks and shipping them while you get other pieces in place. If you think about it, policy files aren't actually used until you connect systems to Chef, and you can't connect systems to Chefs until you have cookbooks that actually do things.

So there's also a ton of things to talk about related to pipelines and not enough time to talk about all of it. So again, if you think of these pipelines as a way to enhance your user experience and enforce your standards and governance for utilizing the platform, you're going to be in good shape for getting these pipelines where they should be in that first pass.

Profiles pipelines. There's a lot of peace. So another long term element in this is training. You've trained engineers working on the initial implementation. But as we talked about earlier, Chef is a dish best served organization wide. No single team or project group should be the ones using Chef. If you want a broad adoption, you should make sure that teams don't feel left behind, that's where that inclusivity plays in. So if it's to be on all servers, teams with a vested interest in those systems should be empowered to use this platform.

And a key to driving that is enabling material and developing and internal training program. So the goal here is to empower anyone who needs to use or interact with the platform. This training program should also be all of the information that you would need in order to successfully get started.

So what you're going to want to do is start with the components overview. Spell out super clearly what all the different parts of Chef are, how they interact, and how the user is going to be interacting with them, and then spell out the development process. What does it look like on your platform to get code from point A to point B? And then what does that mean for the current processes that developers or operations teams are following? How does this change their current workflow that they're using now?

And then Chef brings about continuous configuration. Teams that have automated things in their environment by a script or other means may not be thinking about what running this every 30 minutes or

every hour is going to look like. So ensure that it's clear that all of these changes have to go through Chef after implemented. Ensure it's clear that anything you put in code is going to happen every 30 minutes if necessary, that's crucial. And then make sure you're explicitly clear in that components overview.

As I'm sure you can imagine, this is going to end up being a ton of information to digest. So keep in mind that you're always free to split this into parts. However you want to share this info really just depends on how much time you want to take.

And then the next part is hands on learning. So the hands on learning portion is critical because your goal is to make the users as comfortable as possible. So what you want to do is illustrate that with the controls you put in place by your pipelines and standards, they shouldn't fear working with the platform, the user shouldn't ultimately fear breaking things. So give them the tools to have the confidence in the platform and in their ability to do things without relying on your Chef core team. And make sure you've built your processes so that's true and that it's not likely they'll break something as they're working in the platform.

And then part of giving them confidence is providing enablement materials. So make sure you're documenting everything. Make sure that you require users to include robust cookbook usage documentation in their cookbook. I've always been a firm believer that specific usage documentation should live with the code and not buried in a wiki where nobody will find it.

Also, make sure you're creating a comprehensive wiki where it's easy to find things. Take time to do a little bit of UX design to ensure that users can find what they're looking for when they're looking for it, that's process documentation. This is also about making things easy and straightforward like we've talked about this whole time for your Chef users. So don't create vague documents with no clear processes. Make sure you're outlining what steps the user should take step by step to help them achieve a certain goal.

Things that I've created in the past is like creating a brand new cookbook or updating an existing cookbook, uploading a compliance profile to Chef Automate, and then creating a compliant scan job. These are all examples that I've done before and you can provide a step by step like step 1 do this, step 2 do this to really build that user confidence in the platform.

And when you write that documentation write it to inform as well as guide. Assume that your users have sought out the document because they don't necessarily know a lot about the platform which is not a bad thing at all. And also keep in mind that not everyone is going to learn by documentation, sometimes users will still have questions and be uncomfortable with the process, and that's also totally OK. Everyone's going to learn differently, and we want to bring everyone with us on this journey.

So this is also where you're going to want to think of yourself and really your core team as internal consultants. So we've talked earlier about having representation from multiple teams to help accelerate that Chef adoption. So this is going to be where that is incredibly useful. Not only should you make your team available by reserving some of your capacity to assist users from other groups, but you should also be able to rely on some of those relationships and subject matter experts that you built earlier on in the implementation with that matrix group to help build the platform. Those newly minted subject matter experts are going to be able to contextualize unique challenges their teams may face so the information they're giving them could very well resonate more than your training or your document ever could. Again, the point is to reach all potential internal users so having these avenues to do it is key.

And then the final driver to accelerate Chef across your organization and in doing so amplified DevOps is to build an internal community. So building a community can mean a lot of things. What I found helpful to

do is facilitate conversations amongst all of your users. Not only that though, you want to give them a place to help drive the platform forward. As a governing team facilitating these open spaces where users can express things that would make the platform even better from their perspective will be critical.

And doing this on a bi-weekly or a monthly cadence is going to allow solutioning to happen in real time. It's going to allow those feature conversations to happen in a transparent way. And it's also going to allow the governing team to communicate with their user base on a regular cadence.

But always, always, always have a flexible agenda for this user group meeting. The open forum should not be just a free for all for the whole session, you want to keep it somewhat on track so it's beneficial for attendees. If you don't, more than likely you're going to find reduced attendance because things weren't covered or whatever. So give it just a little bit of structure and make sure that you're giving people space to be heard.

You also want to make sure that you have an area to communicate openly about shift problems or suggestions, don't bury it. The easiest way to do this is create like a Chef channel for Teams or Slack or whatever tool you're using internally, and make sure that it's open to anyone in the organization. Allow people to drop in and ask questions, propose solutions, or just chat about the platform in general. This is essentially your window into your user's needs, engage with them openly all the ways.

A key notion in Chef really is to foster transparency. And there's no better way to foster that transparency than to do so with every facet of your Chef implementation. Keep users outside the core team involved and informed on what's going on. Start a newsletter if you think that's going to be beneficial to keeping users in the loop. Share your new ideas, ask for feedback, invite Chef users to share their ideas more broadly.

And earlier, when we talked about training we talked about the importance of introducing users in your org to components of Chefs as well as providing that hands on training to increase their confidence. So there's actually a longer term opportunity for more training and that's to cover some advanced topics. So keeping users engaged and excited about the platform means driving that engagement.

Offer advanced topics training as well. Maybe you want to lead a session on compliance specifically or maybe you want to offer a session catered to writing a plug-in or a custom resource or whatever other advanced thing you can think about. Use these to get the community excited, your internal community. Doing this is not only going to get them excited, but it's going to aid and driving improvements for your platform as well because you're crowdsourcing basically the future of that platform.

And then finally, I'm always a fan of fun awards. So if you think of quirky fun awards to give out yearly to your community of users and teams, it's going to keep this environment fun and exciting. The awards don't have to be the same each year, really it's more fun if they're not because then somebody is not driving to get that award.

So some examples of awards I've given in the past include the Babysitters Club award for knowing when Chef issues happened before anyone else in the organization. The first pull request of the year, which is a given. The most persistent data bag item uploader award for managing a ton of data bag items secrets and rotating them often which triggered the notification so everybody saw all these things changing. And then the fastest trigger finger for being the fastest to promote more requests from environment to environment which is not always a good thing, but it's nice to bring fun to things that are good and bad sometimes.

And then all awards they of course came with donuts in the pre-COVID days, things are a little different now. And then custom medals to commemorate the occasion, just to give them something tangible, because again it's really about just making things fun and engaging.

And then really with all we've talked about today it kind of just goes to show you that Chef is more than just writing a cookbook. It's all totally doable stuff but there's a lot here. And so if you make sure that you have these key items in place or that you're at least considering them early, you can be sure to kind of reduce that rework and decrease the early confusion. This is also going to aid in the ability to scale faster. The more people you can bring in and the easier you can make onboarding as well, the more you can make the environment standardized and allow for flexibility and then scale that out across your org.

This is not an exhaustive list, there's even more you could and should consider which could probably cover another hour I'm sure. But really it can partially be summed up in saying to approach this platform like a product. You want to make it smooth in terms of onboarding and usability, keep an eye on organizational needs, and then sell it to your teams. And then if you keep all of that in mind you should be on track to having a successful implementation.

And then just as an aside, I want to thank you all for joining in and call out. But if you have any questions at all after the session, please feel free to reach out to me on Twitter, you can reach out on my website. I'm happy to answer any questions or help where I can. And then if you're curious for more in-depth lessons learned and suggestions and guidance on implementing config management in your org, keep your eyes peeled, I'm working on creating this and kind of like a published guide. So implementing config management in your org will be the topic. And then I hope to release that later this year, fingers crossed and then hopefully I get to see you all in person next year. But for now stay safe. And are there any questions I can answer for the members of the live audience?

That's great Brittany. Actually, I'm sure the audience will come up with a couple. But I have some questions.

Yeah. Ask your welcome.

So one of the things you mentioned is that three months is not a realistic timeline to Chef everything, right?

Right.

What are some of the things that end up being gotchas as you get into the process. So you're like, I'm getting started, I know that three months is too long-- is not long enough. But besides human approvals and red tape and that kind of shit, what are the things that actually hold you up?

Yeah. So first and foremost is having organizationally specific core cookbooks. So all of those things that you need, those bottom building blocks, you're going to have to write those. Sure there's public cookbooks you can leverage but they're not going to always meet your use case, you don't want to leverage them directly, you want to wrap those. So there's that facet.

You also have to have some kind of automation to get that from point A to point B. Even if you're not writing like robust kitchen tests in that pipeline, you have to have some form of a pipeline to get your cookbooks to your Chef server supermarket so you can then use them. So having to build all of that and assuming you don't have a team of like 40 engineers to make that happen, having to build all of that while also determining what you want this platform to look like is a really big undertaking.

And so if you kind of rush that out you're going to find that-- I stressed a lot about user experience. So if you rush that you're going to find that your user experience turns terrible because you're going to get

three months and you're going to have stuff half way out and then your going to be like, all these processes you just learned and gotten used to, I'm going to turn them on their head because now we've changed everything. And so then they become frustrated with the platform and they seek other ways to do the thing because they don't want to jump through the hoop and then it becomes a problem.

Yeah, that makes perfect sense. That makes perfect sense.

Yeah.

So in your answer you mentioned have your cookbooks and community cookbooks and that kind of stuff. How do you decide, where is the tipping point between building your own cookbook and using one that you find on the internet.

Yeah. So there's organizational considerations to be made, like are you able to reach to wherever they're pulling a package from in the public cookbook for example without punching a hole in your firewall. But outside of that, if the public cookbook does what you need it to do, leverage it, use it. There's no sense in rewriting that and recreating it because you're just going to basically be copy pasting. But if it is missing an element of what you do, even if you were to fork that cookbook and bring it internal, that's now your cookbook, that's an internal cookbook.

So the line that I've always drawn is, as long as there's not any kind of organizational factors keeping you from leveraging it, and as long as that cookbook is properly maintained at least recently, use it. Just make sure that you wrap it because at the end of the day it is a cookbook that you're not in control of the update scheduled for, so you're not in control of any potential breaking releases. So make sure you wrap it, make sure you pin that version and make sure you bring in any updates with intent.

Awesome. Awesome. And my last question I think, I got some a bit more but. You mentioned a training program and building a training program from the ground up can be daunting to say the least. Is there any place you go to start that kind of thing? Like do you send your engineers anywhere to get them onboarded?

Yeah. So the easiest place to start is obviously to leverage learn Chef, that's always been my go to do first step. Just to get them comfortable with the terminology and understanding some of the pieces. But so my view has always been leverage the tools Chef give you, but you also have to understand that Chef being what it is and that you can use it anywhere you want basically for any purpose that you want which is great, it's a blessing and a curse. You're going to have a lot of things in your environment that Chef can't speak to because you're building this platform for your organization which is different than another one.

So that's why I mentioned starting with that components overview. And so after your internal users have done learned Chef and they've gone through and they understand Chef contextually, then give them that fill in the blank where you go through and you say, OK, in our environment this is exactly what our environment looks like and how all of these pieces interact together and this is how you get your stuff from A to B.

And then the second facet because you don't want it to be too much to take is to let them break things. Let them actually type out the code, let them deliver it to your actual Chef server, let them create things on their own as you are on the call and walking them through what to do because once you've delivered something to a server once and it didn't break then you're not afraid of doing it again because you're like, I didn't know what I was doing then, I've had time to do a Google now, so I can do this. So that's where I suggest starting. That is essentially the initial training program, is just walk them through your environment after they're comfortable with the words and then let them get their hands dirty.

I love it. I love it.

Yeah.

My last question is if you are very early in the journey and you recognize that it's a thing that you need to bring in but you're worried about getting a manager buy in, what are the key things that you think given your experience that managers will look for when deciding whether or not it's worth implementing.

So the easiest sell of Chef is I would say the built-in compliance and that drift detection and correction. Because even if an organization is using like an ephemeral environment for example, you still have images. Those images still have compliance needs, those compliance needs still need to be met and you're not really sure if they're being met or how can you be until your vulnerability scanner says Oh, no, we have STIG requirements that aren't being met. So that's the easiest to sell.

The other part is we're all kind of moving-- as we charge down DevOps we're all kind of moving to this place where engineers sure they have a specialty, they have something that they specialize in, but engineers are expected in some ways to know broadly all of these things. And so in doing that and removing all these silos and making teams like multipurpose teams for lack of better phrasing, you have to give them an avenue to not just have that single engineer that was responsible for this config file for x amount of years that understands how that application is functioning in production.

And by putting it in code and that code runs all the time, you're now giving a common language for every engineer in your org to speak. So even say a network engineer could look at an application configuration cookbook and say Oh, the Tomcat setting looks like this but in this other area it looks like this, so maybe there's something wrong there. So you're essentially facilitating that breakdown of silos by sharing that information in a common form in auditable code.