Changing the workflow at Zynx Health with Chef
Using Chef, AWS and other tools to deliver products quickly and reliably
Zynx was the first Hearst business unit Pauly worked with to evaluate its workflow and see how it could be improved. There were a few reasons why Zynx was a good first choice. First, the team members were experienced Agile practitioners who already practiced continuous integration (CI) and continuous delivery (CD).
Second, their environment was made up of approximately 50% Red Hat Enterprise Linux and 50% Windows machines. They had two existing platforms to build products on, a legacy platform and a greenfield CI/CD platform. Additionally, Zynx had recently acquired a company and integrated its product into the Zynx workflow. The team was eager to manage this third platform with Chef. With all of these possibilities, Pauly could learn how to apply what the team learned to other groups, no matter which systems they used.
Another reason was that the team had approximately half their applications in the cloud and half in their data centers. Again, this gave Pauly a chance to take what he’d learned from the Zynx team and apply it to other groups, no matter where their applications were located.
Finally, the team was already using Chef. They understood the benefits of automation and the concept of infrastructure as code.
Creating a plan
The first step was to find out what the team’s goals were. For Zynx, their priority was less about volume and more about velocity. They wanted to ship more quickly. Their product platforms were at various levels of maturity when it came to infrastructure as code. They wanted to streamline their processes, make them consistent and eliminate rote, mechanical tasks that were slowing them down.
The team accepted that, in the beginning, heavy lifting would be required. In order to address their technical debt, they had to set some time aside, perhaps in each quarter, each month, or each sprint and start chipping away at it. They would prioritize their problems not by technical value but by business value, so that management could easily see the benefits.
They began with a blue sky scenario, where they thought about what an ideal environment would be, without considering any barriers. Then, they built a roadmap that could get them as close to that environment as was realistic.
Pauly brought in people from AWS who reviewed what the team was doing and gave advice on how to proceed. He did the same with Chef.
Moving forward with Chef
The team’s use of Chef made a good jumping off point for changing some of the team’s processes. The Zynx team was using an outdated, open-source version of Chef. They compensated for the lack of features with various scripts. Upgrading to a newer release of Chef relieved some of their technical debt by giving them capabilities that they were, in effect, creating and tweaking themselves.
Updating the Chef server also gave the team a reason to reevaluate their cookbooks. Previously, they had acquired cookbooks from the community and modified them to meet their needs; however, the modifications made it difficult to use updates from the community. Another problem was that they were basing their workflow on a single repository for all cookbooks.
The team’s Linux system engineers had the most expertise with Chef, but through training and by working with the team’s Windows system engineers, everyone deepened their understanding of concepts such as resources, providers, roles, environments, policies, and encrypted data bags. With the whole team working together, they began to deploy the latest version of Chef onto both Windows and Linux machines in the Zynx infrastructure. Community cookbooks were wrapped more strictly to allow updates from the community, and separate source control repos began to be used for each cookbook.
Managing applications and infrastructure
Currently, the Zynx team uses Chef to deploy applications onto its Linux systems and PowerShell scripts combined with WebDeploy for its Windows machines.
For the cloud, they use a mix of Amazon machine images (AMIs), the Zynx AWS Automation Tool (ZAWS), Windows PowerShell Desired State Configuration (DSC), and Chef to manage the infrastructure. They find that AMIs are the most expedient way to create Windows servers because of all the software, patches and server packs that need to be installed.
For Linux machines, they use ZAWS to provision AWS resources. Then, they use Chef to set up applications and handle system configurations.
For their data center they use VMware images combined with PowerShell scripts.
Understanding the Zynx cookbook workflow
Zynx’s most mature workflows are automated from source control to production. Achieving that standard of maturity throughout an enterprise with multiple product platforms requires special attention and focus. For example, the Zynx workflow for cookbooks is customized for each product platform. The green field (CI/CD) platform has the most well defined workflow. It incorporates best practices such as using Foodcritic, lint, and syntax tests.
The Zynx team uses Jenkins to orchestrate the CD pipeline. The pipeline uses automated testing to verify the correctness of all application and infrastructure changes before releasing the code to production.
In terms of reviewing code, ideally, an operations person looks at changes from development and someone from development reviews changes from operations. Pauly points out that, “You get another set of eyes, but it’s another organizational set of eyes, from a different perspective. You get that DevOps approach.”
Although the operations team makes most of the cookbook changes, there are benefits to having the developers involved. Shared responsibility for cookbooks sparks discussions about how to configure and manage environments. Everyone has to look at how those environments are architected and think about how well they will support the applications. Going forward, new applications will be in AWS, so the development and operations teams can work together to decide how best to allocate cloud resources from the start.
Automating the deployment workflow with Jenkins demonstrates another aspect of DevOps. When knowledge is embedded within a tool, there’s no need to rely on just a few people who are specialists and on whom the entire process depends.
One of the benefits of using the newest Chef server was that the Zynx team could use Chef Analytics. Pauly found the feature useful because with it he could present metrics to his managers that showed how much faster deployments were happening as compared to what the team was doing before the changes.
For example, before they improved their process, the team manually deployed a security tool to over 90 servers. A deployment took, on average, 15 minutes per server and it took the team about 22 ½ man-hours, in total. To replace the manual procedure, they wrote a Chef cookbook. The cookbook took about 4 hours to write and it took approximately 30 minutes to deploy to all the servers. The cookbook is a reusable resource that they can use every time they want to install or update the tool.
Engaging the community
Other business units in Hearst are adopting Chef and a Hearst community is developing whose focus is on DevOps and automation. As the Zynx team became more adept and self-assured they wanted to share their knowledge with others, both within the Hearst community and with the open-source community in general.
For example, Aslan Brooke, the team’s Manager of Development Operations, is the author of the open source tool ZAWS and is active on the internal Hearst Chef chat channel. Aaron Blythe, a Senior Software and Systems Engineer, is a strong advocate for contributing back to the open-source community.
Pauly says, “We want to start contributing not just to open source projects that are already out there but to create our own projects internally and then make them open source. We want people to know that Hearst is interested in giving back.”
Looking to the future
The Zynx team has many plans for the future. Like the rest of Hearst, they’re going to expand their use of the AWS cloud. Another plan is to begin adopting the Chef compliance scanner during the first quarter of 2016. Summing up, Pauly says, “Our use of the cloud and open source marks a significant shift for the organization and also opens the doors to exciting new things that we couldn’t do before.”