Transformation at Gannett
Gannett’s DevOps evolution – changing the culture, processes, and tools of a media juggernaut to break down barriers between development and operations.
Gannett is a leading media and marketing company with unparalleled local-to-national reach, which successfully connects consumers, communities and businesses. With the iconic USA TODAY, 92 strong local media organizations in 33 states and Guam, and with more than 160 local news brands online in the U.K., Gannett provides rich content through hundreds of outstanding affiliated digital, mobile and print products.
Each month more than 95 million unique visitors access content from USA TODAY and Gannett’s local media organizations, putting the company squarely in the Top 10 U.S. news and information category. U.S. newspapers add an additional audience of 9 million readers every weekday.
You may already be familiar with Gannett’s talk, delivered at re:Invent 2015, “USA Today Brings Shadow IT into the Light.” In that talk, Franklin Hanson, Senior Manager Platform as a Service Delivery, Erik Bursch, Vice President, Platform as a Service, and Chef’s own George Miranda discussed the transformation at Gannett, which involved changes in culture, processes and tools. In this article, we talk a bit more with Frank and Erik about the changes at Gannett and Chef’s role in standardizing Gannett’s infrastructure.
Gannett’s traditional deployment workflow was characterized by multiple handoffs and manual tests. Maintaining accurate, repeatable builds was difficult. There were many build failures and tests were often run in the wrong environments. Deployment and provisioning times could range from a few days to several weeks.
There were two operations teams, each in its own silo both physically, within different data centers, and organizationally. Neither team had access to the cloud or the development environments.
On the development side, a type of “shadow IT” had emerged. Developers would spin up instances on Amazon’s EC2-Classic and personal Heroku accounts and tie them to the production DNS. There was no sort of oversight.
Other complications were that every group used its own toolset, and there was no accountability to finance or security. No one knew how much an application actually cost. Security had no way to audit the stacks.
Gannett was ready for change. Developers wanted to deploy their applications quickly. Operations wanted a stable infrastructure where they could build and deploy in a repeatable way. Finance wanted to know the true cost of an application. Security wanted to view and audit all stacks and to be able to track changes.
Finally, the cloud had to become a sanctioned way to deploy applications and no longer be an unauthorized and ungoverned choice. Gannett saw that the cloud offered many advantages. Developers had access to standardized resources. It was easier to handle peaky traffic because of cloud’s compute-on-demand model, and handoffs were minimized. In other words, it was time to bring shadow IT into the light.
John Dietz, Platform Architect
Chef has increased the effectiveness and the speed of our development cycle. We’ve been able to use the economies of scale that Chef gives us to move more quickly across the board.
The Workflow for Cookbooks
Gannett has a private Supermarket where it maintains a library of cookbooks developed by the Gannett community, and people try to use those cookbooks as much as possible. They also use cookbooks from the Chef open-source community. Role cookbooks, which have no actual Chef code in them aside from includes and attributes, are managed by individual teams. Utility cookbooks, which are used by multiple teams, are managed by Frank’s team.
Role cookbooks have one GitHub repo per team. All of the applications that use that role cookbook are also in the repo. Utility cookbooks have their own repo.
All cookbooks, whether internally developed or from the open-source community, follow the same release process. They’re first tested locally with Test Kitchen and then with ChefSpec, Foodcritc and RuboCop.
Once a cookbook passes its local tests, a pull request (PR) is generated in GitHub against the repo the cookbook belongs to. The PR is automatically tested against Test Kitchen and EC2, using a Jenkins server to run the same standardized testing. If the PR passes, it’s marked with a check box and one of the senior developers does a sanity check and makes sure that the change follows best practices.
Then, the change is merged to master and goes through the same suite of tests again. When those tests pass, the cookbook is uploaded to Supermarket. If it fails Supermarket it generally means that a dependency is missing. Otherwise, it goes from Supermarket to the Chef server. Gannett uses one Chef server for its data centers and the cloud, and it’s located in production.
All master branches are considered to be in production. Cookbooks are continuously delivered and versioning controls what goes out.
Alon Motro, Manager Content Platform Teams
Prior to using Chef, developers had very little knowledge of how to spin up a box. There was a big disconnect between development and coding versus infrastructure. With the introduction of Chef, we've empowered developers on how to spin up a box, how to deploy their code. It helps with timelines and it helps by giving developers a better understanding of what sort of infrastructure their applications need.
Over the past 18 months Gannett has made significant advances. Their deployments are quicker and more reliable. Application provisioning and deployment, which once could take weeks, now takes minutes.
All new applications are deployed to the cloud with Chef. Those applications are deployed to all environments the same way that they’re deployed to production. Also, testing occurs in each environment, so that the deployments are reliable.
All infrastructure is treated as code, which greatly increases visibility into any changes that occur. Development, operations, security and finance all benefit.
Erik says, “Chef has increased the effectiveness and the speed of our development cycle. We’ve been able to use the economies of scale that Chef gives us to move more quickly across the board. If development team X is using a new technology, they write their Chef cookbook and push it out. Development team Y comes along and chooses the same technology. That cookbook is already in place and they’re able to run it.”
Currently, about 30% of Gannett’s technology organization is using Chef but Gannett has aggressive plans for 100% adoption in 2016. Both Erik and Frank are confident that they can meet this goal. Strong grassroots support, impressive results, and a culture that is committed to going faster are some of the reasons for this confidence, but there is another component as well, which is, in addition to grassroots support, support from above.
Having visibility into the entire pipeline in terms of how the servers are conﬁgured, how the apps are conﬁgured, has allowed us to optimize not only the deployment platform but the end-user experience. We can turn on features that are really speciﬁc to one application with-out having to worry about breaking other platforms or breaking other teams, and we can speed up everything.
Jay Merrifield, Principal Developer
Erik says, “Jamshid Khazenie is our CTO and his buy-in to our automation strategy, which includes Chef, has resonated throughout our technology organization. He understands where Chef sits in our ecosystem and has reiterated that in meetings across the technology organization. That is a huge benefit to us moving forward at the speed we have. How were we able to make these gains in a short amount of time? The fact is we have a leadership directing the whole organization and saying that this is the way we will do things. We will follow this automated method that we’ve established in our Gannett cloud platform.”
Erik also cited Chef’s role in Gannett’s success. “I’ll put out a nod to you guys. You’ve been a good partner and acted as a sounding board on a lot of items. I use “partner” rather than “vendor” as a key difference in my book. You’ve worked step-by-step with us on this process. It’s been a really good relationship.”
Of course, along with 100% adoption of their automation strategy, Erik and Frank have plans for the future. While deployment rates are dramatically faster, they want to shorten them even more. Another goal is to work with their Payment Card Industry (PCI) department to begin automating those requirements. They’re also investigating how to move away from Serverspec toward Chef’s compliance language, InSpec.