Enterprise DevOps at Hearst Business Media
Q&A session with Hearst’s Vice President of DevOps, Pauly Comtois about the challenges of changing the way a large enterprise develops and delivers its software and how Chef fits into the picture.
Hearst Business Media is a global technology leader delivering information, insights, analytics, and workflow solutions to meet worldwide needs in the finance, healthcare and transportation markets. Its holdings include renowned brands such as Fitch Group, one of the leading global ratings agencies, and the Hearst Health network of market-leading healthcare companies which includes First Databank, Zynx, MCG and Homecare Homebase. These companies provide care guidance to patients, hospitals, and insurers. For transportation, there is Black Book, MOTOR Information Systems and Canadian Black Book, the premier suppliers of automotive data.
Chef did a Q&A session with Hearst’s Vice President of DevOps, Pauly Comtois. We talked about the challenges of changing the way a large enterprise develops and delivers its software and how Chef fits into the picture. What follows here is Part I of our conversation, where Pauly talks about some general principles for fostering change. In Part II, he’ll talk about how he introduced Chef to Hearst.
Chef: Could you tell us a bit about your role at Hearst?
Pauly: I have a somewhat unique title. I’m the Vice President of DevOps at Hearst Business Media. Hearst hired me to work with its business units to support the Agile and DevOps transformations they were undertaking.
Hearst embraces a mindset of continuously improving the methods and processes used to create its software. “What got us here won’t get us there” is not just a saying but a way of life at Hearst. We keep a close eye on the markets and new entrants, ensuring that we are always delivering higher quality software faster than our competitors.
Chef: What’s your starting point with a Hearst organization?
Pauly: Basically, I act as an internal, full-time consultant. My job (and also my team’s as we’ve grown) is to support each business unit’s initiatives and to help them be successful. I do have a framework that I like to use when I first come in, although it’s not prescriptive. The business unit CTO drives the roadmap and the engagement schedule. We review frameworks that have been successful at other organizations and how, quantitatively, we can measure success. This focus on collaboration builds trust between the teams and makes the engagements more effective.
Of course, there are some basic issues you always have to address. For example, you have to look at tool consolidation, achieving economies of scale by spreading tool licensing out, that whole cost containment piece. It might not be exciting but it’s something you need to do to enable the organization to grow fast and make as much of an impact on customers as possible. There is an added benefit of being able to leverage a broader community across all the business units so they can share knowledge, tools and support.
Pauly Comtois, Vice President of DevOps
We deploy Chef across multiple teams in the value stream, automating the deployment, configuration and state of environments in Dev, QA and Production.
Chef: What are some things you’ve learned since you started?
Pauly: When I first started, I began by doing the completely wrong thing. I guess if you’re going to do the wrong thing, doing it first is the best way to do it. I tried to go into all the businesses at the same time, gather information, build relationships, get a context of how things worked and engage change. I got a pretty severe bloody nose from that.
I quickly realized that this approach was a mile wide and an inch deep. I couldn’t actually accomplish anything tactically. The only way I could work, realistically, was to rigidly adhere to that framework I mentioned. I couldn’t customize per business unit because I didn’t have the bandwidth. So, that failed miserably, which is OK. I learned a lot from that experience. Instead, the CTOs now provide the context and detail for each business unit. This partnership is critical to the success of any initiative.
The CTOs drive the creation of a unique roadmap for their business. This is included in a larger roadmap that includes all the units. My team then does a dedicated and custom engagement per business unit using an Agile approach.
Although the initial approach was focused on DevOps and Agile, it has expanded to include the entire value chain. This includes the product owners, QA, the developers (especially around Agile and the CI pipeline), then the operations people and the CD pipeline.
I’ve been an evangelist, not just for Chef, but many other tools—event enrichment, logging, monitoring. It’s an entire tool chain that I’ve been pushing. When multiple teams use the same tool across the value stream, we are able to create common ground that breaks down silos. Automation and collaboration are the two main pillars we focus on.
Chef: Did anything surprise you?
Pauly: I came into this job expecting that the relations between developers and operations would be strained and siloed. I haven’t seen it anywhere. The development and operations teams get along really, really well. There is no issue of throwing it over the wall at any of the ten business units.
Chef: Could you describe the development and deployment environments at Hearst?
Pauly: We have over 2,500 folks just in dev and ops and a heterogeneous environment. Traditionally, it’s been predominantly made up of Windows machines and data centers. Over the last year, we’ve worked really hard at migrating and putting systems into EC2. There are still some data centers and probably will be for the foreseeable future. Mainly, that’s because there are some legacy applications that are rock solid, awesome applications but not ones that we’re necessarily heavily investing in for the future, so we’re just going to leave them where they are.
You can think of that as track 1. You can think of track 2 as the new, exciting enterprise stuff we’re doing in the cloud. Much of our new development uses open source in addition to .NET and Windows. That marks a significant shift for the organization and also opens the doors to exciting new things, like contributing back to the open source community.
Pauly Comtois, Vice President of DevOps
I’m not a fan of the term “Change the Culture.” I believe that if you focus your efforts on the processes, behaviors and workflows you will, over time, influence the culture. This approach allows you to include everyone in the process and the culture will follow.
Chef: What do you see as your biggest challenge?
Pauly: The big challenge is the uniqueness of each organization. For one thing, you have substantial cultural differences between each of them. You’re talking about one business unit in LA, one in Dallas, one in New York, one in Detroit, another in Miami. If you’re talking about even the standard cultural differences between humans in those geographic localities, it’s very challenging. The approaches can be drastically different. One place is much more rigid; another is much more laid back. For example, the Seattle business world is very different from the Manhattan business world, culturally.
There’s no one right answer to anything, even for adopting something with clear benefits, like Chef. It can be different from one organization to another. In some cases, you end up using CloudFormation in conjunction with using Chef. In other cases, it means a change in ways you didn’t predict. If you have a team of people and all they did was rack and stack in a data center and you don’t have a data center anymore, you provide retraining for those folks in things like CloudFormation, Terraform and other methodologies. They’re doing more or less the same thing, but now it’s in the cloud. You don’t reorg and push those people out. You change the job a little bit and you enable them through training and support so they can do the new job.
Chef: Other people who are working toward the same sort of transformation that you are say that culture is the most difficult thing to change. Do you agree?
Pauly: I’m not a fan of the term “Change the Culture.” I believe that if you focus your efforts on the processes, behaviors and workflows you will, over time, influence the culture. This approach allows you to include everyone in the process and the culture will follow. It’s important to note that you start with the culture you want in mind, and build those processes, behaviors and workflows to reach that goal.
Tools can be an incredible asset in this regard. We deploy Chef across multiple teams in the value stream, automating the deployment, configuration and state of environments in Dev, QA and Production. Deploying the tool before the culture is ready, however, can lead to failure before realizing the true benefit.
That introduction of a tool prior to cultural acceptance can clash with something I call the “enterprise immune system.” Replacing a tool is not often difficult, however people become attached to their workflows and tools. A change that is not properly explained and accepted is met with resistance from the enterprise immune system and often rejected despite its possible benefits.
With a strong focus on communicating the “why” (context of the change) first and then moving to the “how” and “what” has proven very successful. Moving slowly at first and showing value early can win over naysayers. You may not garner unanimous approval, but with a collaborative and open approach you are much more likely to enact systemic, long-term positive change with the tool. Remember, it’s not just about the tool.