Chef Server 12.2.0 Release Announcement

Ohai Chefs!

I’m happy to announce that Chef Server 12.2.0 is now live on Hosted Chef, and is also available for download on the Chef Downloads Page and via our Apt/RPM repositories. Here are some of this release’s highlights:

  • External PostgreSQL: we now support using a PostgreSQL server other than the one shipped with Chef Server, including Amazon RDS.
  • Policyfile API enhancements.
  • Organization policy changes.
  • New chef-server-ctl commands and enhancements.
  • Several bug fixes. Read more ›
Posted in announcements, release Tagged with: ,

What’s Up With The Chef Community Summit?

With the Chef Community Summits (Seattle and London) fast approaching, I wanted to share my experiences as a new member of the Chef community about the value of the Chef Community Summit.

Last fall, I attended the Chef Community Summit in Seattle. It was my first Chef community event larger than a user group. I really didn’t know what to expect, especially given the schedule which was two days of open spaces. I was excited to meet more of the Chef community. I was worried that, as a Windows Server admin, I wouldn’t find discussions that would be relevant to me. I was nervous that my short exposure to Chef would make it difficult to join in or marginalize my contributions to the discussions.

I didn’t have to worry – the Chef Summit really cemented for me what it means to be part of the Chef community. There were people with a wide range of Chef experience (though, to be fair, most of the audience was pretty familiar with Chef). The Summit started out with an opening circle, where the format and plan for the event were discussed and people were allowed to submit suggestions for topics. I was surprised to see several topics suggested on Windows Server related topics – especially around testing and remote management – and they weren’t from me!

Read more ›

Posted in community

Lets build a Minecraft Server with Chef

There are many ways to use Chef and not all of them have to be related to your day job. If you’re a fan of Minecraft you can use Chef to create a Minecraft server on Digital Ocean. The initial setup is only done the first time, then you’ll be able to do just the create command when you want a new Minecraft Server. Read more ›
Posted in community

Chef Analytics 1.1.6 Released

Ohai Chefs,

We are pleased to announce that Chef Analytics 1.1.6 is now available. This is a bug fix release; details of the issues resolved are provided below.

Fixes

  • Data was not appearing in the nodes view for organizations having more than 25 nodes.
  • Data was not appearing in the nodes view when switching organizations.
  • Analytics was not working properly when using a certificate that didn’t have localhost as a Subject Alternative Name. This issue was introduced in release 1.1.4. We have now removed localhost from the Subject Alternative Name and rely on a custom hostname verifier to allow localhost connections. Certificates generated by Chef no longer include Subject Alternative Name.
  • Nginx SSL port setting from the opscode-analytics.rb attributes file was not being applied properly. The fix for this issue allows you to operate Chef Analytics on a Chef Server, e.g. for AMIs.
  • Zookeeper snapCount settings were resulting in large on-disk log size. In verification testing, this change to the snapCount settings resulted in 80-90% reduction of log size.
 

Where can I get it?

As always, you can download new releases of Analytics from downloads.chef.io.

Posted in community

Chef Push 2.0 alpha is available

Push 2.0 alpha is out

This has all of the major features planned for 2.0, including:

  • Encryption
  • Output capture
  • Environment control
  • Server Sent Event (SSE) feeds
This is an alpha release, so there will be bugs. Please file issues as you find them against Chef Push Issues.

Push 2.0 client and server are not backward compatible with 1.x versions. We are investigating what it would take to make the 2.0 client able to work with the 1.x server, but that isn’t there yet.

Current known issues in the alpha-3 release which will be fixed before 2.0 final include:

  • The windows client is broken; it apparently is having trouble with the zeromq library.
  • Renaming; push has had a lot of names over the years and that’s being cleaned up.
  • knife push client is incomplete; SSE feeds aren’t used by the knife command.
  • RHEL7 support for client/server
  • Documentation of the APIs and features
  • Testing of server and client upgrades from the 1.x series.
Some of these changes may be breaking changes; the renaming in particular may make it difficult to upgrade directly from one alpha to the next.

Brief outline of the new features:

Encryption

All communications take place over SSL or CurveZMQ. CurveZMQ is based on the CurveCP protocol. The one exception to this is the server heartbeat, which is broadcast in the clear (but is still signed with the server key for integrity).

Command Output Capture

The knife-push library now provides options to direct the client to capture the job output and return it to the server for inspection:
% knife job start "echo foobar" test --capture
 Started. Job ID: 26e98ba162fa7ba6fb2793125553c7ae
 .Complete.
 % knife job output 26e98ba162fa7ba6fb2793125553c7ae test --channel stdout
 foobar

Environment Control

The user has a lot more control over the execution environment of the remote command.

This includes:

  • Environment variables (‘–with-env’)
  • The execution directory (‘–in-dir’)
  • A data file sent from the user to the push client (‘–file’)
% knife job start "print_execution_environment" test
--file .chef/knife.rb --capture --with-env '{"test": "foo"}'
--in-dir "/tmp" --as-user daemon
 Started. Job ID: 26e98ba162fac37787292637362808cb
 ...
 % knife job output 26e98ba162fac37787292637362808cb test --channel stdout
 {"HOME"=>"/home/vagrant",
 ...
 "CHEF_PUSH_JOB_FILE"=>"/tmp/pushy/pushy_file20150813-14250-125xv4u",
 "CHEF_PUSH_JOB_ID"=>"26e98ba162fac37787292637362808cb",
 "CHEF_PUSH_NODE_NAME"=>"test",
 "PWD"=>"/srv/piab/mounts/opscode-pushy-client",
 "SHELL"=>"/bin/bash",
 "test"=>"foo"}
Note that there are some new special environment variables:
  • CHEF_PUSH_JOB_FILE: The path to the file sent from the server
  • CHEF_PUSH_JOB_ID: The id of the push job being executed
  • CHEF_PUSH_JOB_NODE_NAME: The node name that the job is being executed on

Server Sent Event Feeds

There are two new endpoints that provide feeds for the state of jobs on the server. There’s a per-org-feed, that provides high level job start/completion information, and a per job feed that provides node level state changes for a particular job.

The event feed for a job might look like:

id: 1
 event: start
 data: {"command": "chef-client", "run_timeout": 60, ...}
 id: 2
 event: quorum_vote
 data: {"node": "moe", "status": "success"}
 id: 3
 event: quorum_succeeded
 data:
 id: 4
 event:run_start
 data:
 id: 5
 event: run_complete
 data: {"node": "moe", "status": "success"}
 id: 6
 event: job_complete
 data: {"status": "complete"}
The knife-push plugin will support SSE feeds in a later release.

Getting started

You’ll need to download the latest knife-push plugin (0.9 or later) (github for now, rubygems soon) as well as the latest client and server from packagecloud current

At the time of writing those are push-jobs-client-2.0.0~alpha.3-1 and opscode-push-jobs-server-2.0.0-alpha-3.1.

Posted in announcements, chef push, engineering Tagged with:

Chef and AWS Bring DevOps to AWS London Loft

Chef and AWS Meet the Rising Demand for DevOps Training in the UK

LONDON – Aug. 20, 2015 – Chef, the leader in automation for DevOps, today announced it is bringing its DevOps expertise to the new Amazon Web Services (AWS) Pop-up Loft in London. The AWS Pop-up Loft will open on Sept. 10, 2015 and Chef will offer hosted sessions and training curriculum to developers, engineers, entrepreneurs and tech enthusiasts visiting the Loft.

Chef’s support in the new London Loft builds on the success of the sessions hosted in the AWS Pop-up Lofts in San Francisco and New York City.

Read more ›

Posted in announcements, cloud, partners, press releases

Join us at VMworld 2015

VMworld 2015 kicks of in San Francisco, CA, on August 31, and your #cheffriends will be out in force. Visit us at booth 2523 to talk Chef, automation, DevOps, and, of course, share some #hugops.

We’re also participating in a number of presentations, including:

Looking forward to seeing you all in San Francisco!

yoVT3vYf

Posted in chef, community, events, partners

Chef Management Console 1.21.0 Released

Manage 1.21.0 is now available from the Chef downloads site.

This release fixes a bug with password resets returning a 500 (see the issue here) that was introduced in Manage 1.19.0. It also adds a configurable session timeout setting (session_timeout_absolute) independent of the previous activity-based session timeout setting that will log users out after a fixed time (by default, one week — see the Manage docs for more information).

A full change log for this release can be found at https://manage.chef.io/changelog.

Thank you for using Chef!

Posted in release Tagged with:

Policyfiles: A Guided Tour

If you watched the ChefConf keynote, attended last years’ community summits, or follow our open source mailing lists, you’ve probably heard about Policyfiles.

If you haven’t, here’s the deal: Policies are a new feature of Chef that combine the very best parts of Roles, Environments, and client-side dependency resolvers like Berkshelf into a single easy to use workflow. Policies are built by defining a Policyfile, which looks similar to a Chef Role combined with a Berksfile. When a Policy is ready for upload, a workstation command included with the ChefDK compiles the Policyfile into a Policyfile.lock file. This locked Policy, along with all of the cookbooks it references, are treated as a single unit by the Chef tooling. The bundle of Policyfile.lock and cookbooks are uploaded to the server simultaneously. They are also promoted simultaneously through the deployment lifecycle, from dev to QA to production.

Policies make your chef-client runs completely repeatable, because cookbooks referenced in a Policy are identified by a unique hash based on their contents. This means that once the lock file + cookbook bundle has been generated, the code underlying it will never change.

For more information, see the Policyfile README in the ChefDK repo.

Previously we’ve recommended that you only use Policyfiles in specialized testing environments because using Policyfiles in compatibility mode along side existing infrastructure could cause unexpected behavior. With Chef Server 12.1 and ChefDK 0.7, Policyfile data is now stored via specialized APIs, making it safe (and a lot easier) to use Policyfiles in your existing Chef Server setup.

To help you get familiar with the workflows Policyfiles make possible, we’ll walk through deploying a simple demo application using Policyfiles to manage our dependencies across the application’s lifecycle.

The code here is based on a demo my colleague created for the London Chef meetup. It deploys the “Awesome Appliance Repair” Python application.

Read more ›

Posted in chefdk, engineering, featured

Chef and DevOps at JustGiving

JustGiving is the world’s largest fundraising platform. In the last twelve months, JustGiving has helped tens of millions of people raise over $700,000,000 dollars.

This post describes how JustGiving uses Chef and DevOps to help manage its infrastructure and create a flexible, cooperative work environment. Contributors were Richard Atkinson, the Chief Information Officer, Owain Perry, Software Architect, and Mark Chamberlain, Head of Services.

JustGiving is 15 years old. Historically, it has used Windows but, over the last few years, it has also begun to use Linux. There have been other changes as well. Originally, the company used managed data centers, but they are moving towards an architectural pattern of microservices that run on Amazon Web Services. New services are immediately deployed to the cloud while the legacy codebase is gradually being transformed to follow this new pattern.

The company began using Chef at about the same time that they began to adopt the microservices approach. Currently, Chef automates a large percentage of JustGiving’s cloud-based web operations. In addition, their continuous delivery pipeline for deploying microservices is configured and managed with Chef. They also use Chef for cloud orchestration—servers in the cloud are managed using Chef resources.

They use hundreds of cookbooks. Each microservice has its own cookbook, and there are others for running the infrastructure. For example, JustGiving has created a cookbook for provisioning cloud instances that uses heavyweight resource providers (HWRPs). JustGiving also uses some community cookbooks.

The company has adopted DevOps principles, both social and technological. Originally, the Operations team was separated from the Development team but, gradually, barriers between teams came down, and they began to collaborate. The team that writes the infrastructure code is now co-located with the ServiceOps team.

The ServiceOps team is responsible for running all production services. In close collaboration with the Development team, they maintain a backlog of tasks, and use a quasi sprint-planning process that produces two-week cycles. They use JIRA to manage the process.

There are a variety of other teams as well, each with specific skills such as front-end development, backend development, quality assurance, user experience design, and architecture.

Richard Atkinson says,

“The teams come together in ‘swarms,’ which have all the skills they need to deliver to business objectives, and include nontechnical skills such as marketing. Everyone sits on the same floor. We also have blameless post mortems. They are vital to self-improvement after a failure or an outage.”
Members of the teams are free to work from home or anywhere around the office. Everyone has a high-spec laptop and there is a range of tools to help foster cohesion, even when working remotely. For example, JustGiving uses JIRA, HipChat and cloud repositories in GitHub.

JustGiving has a continuous deployment pipeline that is used for both infrastructure and application code. They use TeamCity as the build server and Go to manage the pipeline. The development team can use the pipeline to provision their own infrastructure as they need to.

Owain Perry described the workflow as code moved from a developer’s laptop to production.

“When a developer has a requirement for a new service, they copy a template Git repository, fill in the blanks, which includes a Chef cookbook for the service, create (clone) the relevant pipelines in TeamCity and Go and commit their hello world application.”
They then clone the infrastructure Git repository and define the infrastructure that’s required in a data-driven document and a Chef recipe, and then push the change. The document and recipe will build their infrastructure all the way into production.

From this point onwards, developers can create code on their laptops and push changes. The code is compiled and unit tested in TeamCity; a versioned package is stored in Artifactory; and a versioned cookbook is stored in Chef.

The Go pipeline can deploy that version of the service through our environments into production. After each deployment into an environment, including production, post-deployment tests are executed. If the tests are happy, the environment is switched from inactive to active (blue/green) to turn the code on.

Richard Atkinson discussed the effect Chef has had on JustGiving.

“As well as speeding up our deployment cycle such that we’re releasing code almost constantly, there have been situations where we have turned around hotfixes within an hour of spotting symptoms. We have seen service resilience improve as services are removed from the production environment to be replaced by new instances. The deployment process is less disruptive to end users, with blue/green deployments keeping service capacity online throughout.”
Mark Chamberlain added,
“We spin up environments regularly due to demand spikes. We also use Chef to power down/up our staging and development environments when they aren’t being used overnight, producing reasonable savings. We have pre-approved configurations available, and we have moved on to baking full machine images, using Chef, that enter our configuration and deployment process. We see this as an area for development. We feed the Chef configuration and change logs into our environment, which is an ELK (Elasticsearch, Kibana) stack. We have microservice and cookbook changes automated in our staging and production environments so that there is a central change JIRA ticket for every change in these environments.”
Looking toward the future, Richard is excited at the prospects.
“What I see today with DevOps reminds me of my early career where I saw the rise of concurrent engineering methodologies in aerospace organizations, where production expertise was being extended upstream to ensure products were being designed for factors other than utility, such as manufacturability.

“I think there are interesting parallels from history, too. Ask anyone who invented the car and most people would mention names like Karl Benz or Henry Ford, but actually the functional principles were invented 200 years earlier by Verbiest. What Messrs. Benz and Ford did was make the car a reality for the masses by removing blockers to economic scale and performance. Where am I going with this? I believe computing has not yet had its Ford Model T moment, but history will show that now is the defining era when the conditions were created, through solutions like Chef, for IT to go mainstream. That’s cloud. That’s mobility. That’s DevOps. Although we may think computing has been done, the best is yet to come.”

 

ford-chef-01

Posted in awesome chefs, chef, community, customers, devops, EMEA