Ontology, Infrastructure Classification, and the Design of Chef

An example ontology specification. CC-Attribution-NoDerivs by gertcha on Flickr

An example ontology specification. CC-Attribution-NoDerivs by gertcha on Flickr

In philosophy, ontology is (as Wikipedia says) “the study of what entities exist or can be said to exist, and how such entities can be grouped, related within a hierarchy, and subdivided according to similarities and differences.” Wikipedia goes on to say that ontology is often paired with taxonomy (the science of classification) in IT applications. Chef, however, was explicitly designed to not be an ontological system, in contrast to many other solutions on the market. Why is that? I’d like to take a few moments to explain the design thinking behind Chef — and why we feel that not being an ontology allows us to be the most flexible and extensible automation platform. Read more ›

Setting Up Your Private Supermarket Server

This is an updated version of the previous post from August, 2014: Getting started with oc-id and Supermarket

Chef Server 12 includes oc-id, the OAuth2 service that powers id.chef.io. After upgrading to this release, Chef customers can now run their own Supermarket service behind a firewall.

Read more ›

Security Release: Chef Server 12.0.8 and Enterprise Chef 11.3.1

Ohai Chefs!

Chef Server 12.0.8 and Enterprise Chef 11.3.1 are available for immediate download. This release addresses the following vulnerabilities:

This corresponds to chef-server issue 142, “Update Embedded Openresty NGINX”.

Additional Changes

Chef Server 12.0.8 has been further updated as follows:

  • The Chef Server 12.0.8 release is the first to enable Server API Versioning and sets the baseline API version at 0, while enabling versioned API behaviors for future releases. This is an internal update that has no outward effect on client or server beyond exposing a new endpoint as described in the RFC.
  • opscode-omnibus issue 744 – chef-server-ctl password command has been fixed
There have been no additional changes to Enterprise Chef 11.3.1.

Release

To apply this security update, upgrade your existing Chef Server installation to the latest available version:

Overview of Test Driven Infrastructure with Chef

This post is all about test driven infrastructure with Chef: an overview in testing Chef cookbooks for the current landscape. This post is focused on tools included in ChefDK. Some other tools and projects are mentioned for completeness or historical purposes. This post serves as general overview of the various components and tools that are currently used, or have been used in the past to do test driven infrastructure with Chef. For a full book on the topic, see Test Driven Infrastructure with Chef, by Stephen Nelson-Smith. For training materials, see this learnchef repository. Read more ›

Chef + Splunk – an integrated solution for DevOps

This is a guest post by Jon Rooney, Director of Developer Marketing at Splunk

Two weeks ago at ChefConf, the Chef team announced an integrated app for Splunk to gain real-time insights from your Chef infrastructure. The Chef Analytics App for Splunk is available for free on Splunkbase, the Splunk app marketplace and provides Chef users with visibility into metrics such as success / failure rates, most active users and most active organizations. The Chef Analytics App for Splunk also helps you understand the frequency of the details of errors across infrastructure so that you can catch and troubleshoot high impact issues, like a major bug in a cookbook or an infrastructure issue like network connectivity, in real time.

With the Chef Analytics App for Splunk, you can identify any non-idempotent resources to make your cookbooks safer.

Untitled

 

Untitled2

 

Getting the Chef Analytics App for Splunk running requires just a few steps to enable the integration service—just configure notifications for your Splunk instance and specify rules for sending data. For details, see installation instructions. Once the data gets into Splunk, you can query it using the Splunk Search Language (SPL) or by downloading and installing the app with pre-built dashboards for monitoring. The Chef Analytics App for Splunk is also fully extensible, so you can easily customize the panels and searches driving it.

Notably, with the help of the newly released Splunk Developer Guidance, the Chef team (thanks Serdar!) built the integration and the Splunk app in less than week.

Together with Chef and Splunk, you can now fine-tune your workflows for continuous delivery!

Integrating Chef Analytics with Splunk

Ohai Chefs,

One of the exciting features in Chef Analytics 1.1.2 that is the ability to link Chef Analytics to Splunk. This features give you the ability to extract meaningful insights about your Chef infrastructure if you are using Splunk.

We have also created a basic Splunk App for Chef Analytics that showcases a few things you can do. Here are some screenshots from this app:

Nodes Activity Dashboard Screenshot Server Activity Dashboard Screenshot

To give this a try, check out the instructions on https://github.com/chef/analytics-splunk-app and create an issue here if you run into any problems.

Validatorless Bootstraps

Starting with the Chef 12.2.0 Client there is no longer any need to use the validation key to provision new chef nodes with knife. Furthermore, all that needs to be done to take advantage of this feature is to delete your validation keys and optionally remove the validator configuration from your knife.rb file.

Instead of shipping a validation key up to the newly provisioned node and having the node use the validation key to authorize itself to provision a new client and node, the knife bootstrap command will use the user’s key to create a client key for the node, use the client key to create a node object, and then ship the client key up to the node.

Configuration Details

Starting with Chef 12.2.0 existing Users will begin seeing a new banner on new knife bootstraps:

Doing old-style registration with the validation key at /home/lamont/.chef/myorg-validator.pem...
Delete your validation key in order to use your user credentials instead

In order to use the new behavior it is as simple as deleting the validator key:

rm -f /home/lamont/.chef/myorg-validator.pem

The existing validation_client_name and validation_key parameters in the knife.rb file can also be deleted. Note that the default value of the validation_key is “/etc/chef/validation.pem” and if that file happens to exist on the workstation or server that it will attempt to be used after removing the validation_key setting. That file should either also be deleted, or else the validation_key should be set to something like “/nonexist” to disable it.

Provisioning Details

The new output of knife bootstrap when not using a validation key will look similar to:

desktop% knife bootstrap 10.1.1.1 -N foo01.acme.org \
   -E dev -r 'role[base] -j '{ "foo": "bar" }' \
   --ssh-user vagrant --sudo
Node foo01.acme.org exists, overwrite it? (Y/N)
Client foo01.acme.org exists, overwrite it? (Y/N) 
Creating new client for foo01.acme.org
Creating new node for foo01.acme.org
Connecting to 10.1.1.1
10.1.1.1 Starting first Chef Client run...
[....etc...]

What you can see here is that if the node and client already exist that knife bootstrap will prompt to overwrite them. The ‘-y’ command line flag can be used to skip the prompts and answer ‘y’ to both questions. The new client is created first and the new node is then created with the client key.

Behind the scenes the ‘-r’ and ‘-E’ and ‘-j’ flags to knife bootstrap are already applied to the new node which gets created — so the object in the database will have its run_list, environment and initial ‘normal’ json attributes saved. This avoids the edge condition where for some reason if the initial chef-client run fails the node is never saved and it ‘forgets’ its own run_list and environment. Since the node is saved with that correct data before provisioning starts on the host, the run_list will still be correct even if the initial chef-client run fails for some reason.

Summary

The validatorless bootstrap changes to Chef 12.2.0 achieve a few key things:

  • No more need for the validation key (fewer things, reduced fussiness)
  • Ability to eliminate shared access (and ultimately have better auditing around provisioning actions)
  • Eliminating the first-run failure edge conditions where a node forgets its run_list, environment or attributes
  • Gracefully handling the situation where an old Client key or Node object exist in the database

ChefConf 2015: DevOps, Velocity, and Community

ChefConf 2015 welcomed 1500 attendees and 42 sponsors over our four day event. If you weren’t able to attend or just want to relive it all over again, here are a few of the highlights:

Keynotes:

Read more ›

Standard Bank: Our DevOps Journey Blog Posts

For those of you who would like to read Standard Bank: Our DevOps Journey in its entirety, here are the blog posts.

Standard Bank: Our DevOps Journey – The Final Chapter (Part 6)

This is the sixth and final entry in our ongoing, bi-weekly series examining our customer Standard Bank’s DevOps journey. You can read the first entry here, the second entry here, the third entry here, the fourth entry here, and the fifth entry here. Continue below for part six.

On February 11, the Chop Chop team went live with their prepaid feature. Currently, it’s available internally, on the Standard Bank network, but anyone who’s a part of that network can use it. The next day, there was a large, internal IT conference at Standard Bank where executives discussed what they want to do in the coming year. Dawie Olivier gave a presentation, showing how the app was promoted through the different quality gates into production, and how a person could log on and use it.

The Numbers

The Chop Chop team gathered some metrics to demonstrate what they’ve accomplished with Chef and DevOps.
  • Time to build the stack: 26 minutes to build for production nodes (2 web servers, 2 app servers, with end to end deployment and testing)
  • Number of automated tests embedded into Bamboo:
o   Test Environment – Total 209 Tests
  • 31 Infrastructure
  • 178 Application – already existed before Chop Chop
o   Production Environment – Total 39 Tests
  • 31 Infrastructure
  • 8 Application – written by Chop Chop
  • Number of cookbooks – 12 (5 custom, 7 community)
  • Time to market for pilot Internet Bank Refresh deployment – 12 weeks
  • Number of catalogued automated services: 3
o   Redhat Linux VM

o   Enterprise Application Platform (EAP)/JBoss on Redhat

o   Apache on Redhat

Read more ›