How to Set Up Progress Chef Automate HA for Managed AWS Service


Progress Chef Automate is an enterprise platform that allows developers, operations and security engineers to collaborate and deliver application and infrastructure changes more effectively. As software deployment speed increases across the organization, consistent real-time collaboration with high availability (HA) becomes critical. Chef Automate facilitates this collaboration by allowing different teams to accomplish various tasks simultaneously through the interface.

Chef Automate provides tools for local development and integration points, including APIs, software development kits (SDKs) and deployment pipelines that support typical workflows.

What is High Availability?

HA refers to a system or application offering high operational availability. This means the entire site or application will not be down if one server goes down due to traffic overload or other issues. HA implies that the application remains available with zero interruption. HA is achieved when an application continues to operate, even when one or more underlying components fail.

Thus, HA is designed to avoid loss of service by reducing failures and minimizing unscheduled downtimes caused by power outages or component losses.

In this blog, we learned about installing Chef Automate HA in detail. We will look at the concepts of OpenSearch, PostgreSQL, Bastion host and VPC with their detailed installation and configuration steps concerning Automate HA for AWS Managed Services.

Chef Automate HA Setup_Image_1


  • AWS Virtual Private Cloud (VPC):
  •       o Three private subnets
          o Three public subnets

  • Bastion host (t2.large, 140GB Disk)op
  • SQL Database:
  •       o AWS RDS PostgreSQL: Version 13.5

    • NoSQL Database:
          o AWS OpenSearch: Version 1.3

    • AWS S3 bucket

    • AWS credentials certificates:
          o Infra
          o Automate  

    Setting up your AWS Infrastructure

    Virtual Private Cloud (VPC)

    More details on VPC


    Here, we have used a self-signed certificate for easier understanding.

    Amazon S3 bucket

    Learn more information on Backup and Restore with Managed Service.

    Identity and Access Management (IAM)

  • Get AWS credentials (aws_access_key_id and aws_secret_access_key) with privileges:
  •       o AmazonS3FullAccess
          o AdministratorAccess

    Learn more about creating IAM Users.

    Bastion Host

  • Network Setting -> VPC
  •       o Select the VPC created during the first step
          o Select the public subnet

  • Configuration Storage
  •       o Increase the Root volume equal to or greater than 140GB

    Setting up OpenSearch

  • Search for OpenSearch in the search box and select Amazon OpenSearch
  • Select the Standard Create option and enter a valid Domain name
  • Select the Dev/test template from the templates section
  • Select Domain without standby for Deployment Options with 3 Availability Zones
  • Select version 1.3
  • Select as an instance type with three as the number of nodes
  • Choose Storage type and volume type based on your requirements
  • Select the previously created VPC and add all the public subnets with the help of the dropdown
  • Select the security group based on the VPC and bastion host
  • Select Create master user from the section titled Fine-grained Access Control
  • Provide a username and choose a secure password
  • Select Configure domain level access policy from the section Action policy
          o Make sure to select Allow under Action
  • You can confirm your OpenSearch installation by checking the Cluster health.

    If you have trouble accessing OpenSearch UI, you need to configure external OpenSearch.

    Setting Up RDS

  • Search for RDS in the search box and select Managed Relational Database Service
  • Create a new Database
  • Choose Standard Create from the creation method
  • Select PostgreSQL from the Engine options
  • Select the Engine version (13.X version)
  • Choose Production with Multi-AZ DB instance from the Templates and Availability sections
  • Provide a naming convention for your DB identifier with proper credentials settings
  • Choose Standard classes (db.m6i.large)
  • Enable storage autoscaling
  • Select Don’t connect to an EC2 compute resource and IPV4 from the Connectivity section
  • Select your VPC under Virtual Private Cloud with public access set to No
  • Select Choose existing from the VPC firewall section
  • Keep the default setting for RDS Proxy
  • Keep Database authentication as Password authentication
  • Keep the default settings for Monitoring
  • You can confirm your RDS installation with the status sign (Available)

    Automate HA Setup

  • SSH to your Bastion host
  • Run the following commands as sudo to download the latest Automate CLI and Airgapped Bundle:

  • $ curl \ | gunzip - > chef-automate && chmod +x chef-automate \ | cp -f chef-automate /usr/bin/chef-automate

    The previous commands will download the Latest Automate CLI

    $ curl -o automate.aib

    The previous command will download the Latest Airgapped Bundle.

    $ chef-automate config gen config.toml

    The previous command will generate config.toml

    You can also generate a configuration file using the init-config subcommand.

    $ chef-automate init-config-ha aws

    $ mkdir -p ~/.aws echo "[default]" >> ~/.aws/credentials echo "awsaccesskeyid=AKIAXHDZVFZXXXXXXX" >> ~/.aws/credentials echo "awssecretaccesskey=n8x1hYXXXXXXXGg6PIKXPy5eM0X/Tn" >> ~/.aws/credentials

    Note: You can also assign the bastion host with the IAM roles. If you choose to assign the role via the above command, you might need to restart the instance.

    Upload your SSH key Pair to Bastion Machine and note down the file's path. Your config.toml file will reference this later.

    Update config.toml file with all the relevant details.


  • Section: []
  •       o Give ssh_user - Which has access to all the machines.
               Example: ubuntu

          o ssh_group_name - Make sure the given group name is available in all machines. This value will default to sshuser.
                Example: ubutnu

          o Give ssh_port - If your AMI runs on a custom ssh port. The default value is 22.
               Example: 22

          o Give ssh_key_file path - Downloaded from AWS SSH Key Pair, which you want to use to create all the VMs. This will let you access all the VMs.
               Example: /tmp/xyz-automate.pem

          o Set backup_config to "s3" or efs
               Example: s3
               If backup_config is s3, uncomment and set the value for the following s3bucketName attribute to your bucket name. If the bucket name does not exist, it will be created for you automatically.

  • Section: [automate.config]
  •       o Set admin_password to access Chef Automate UI for user admin.
              Example: Admin2Admin

          o Don’t set fqdn for the AWS deployment.

          o Set instance count for Chef Automate.
               Example: 1

  • Section: []
  •       o Set instance count for Chef server.
               Example: 1

  • Section: [opensearch.config]
  •       o Set instance count for OpenSearch.
               Example: 0

  • Section: [postgresql.config]
  •       o Set instance count for PostgreSQL.
               Example: 0

  • Section: [aws.config] – AWS Network Config
  •       o Set profile. The default value of the profile is "default."
               Example: default

          o Set region. The default value of the region is "us-east-2."
               Example: us-east-2

          o Set awsvpcid, created in the Prerequisites step.
               Example: "vpc741sac4c".

          o Set awscidrblockaddr
               Example:

          o If AWS VPC uses Subnet, set private_custom_subnets to the three private subnets we created in the prerequisites step and public_custom_subnets to the three public subnets we created in the prerequisites step.
               Example: private_custom_subnets = ["subnet-XXXXXX", "subnet-XXXXX", "subnet-XXXXX"]
               Example: public_custom_subnets = ["subnet-XXXXXX", "subnet-XXXXX", "subnet-XXXXX"]

          o Set ssh_key_pair_name, an SSH Key Pair created as a prerequisite. The pair value should be the name of the AWS SSH Key Pair without a .pem extension. The ssh key content should be the same as the content of the sshkeyfile.
               Example: akshay-automate

  • Section: [aws.config] – Managed Services
  •       o Set setup_managed_services, as these deployment steps are for Managed Services AWS Deployment. The default value is false.
              Example: true

          o Set managed_opensearch_domain_name
              Example: implement-automate-ha-os

          o Set managed_opensearch_domain_url
               Example:

          o Set managed_opensearch_username
               Example: opensearch

          o Set managed_opensearch_user_password
              Example: Admin1234!

          o Set managed_rds_instance_url
              Example:

          o Set managed_rds_superuser_username
              Example: Postgres

          o Set managed_rds_superuser_password
               Example: Admin1234!

          o Set managed_rds_dbuser_username
               Example: Postgres

          o Set managed_rds_dbuser_password
               Example: Admin1234!

  • Section: [aws.config] – EC2 Instance Config
  •       o Set the ami_id value, which depends on the AWS Region and the Operating System image you want to use.
               Example: ami-05147510eb2885c8

          o Set Instance Type for automate_server_instance_type
              Example: m5.large

          o Set Instance Type for chef_server_instance_type
              Example: m5.large

          o Set opensearch_server_instance_type
              Example: m5.large

          o Set postgresql_server_instance_type
              Example: m5.large

          o Set automate_lb_certificate_arn with the arn value of the Certificate created in AWS ACM for the DNS entry of
               Example: arn:aws:acm:us-east-2:155397717:certificate/XXXXXX

          o Set chef_server_lb_certificate_arn with the arn value of the Certificate created in AWS ACM for the DNS entry of
               Example: arn:aws:acm:us-east-2:155397717:certificate/XXXXXX

          o Set automate_ebs_volume_iops
               Example: 100

          o Set automate_ebs_volume_size
               Example: 50

          o Set automate_ebs_volume_type
               Example: gp2

          o Set chef_ebs_volume_iops
               Example: 100

          o Set chef_ebs_volume_size
               Example: 50

          o Set chef_ebs_volume_type
               Example: gp2

          o Set opensearch_ebs_volume_iops
               Example: 100

          o Set opensearch_ebs_volume_size
               Example: 50

          o Set opensearch_ebs_volume_type
               Example: gp2

          o Set postgresql_ebs_volume_iops
               Example: 100

          o Set postgresql_ebs_volume_size
               Example: 50

          o Set postgresql_ebs_volume_type
               Example: gp2

  • Section: [aws.config] – EC2 Instance Type
  •       o X-Contact = [email protected]

          o X-Dept = “TPMM”

          o X-Project = “AutomateHA”

    Save the config.toml file.

    Run the following command as sudo.

    $ chef-automate provision-infra config.toml --airgap-bundle automate.aib

    The above command will provision “automate.aib” with the set “config.toml” file.

    $ chef-automate verify config –c config.toml

    The above command will check for any misconfiguration in the config.toml file.

    $ chef-automate deploy config.toml --airgap-bundle automate.aib

    The above command will deploy “automate.aib” with set “config.toml” file.

    Check the status of the Automate HA services, by running the following command:

    $ chef-automate status summary

    Get all the information about the deployment of Automate HA, by running the following command:

    $ chef-automate info

    Access the credentials file, by running the following command:

    $ sudo cat automate-credential.toml

    Posted in:

    Akshay Parvatikar

    Akshay Parvatikar is a Technical Product Marketing Manager at Progress. With a career of over seven years and a bachelor's degree in Engineering, Akshay has worked in various roles such as solution engineering, customer consulting, and business development in web performance for Telecom and the e-commerce industry.