Chef Automate Now Supports Ingestion of Large Compliance Reports

Chef Automate previously had a restriction on the size of Compliance reports it could ingest. Any report beyond 4MB in size was automatically rejected.  However, with elaborate profiles and controls, Compliance reports often extend beyond the 4MB size limitation. Hence, we enhanced Chef Automate in our latest release to support Compliance reports of bigger sizes.

The new Chef Automate version 4.2.22 can now support the ingestion, display, and export of large Compliance reports.

Challenges

The challenge for Chef Automate was not only in data ingestion but also in displaying and exporting this data. Since not all users might have large Compliance reports to be fed to Chef Automate, it was necessary to find a way that supports existing data sets as well as large Compliance data sets.

Architecture Change

The latest enhancement in Chef Automate that enables the ingestion and management of large Compliance reports can be explained in the following diagram.

Firstly, Chef Automate would require a Minio Object Storage Bucket or AWS S3 bucket that is available out of the Automate ecosystem and is reachable from the machine running Chef Automate.

Next, the data flowing into Chef Automate will be pushed to this Minio Bucket/AWS S3 without any modifications while Chef Automate stores a copy in OpenSearch with some modifications. These modifications are needed for keyword search and display in the User Interface.

Enabling Ingestion of Large Compliance Reports

Since not every user might work with large Compliance reports, this enhancement is not available by default in Chef Automate. In order to enable ingestion of large data sets, the following changes need to patch into a running Chef Automate instance:

[global.v1.external.minio]
   endpoint = "<minio server end point>:<port>"
   root_user = "<username>"
   root_password = "<password>"

[global.v1.large_reporting]
   enable_large_reporting = true

Please refer to the following document for detailed instructions.

Impact of the Change

This change will allow ingestion of large Compliance reports but will have some impact on the ingestion performance. This is due to the introduction of a new component in  data processing which needs to be communicated over a network. We recommend considering this benchmark test summary report for use.

Exporting Compliance reports from the Compliance tab in Chef Automate will now be a two-step process:

  • Generation of Report
  • Download of Report

Refer here for the detailed steps.

Posted in:
Tags:

Durga Sarat Chandra Maddu

Durga Sarat Chandra Maddu is the Principal Software Engineer at Progress Chef.

Kallol Roy

Kallol Roy is the Software Engineering Manager at Progress Chef

Ankur Mundhra

Ankur Mundhra is the Senior Product Manager at Progress Chef

Chefconf Blog Banner Bottom

Chefconf September 11-14 2022
image_334

FOLLOW @CHEF