Blog-Delivery_100x385

Advanced Remediation Guidance for Elastic Search in Chef Automate 2.0 and Backend

By

Guidance for users who completed the Chef Backend 2.2.0 migration and are still being alerted by scanning tools that Elastic Search may have a vulnerability.   

We’ve recently been notified by customers who took the actions outlined in the blog “Is Chef vulnerable to CVE-2021-44228 (Log4j)?” to remediate the Log4J vulnerabilities reported in December, that their security scanning tools are picking up that Elastic Search may have vulnerabilities associated with it.  What the scanning tools may be identifying is that – even though the JNDI is disallowed in security file – the Java class file is still present and that there is still a risk that any system could override the configuration and call into those methods.   

This blog post shows how to identify and remove the class file and remediate Log4J the way that Elastic Search has patched their product since the vulnerability discovery. 

Chef’s use of Elastic Search does not use any of these macros in our logging, nor do we make any lookups to LDAP: in effect, we do not require this Log4J file and removing it does not impact any Chef functionality. 

The additional steps below are being supplemental to the initial guidance: 

The basic fix is to disable LDAP lookups by adding to: /etc/elasticsearch/jvm.options
File the following flag:-Dlog4j2.formatMsgNoLookups=true as described in https://gist.github.com/teknofire/a62ce04fa9201f72e691db5e2fee3a05 ;   
In Chef Backend’s Elastic Search this configuration should go in /etc/chef-backend/chef-backend.rb and you will have to do chef-backend-ctl reconfigure to reapply the configuration. 

The steps in this article are not required, however they may give additional assurance in cases where a customer cannot upgrade their Elastic Search instance or cluster immediately, and wants to apply the same steps ES performed to remediate and patch their versions for this one specific vulnerability.  Note as well, that many enterprise firewalls already block this exploit, even in a completely unpatched state. 

As always, you should practice the steps in your own QA environment before modifying a production system.  The paths below are representative of the default installation: your system may have a different directory for Elastic Search.  If you have multiple ES nodes, you should apply any changes to all nodes. 

Method 1 – use the Google toolset to find the JAR file and remove the JndiLookup.class file automatically 

One of the Googlers wrote a tool that searches for compromised class files and can delete the JndiLookup.class file inside the JAR.  The following set of steps were taken on a Chef installation: 

  1. Download Go onto the Elastic Search server, if it is not already there. 
  2. Download the log4jscanner from https://github.com/google/log4jscanner  

    go install github.com/google/log4jscanner@latest 

  3. Locate the directory where Elastic Search JAR files are located and rewrite log4j-core-*.jar  Typically, this is in /usr/share/elasticsearch.

    log4jscanner --rewrite /myfolder-with-jar-files 
    zipinfo /myfolder-with-jar-files/log4j-core-2.12.1.jar  
    | grep Jndi 
This should show no remaining JndiLookup.class files in the JAR.  Since JAR files are effectively ZIP files, zipinfo comes in handy here. 

 

Method 2 – manually find the JAR file and remove the JndiLookup.class file 

If you do step #3 above and locate the Elastic Search JAR files, including the log4j core ones, you can use ZIP to delete a specific file out of the JAR: 

zip -q -d log4j-core-*.jar
org/apache/logging/log4j/core/lookup/JndiLookup.class 

Rerun zipinfo, and again you should see no JndiLookup.class file: 


zipinfo log4j-core-2.12.1.jar | grep Jndi 

Method 3 – try out a cookbook and InSpec profile to identify and remediate Elastic Search 

Our own Richard Nixon developed a Chef-native way to do all this without loading extra tools – both an InSpec profile and a cookbook to remediate any places where this Log4J class is found.  In the InSpec profile shown below, we simply check for JAR files and inspect the classes to see if we find JndiLookup.class: 

This style of search has broader applicability to finding errant Java class files in other applications and uses the power of the Chef domain-specific language embedded in the cookbooks.   

Mr. Nixon also accompanies this with a Chef patch cookbook, able to remediate and remove the offending file.  Simply pull up Chef Workstation, download the cookbook (or copy it over if you do not have a direct connection to the Internet or Chef Supermarket), and upload it for the Chef clients on each of those Elastic Search machines to pick up. 

cd <your cookbooks directory> 
git clone https://github.com/trickyearlobe/patch-log4j 
knife cookbook upload patch-log4j 

The cookbook is straightforward and shown below:

 

Next updates

We have a couple of updates on deck for Chef Automate and other Chef offerings which will bring our PostgreSQL up to version 13.4 and provide both an upgrade for Elastic Search and an option to convert over to Open Search.  These updates will feature upgrades to Elastic Search versions 6.8.23 and 7.16.1 as well as to Open Search version 1.2, all of which contain the latest Log4j 2.17.1 and are fully remediated. 

Finally, please reach out to your customer service team for complex situations or for any help on these steps. 

References: 

  1. Elastic Search Post- https://discuss.elastic.co/t/apache-log4j2-remote-code-execution-rce-vulnerability-cve-2021-44228-esa-2021-31/291476  
  2. Elastic Search new versions - https://www.elastic.co/blog/new-elasticsearch-and-logstash-releases-upgrade-apache-log4j2  
  3. Log4J vulnerability page - https://logging.apache.org/log4j/2.x/security.html  
  4. CVE-2021-44228 - https://nvd.nist.gov/vuln/detail/CVE-2021-44228  
  5. CVE-2021-45046 - https://nvd.nist.gov/vuln/detail/CVE-2021-45046  
  6. CVE-2021-44832 - https://nvd.nist.gov/vuln/detail/CVE-2021-44832  
  7. CVE-2021-45105 (denial-of-service) - https://nvd.nist.gov/vuln/detail/CVE-2021-45105  
  8. Associated Elastic Search update for 45105 https://www.elastic.co/guide/en/elasticsearch/reference/6.8/release-notes-6.8.23.html  
  9. A developer-focused post on ES specifically - https://xeraa.net/blog/2021_mitigate-log4j2-log4shell-elasticsearch/  
  10. SEI note on same - https://kb.cert.org/vuls/id/930724  
  11. Microsoft response - https://msrc-blog.microsoft.com/2021/12/11/microsofts-response-to-cve-2021-44228-apache-log4j2/  

 

 

 

 

Posted in:
Tags:
Chef Automate Chef Backend Chef Infra Server

Brian Loomis

Brian is Chef’s Director of Architecture and leads a small team defining the future of DevSecOps through creating Progress’ Chef platform. He has over 20 years of management and technology consulting with Microsoft – part of both the .NET and Azure launches – and leading his own agile practice defining platforms and digital products with select customers. Brian has led innovative thinking and launched over a dozen digital cloud products for organizations in the Fortune 50 through startups in manufacturing, healthcare, education, and high-tech. He has degrees from Princeton and Cal State, serves as an advisor to several software architecture professional groups, and was formerly an officer in the U. S. Air Force.

Categories

INTERNET US

Company
Using Chef
Legal
Connect with us
Contact Us

Chef is part of the Progress product portfolio. Progress is the leading provider of application development and digital experience technologies.

Copyright © 2024 Progress Software Corporation and/or its subsidiaries or affiliates. All Rights Reserved.

Progress, Telerik, Ipswitch, Chef, Kemp, Flowmon, MarkLogic, Semaphore and certain product names used herein are trademarks or registered trademarks of Progress Software Corporation and/or one of its subsidiaries or affiliates in the U.S. and/or other countries. See Trademarks for appropriate markings.

Do Not Sell or Share My Personal Information

Powered by Progress Sitefinity