Notice

The repo is froked from https://github.com/aws-samples/kubernetes-log4j-cve-2021-44228-node-agent, the original one is only for AWS environment.

In this repo, we enhanced it for general enrionment.

Apache Log4j2 CVE-2021-44228 node agent

AWS has developed several jars that performs a JVM-level hot-patch which disables JNDI lookups from the Log4j2 library, mitigating Log4j2 CVE-2021-44228.

The Apache Log4j2 CVE-2021-44228 node agent is an open source project built by the Kubernetes team at AWS. It is designed to run as a DaemonSet and mitigate the impact of Log4j2 CVE-2021-44228, which affects applications running Apache Log4j2 versions < 2.15.0 when processing inputs from untrusted sources. Running this DeamonSet will patch JVMs running in containers as well as on the host.

What it does: A cron entry will be installed on every worker node that runs a process looking for running JVMs and injects an agent which mitigates the Log4J2 CVE. All JVMs, including those running in containers will be hot-patched in this way. Currently, the hot-patch process is configured to run every 30min with a 15min jitter. The effective window can range from 15 to 45min between runs.

Note: You can find additional information about the Log4j2 vulnerability and AWS response in the IBM xforce.

Installation instructions

1. Apply the manifest:

kubectl apply -f daemonset.yaml

2. Check the logs for one or more of your DaemonSet pods.

Spot check a single pod:

kubectl get pods -l job=node-patch-installer -n node-configuration-daemonset

kubectl logs <pod-name> -c node-patch-installer -n node-configuration-daemonset

Considerations

• This project is meant to act as a temporary, best effort mitigation until you can update the Log4j2 dependency in all of your Java based Kubernetes applications to at least Log4j version 2.15.0. Do not rely on this agent as a long-term mitigation. This tool may help you mitigate the risk when updating dependencies is not immediately possible.

• With the default runtime frequency of 30 mins, the agent is better suited for long-running containers.

• The RPM only works with the following Java distributions:

  • Credo
  • Corretto
  • OpenJDK

About CVE-2021-44228

Apache Log4j2 < 2.15.0 JNDI features used in configuration, log messages, and parameters do not protect against attacker controlled LDAP and other JNDI related endpoints. An attacker who can control log messages or log message parameters can execute arbitrary code loaded from LDAP servers when message lookup substitution is enabled. From Log4j2 versions < 2.15.0, this behavior has been disabled by default. Full details can be found in the CVE bulletin.

You can find additional information about the Log4j2 vulnerability in the IBM xforce.

How this works

It will performs a JVM-level hotpatch disabling JNDI lookups from the Log4j2 library, mitigating the Log4j2 issue for that applies to JVMs on the host as well as JVMs running in containers. This project packages up the script and jars as a Kubernetes DaemonSet.

When installed, a process will run on every worker node that looks for running JVMs and injects an agent into the JVM to mitigate the Log4j2 vulnerability. The agent attempts to patch the lookup() method of all loaded org.apache.logging.log4j.core.lookup.JndiLookup instances to unconditionally return the string Patched JndiLookup::lookup(). This is designed to address the CVE-2021-44228 remote code execution vulnerability in Log4j2 without restarting the Java process.

This process by default is configured to run every 30 mins, and will add a layer of protection in clusters where applications have yet to be patched with an updated Log4j2 dependency.

Test

We supplied 2 test cases and 3 senario for testing

1. demo-log4j2
   This is a Jave sample project with log4j2 as dependency. Run it with command:

   java -jar target/gs-maven-0.1.0.jar
   

   and then you can see:

   root@weaseled1:~/go/src/github.com/vincent-pli/kubernetes-log4j-cve-2021-44228-node-agent/demo-log4j2# java -jar target/gs-maven-0.1.0.jar
   00:49:12.410 [main] INFO  com.ibm.app.App - Hello World! :)
   

   After the daemonset installed, there will be some logs like that:

   root@weaseled1:~/go/src/github.com/vincent-pli/kubernetes-log4j-cve-2021-44228-node-agent/demo-log4j2# java -jar target/gs-maven-0.1.0.jar
   00:49:12.410 [main] INFO  com.ibm.app.App - Hello World! :)  
   Loading Java Agent version 1 (using ASM6).
   Patching class org.apache.logging.log4j.core.lookup.JndiLookup (jdk.internal.loader.ClassLoaders$AppClassLoader@55054057)
   Transforming org/apache/logging/log4j/core/lookup/JndiLookup (jdk.internal.loader.ClassLoaders$AppClassLoader@55054057)
   

2. demo-without-log4j2
   This is a very simple Java project, without log4j2. Run it with command:

   javac HelloWorld.java
   java HelloWorld
   

   The you can see:

   root@weaseled1:~/go/src/github.com/vincent-pli/kubernetes-log4j-cve-2021-44228-node-agent/test# java HelloWorld
   Hello world!
   

   After the daemonset installed, there will be some logs like that:

   root@weaseled1:~/go/src/github.com/vincent-pli/kubernetes-log4j-cve-2021-44228-node-agent/test# java HelloWorld
   Hello world!
   Loading Java Agent version 1 (using ASM6).
   Vulnerable classes were not found. This agent will continue to run and transform the vulnerable class if it is loaded. Note that if you have shaded or otherwise changed the package name for log4j classes, then this tool may not find them.
   

   2.1 In container You can startup a pod with Jave running to test.

   kubectl create -f deployment.yaml
   

   and see the result:

   root@weaseled1:~/go/src/github.com/vincent-pli/kubernetes-log4j-cve-2021-44228-node-agent/test# kubectl get po
    NAME                               READY   STATUS    RESTARTS       AGE
    java-deployment-6f64576c78-ngtgs   1/1     Running   3 (8m3s ago)   3h8m
   

3. The result of hot patch daemonset

    [log4j-hotpatch] Starting up now...
    [log4j-hotpatch] Found JVMs with pids [1532201 1591157]
    [log4j-hotpatch] Attempting to patch 1532201
    [log4j-hotpatch] Running in a container with container pid: 1
    [log4j-hotpatch] Found JVM for pid 1532201 at /usr/local/openjdk-11/bin/java
    [log4j-hotpatch] Found JVM running with effective UID of 0
    [log4j-hotpatch] JVM version is openjdk version "11.0.13" 2021-10-19
    [log4j-hotpatch] Identified JVM[openjdk] of (openjdk version "11.0.13" 2021-10-19) with major version 11
    [log4j-hotpatch] Using Java 11 hotpatch
    Successfully loaded the agent into JVM process 1
    Look at stdout of JVM process 1 for more information
    [log4j-hotpatch] Hotpatch application returned 0
    [log4j-hotpatch] Attempting to patch 1591157
    [log4j-hotpatch] Found JVM for pid 1591157 at /usr/lib/jvm/java-11-openjdk-amd64/bin/java
    [log4j-hotpatch] Found JVM running with effective UID of 0
    [log4j-hotpatch] JVM version is openjdk version "11.0.11" 2021-04-20
    [log4j-hotpatch] Identified JVM[openjdk] of (openjdk version "11.0.11" 2021-04-20) with major version 11
    [log4j-hotpatch] Using Java 11 hotpatch
    sudo: unable to resolve host node-patch-installer-zn5dq: Name or service not known
    Successfully loaded the agent into JVM process 1591157
    Look at stdout of JVM process 1591157 for more information
    [log4j-hotpatch] Hotpatch application returned 0
    Waiting for 30 seconds...
    Task Completed
   

Last but not least

Actually, the stuff who really take effective is the jar.

In these code, they leverage:

• Java Agent
  The guide is helpful: Guide
• ASM, a opensource project could used for modify existing classes, see the Guide

With the two things, it can patch the existing classes in the JVM and no need to restart it, the simple process should be:

1. Start the jar which has Java Agent related code with same JVM as the target Java process
2. The Java Agent attached to the target Java process and go through all classes loaded
3. Found the target class and modified it with ASM
4. Done