Protect Elasticsearch from Apache Log4j Vulnerability #

CVE Address

https://github.com/advisories/GHSA-jfh8-c2jp-5v3q

Vulnerability Description

Apache Log4j is a very popular open source logging toolkit used for the Java runtime environment. Many Java frameworks including Elasticsearch of the latest version, use this component. Therefore, the scope of impact is huge.

The latest vulnerability existing in the execution of Apache Log4j’s remote code was revealed recently. Attackers can construct malicious requests and utilize this vulnerability to execute arbitrary code on a target server. As a result, the server can be controlled by hackers, who can then conduct page tampering, data theft, mining, extortion, and other behaviors. Users who use this component are advised to immediately initiate emergency response for fixing.

Basically, if a log output by Log4j contains the keyword ${, the log is replaced as a variable and then the variable operation is executed. Attackers can maliciously construct log content to make Java processes to execute arbitrary commands, achieving the attack purpose.

Vulnerability Level: very urgent

The vulnerability is caused by the lookup function provided by Log4j2. This function allows developers to read configurations in the environment by using a number of protocols. However, the input is not strictly judged in the implementation, resulting in the vulnerability.

Impact Scope: Java products: Apache Log4j 2.x < 2.15.0-rc2

Attack Detection

You can check logs for jndi:ldap://, jndi:rmi, and other characters to find out possible attacks.

Handling Method #

If Elasticsearch does not support configuration modification, Jar package replacement of Log4j, or cluster restart, you can use INFINI Gateway to intercept requests, replace parameters, and even directly block requests. You can use INFINI Gateway to check parameters in requests sent to Elasticsearch and replace or reject content that contains the sensitive keyword ${. In this way, INFINI Gateway can prevent the execution of malicious attack commands during Log4j logging after attack-contained requests are sent to Elasticsearch, thereby preventing attacks.

Reference Configuration #

Download the latest 1.5.0-SNAPSHOT version: http://release.elasticsearch.cn/gateway/snapshot/

The context_filter filter of INFINI Gateway can be used to detect the keywords of the request context _ctx.request.to_string and filter out malicious traffic, thereby blocking attacks.

path.data: data
path.logs: log

entry:
  - name: es_entrypoint
    enabled: true
    router: default
    max_concurrency: 20000
    network:
      binding: 0.0.0.0:8000

router:
  - name: default
    default_flow: main_flow

flow:
  - name: main_flow
    filter:
      - context_filter:
          context: _ctx.request.to_string
          action: redirect_flow
          status: 403
          flow: log4j_matched_flow
          must_not: # any match will be filtered
            regex:
              - \$\{.*?\}
              - "%24%7B.*?%7D" #urlencode
            contain:
              - "jndi:"
              - "jndi:ldap:"
              - "jndi:rmi:"
              - "jndi%3A" #urlencode
              - "jndi%3Aldap%3A" #urlencode
              - "jndi%3Armi%3A" #urlencode
      - elasticsearch:
          elasticsearch: es-server
  - name: log4j_matched_flow
    filter:
      - echo:
          message: 'Apache Log4j 2, Boom!'

elasticsearch:
  - name: es-server
    enabled: true
    endpoints:
      - http://localhost:9200

Use urlencode to convert the test command ${java:os} into %24%7Bjava%3Aos%7D.

Request calling execution result when requests do not need to pass through the gateway:

~%  curl 'http://localhost:9200/index1/_search?q=%24%7Bjava%3Aos%7D'
{"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index","resource.type":"index_or_alias","resource.id":"index1","index_uuid":"_na_","index":"index1"}],"type":"index_not_found_exception","reason":"no such index","resource.type":"index_or_alias","resource.id":"index1","index_uuid":"_na_","index":"index1"},"status":404}%

Logs on Elasticsearch are as follows:

[2021-12-11T01:49:50,303][DEBUG][r.suppressed             ] path: /index1/_search, params: {q=Mac OS X 10.13.4 unknown, architecture: x86_64-64, index=index1}
org.elasticsearch.index.IndexNotFoundException: no such index
	at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver$WildcardExpressionResolver.infe(IndexNameExpressionResolver.java:678) ~[elasticsearch-5.6.15.jar:5.6.15]
	at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver$WildcardExpressionResolver.innerResolve(IndexNameExpressionResolver.java:632) ~[elasticsearch-5.6.15.jar:5.6.15]
	at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver$WildcardExpressionResolver.resolve(IndexNameExpressionResolver.java:580) ~[elasticsearch-5.6.15.jar:5.6.15]

The logs above show that q=${java:os} in query conditions is executed and is changed to q=Mac OS X 10.13.4 unknown, architecture: x86_64-64, index=index1.

Request calling execution result when requests need to pass through the gateway:

medcl@Medcl:~%  curl 'http://localhost:8000/index1/_search?q=%24%7Bjava%3Aos%7D'

Apache Log4j 2, Boom!%

The logs above show that requests are filtered out.

You can try other commands to check whether malicious requests are intercepted:

#{java:vm}
~%  curl 'http://localhost:9200/index/_search?q=%24%7Bjava%3Avm%7D'
[2021-12-11T02:36:04,764][DEBUG][r.suppressed             ] [Medcl-2.local] path: /index/_search, params: {q=OpenJDK 64-Bit Server VM (build 25.72-b15, mixed mode), index=index}

~%  curl 'http://localhost:8000/index/_search?q=%24%7Bjava%3Avm%7D'
Apache Log4j 2, Boom!%

#{jndi:rmi://localhost:1099/api}
~%  curl 'http://localhost:9200/index/_search?q=%24%7Bjndi%3Armi%3A%2F%2Flocalhost%3A1099%2Fapi%7D'
2021-12-11 03:35:06,493 elasticsearch[YOmFJsW][search][T#3] ERROR An exception occurred processing Appender console java.lang.SecurityException: attempt to add a Permission to a readonly Permissions object

~%  curl 'http://localhost:8000/index/_search?q=%24%7Bjndi%3Armi%3A%2F%2Flocalhost%3A1099%2Fapi%7D'
Apache Log4j 2, Boom!%

The benefits of using INFINI Gateway is that no change needs to be made to the Elasticsearch server, especially in large-scale cluster scenarios. The flexible INFINI Gateway can significantly reduce workload, improve efficiency, shorten the security processing time, and reduce enterprise risks.