0-day in log4j package #81620

aSapien · 2021-12-10T09:56:21Z

Hi Elastic,

A 0-day exploit in log4j package has been published and it looks like ElasticSearch could be affected by a vulnerable version:

elasticsearch/build-tools-internal/version.properties

Lines 16 to 18 in 68836bb

    
           # when updating log4j, please update also docs/java-api/index.asciidoc 
        
           log4j             = 2.11.1 
        
           slf4j             = 1.6.2

Vulnerability:
apache/logging-log4j2#608

Please look at it and advice on the best course of action to secure an elastic cluster and prevent compromise ASAP.

Thanks!

The text was updated successfully, but these errors were encountered:

ghost · 2021-12-10T10:26:16Z

Duplicate of #81618
Mitigation in there as well.

JerryGuos · 2021-12-10T11:09:55Z

We are currently using two versions, 5.5.2 and 7.7.1.
For log4j-2.10.x , I can add -Dlog4j2.formatMsgNoLookups=true to ES_JAVA_OPTS.
However, in version 5.5.2, the version is 2.8.2 and -Dlog4j2.formatMsgNoLookups=true is invalid.

jaywilliams · 2021-12-10T13:35:24Z

Added this to my /etc/elasticsearch/jvm.options file to set the flag.

# CVE-2021-44228
-Dlog4j2.formatMsgNoLookups=true

akx · 2021-12-10T17:39:45Z

Looks like #81622, #81623, #81624, #81625 are addressing this. :)

mehta-ankit · 2021-12-10T18:13:00Z

I see #81624 merged, does that mean this also release a new docker image here: https://www.docker.elastic.co/r/elasticsearch/elasticsearch
Or will that happen when a new release is cut for 7.16 (like 7.16.1) ? nvm i see the label 7.16.1 on the PR

jkowall · 2021-12-10T21:09:06Z

OpenSearch is patched, feel free to adopt :)

opensearch-project/OpenSearch#1698

p-ob · 2021-12-10T21:53:20Z

We are currently using two versions, 5.5.2 and 7.7.1.
For log4j-2.10.x , I can add -Dlog4j2.formatMsgNoLookups=true to ES_JAVA_OPTS.
However, in version 5.5.2, the version is 2.8.2 and -Dlog4j2.formatMsgNoLookups=true is invalid.

@JerryGuos We have some 5.6.x Elastic infrastructure still running, and have dropped the resolved jar file in and removed the 2.8.x version (after stopping the service). Started the service afterwards and run some tests against it. This appears functional (albeit less-than-ideal).

ymmv though.

enoch85 · 2021-12-11T17:33:39Z

cc @Ark74, what about your Docker?

tanguofu · 2021-12-12T02:58:52Z

I see #81624 merged, does that mean this also release a new docker image here: https://www.docker.elastic.co/r/elasticsearch/elasticsearch Or will that happen when a new release is cut for 7.16 (like 7.16.1) ? nvm i see the label 7.16.1 on the PR

is there any es v7.16.1 release plan ?

t0klian · 2021-12-12T10:12:24Z

Some sources state that simply adding "-Dlog4j2.formatMsgNoLookups=true" is not an acceptable mitigation, because there are too many variables that may make it weak or ineffective.

When can we expect update of the log4j dependency for Elasticsearch? What versions will receive that update?

aSapien · 2021-12-12T11:04:25Z

@t0klian can you please provide links to these sources?

t0klian · 2021-12-12T11:17:33Z

@aSapien I've just re-phrased it to "some".

aSapien · 2021-12-12T12:36:34Z

@t0klian please share which variables other than log4j-core version might make the flag weak or ineffective, in which circumstances and according to which sources. If there is a bypass to the mitigation strategy we all need to know about it asap.

xuxiaowei-com-cn · 2021-12-13T00:40:53Z

我使用的 elasticsearch 版本是 7.13.3，安装方式：yum，log4j相关 jar 包是：log4j-api-2.11.1.jar、log4j-core-2.11.1.jar，位置是：/usr/share/elasticsearch/lib。
升级步骤：

将 /usr/share/elasticsearch/lib 中的 log4j-api-2.11.1.jar、log4j-core-2.11.1.jar 文件备份到其他文件夹后删除 /usr/share/elasticsearch/lib 中的 log4j-api-2.11.1.jar、log4j-core-2.11.1.jar。
重启 elasticsearch，发现报错：

[root@log lib]# systemctl status elasticsearch.service && journalctl -xe
● elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Mon 2021-12-13 08:25:50 CST; 16s ago
     Docs: https://www.elastic.co
  Process: 25607 ExecStart=/usr/share/elasticsearch/bin/systemd-entrypoint -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=1/FAILURE)
 Main PID: 25607 (code=exited, status=1/FAILURE)

Dec 13 08:25:50 log systemd-entrypoint[25607]: Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/logging/log4j/status/StatusListener
Dec 13 08:25:50 log systemd-entrypoint[25607]:         at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:79)
Dec 13 08:25:50 log systemd-entrypoint[25607]: Caused by: java.lang.ClassNotFoundException: org.apache.logging.log4j.status.StatusListener
Dec 13 08:25:50 log systemd-entrypoint[25607]:         at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:636)
Dec 13 08:25:50 log systemd-entrypoint[25607]:         at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:182)
Dec 13 08:25:50 log systemd-entrypoint[25607]:         at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:519)
Dec 13 08:25:50 log systemd-entrypoint[25607]:         ... 1 more
Dec 13 08:25:50 log systemd[1]: elasticsearch.service: Main process exited, code=exited, status=1/FAILURE
Dec 13 08:25:50 log systemd[1]: elasticsearch.service: Failed with result 'exit-code'.
Dec 13 08:25:50 log systemd[1]: Failed to start Elasticsearch.

下载 log4j-api-2.16.0.jar、log4j-core-2.16.0.jar 并放到 /usr/share/elasticsearch/lib 文件夹中。
重启 elasticsearch 后正常。

sevenJustFine · 2021-12-13T01:55:26Z

你这两个jar包是哪里来的?版本变了没影响吗?不改其他的参数设置的情况下

xuxiaowei-com-cn · 2021-12-13T02:10:01Z

你这两个jar包是哪里来的?版本变了没影响吗?不改其他的参数设置的情况下

上面有网址呀

vCassius · 2021-12-13T05:43:39Z

jar报的下载已经不行了,官方撤回了?

sevenJustFine · 2021-12-13T06:07:05Z

jar报的下载已经不行了,官方撤回了?

https://logging.apache.org/log4j/2.x/download.html?spm=a2c4g.11174386.n2.3.37314c07isUvXE# 从这里下载

xuxiaowei-com-cn · 2021-12-13T06:27:03Z

apache官方把 2.15.1 删除了，发布了 2.16.0

vCassius · 2021-12-13T07:31:17Z

嗯嗯谢谢我也刚发现他发布了2.16 rc1

cuishuaigit · 2021-12-13T08:17:33Z

请问怎么验证这个问题被解决了呢？或者怎么复现

Ark74 · 2021-12-13T08:42:06Z

cc @Ark74, what about your Docker?

@enoch85 I'll review later today, thanks for the heads up.

gastonsaid-ey · 2021-12-13T12:12:56Z

I made docker pull for Elasticsearch 7.16.1 and I still having problems with Log4j2 according with Aquasec scan.
Any other have this problem too?

xuxiaowei-com-cn · 2021-12-13T15:31:46Z

Elasticsearch 版本 7.16.1 已发布

FuviTeam · 2021-12-13T17:01:21Z

I've just upgraded to 7.16.1 in Ubuntu, and it's still the log4j-api-2.11.1.jar in the share lib ?

jheiselman · 2021-12-13T17:03:03Z

I've just upgraded to 7.16.1 in Ubuntu, and it's still the log4j-api-2.11.1.jar in the share lib ?

The version of log4j2 wasn't bumped, but the out-of-box logging configuration was updated to take into account the recommended mitigation strategies.

https://www.elastic.co/guide/en/elasticsearch/reference/current/release-notes-7.16.1.html

rdnm · 2021-12-13T19:26:04Z

For the latest updates on this issue, please refer to https://discuss.elastic.co/t/apache-log4j2-remote-code-execution-rce-vulnerability-cve-2021-44228-esa-2021-31/291476

Elastic's security reporting guidelines are available at https://www.elastic.co/community/security. Per those guidelines, all reports of potential security issues or vulnerabilities should be sent via email to security@elastic.co. We are unable to discuss potential issues of this nature here. Please send your report to the email address above, where it can be appropriately handled.

xuxiaowei-com-cn · 2021-12-14T02:23:01Z

经过测试：

如果要升级 elasticsearch，需要将自己升级（调整）过的 jar 包还原。否则将无法重启。例如：
本人昨天已经将 elasticsearch 中的 log4j-api、log4j-core 升级至 2.16.0，今天将 elasticsearch 从 7.13.3 升级至 7.16.1 后，重启出现如下问题：

[2021-12-14T10:11:30,107][ERROR][o.e.b.Bootstrap          ] [log] Exception
java.lang.IllegalStateException: jar hell!
class: org.apache.logging.log4j.Logger
jar1: /usr/share/elasticsearch/lib/log4j-api-2.16.0.jar
jar2: /usr/share/elasticsearch/lib/log4j-api-2.11.1.jar
	at org.elasticsearch.jdk.JarHell.checkClass(JarHell.java:297) ~[elasticsearch-core-7.16.1.jar:7.16.1]
	at org.elasticsearch.jdk.JarHell.checkJarHell(JarHell.java:192) ~[elasticsearch-core-7.16.1.jar:7.16.1]
	at org.elasticsearch.jdk.JarHell.checkJarHell(JarHell.java:75) ~[elasticsearch-core-7.16.1.jar:7.16.1]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:219) ~[elasticsearch-7.16.1.jar:7.16.1]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:434) [elasticsearch-7.16.1.jar:7.16.1]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:166) [elasticsearch-7.16.1.jar:7.16.1]
	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:157) [elasticsearch-7.16.1.jar:7.16.1]
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:77) [elasticsearch-7.16.1.jar:7.16.1]
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:112) [elasticsearch-cli-7.16.1.jar:7.16.1]
	at org.elasticsearch.cli.Command.main(Command.java:77) [elasticsearch-cli-7.16.1.jar:7.16.1]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:122) [elasticsearch-7.16.1.jar:7.16.1]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:80) [elasticsearch-7.16.1.jar:7.16.1]
[2021-12-14T10:11:30,115][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [log] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: jar hell!
class: org.apache.logging.log4j.Logger
jar1: /usr/share/elasticsearch/lib/log4j-api-2.16.0.jar
jar2: /usr/share/elasticsearch/lib/log4j-api-2.11.1.jar
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.16.1.jar:7.16.1]
	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:157) ~[elasticsearch-7.16.1.jar:7.16.1]
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:77) ~[elasticsearch-7.16.1.jar:7.16.1]
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:112) ~[elasticsearch-cli-7.16.1.jar:7.16.1]
	at org.elasticsearch.cli.Command.main(Command.java:77) ~[elasticsearch-cli-7.16.1.jar:7.16.1]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:122) ~[elasticsearch-7.16.1.jar:7.16.1]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:80) ~[elasticsearch-7.16.1.jar:7.16.1]
Caused by: java.lang.IllegalStateException: jar hell!
class: org.apache.logging.log4j.Logger
jar1: /usr/share/elasticsearch/lib/log4j-api-2.16.0.jar
jar2: /usr/share/elasticsearch/lib/log4j-api-2.11.1.jar
	at org.elasticsearch.jdk.JarHell.checkClass(JarHell.java:297) ~[elasticsearch-core-7.16.1.jar:7.16.1]
	at org.elasticsearch.jdk.JarHell.checkJarHell(JarHell.java:192) ~[elasticsearch-core-7.16.1.jar:7.16.1]
	at org.elasticsearch.jdk.JarHell.checkJarHell(JarHell.java:75) ~[elasticsearch-core-7.16.1.jar:7.16.1]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:219) ~[elasticsearch-7.16.1.jar:7.16.1]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:434) ~[elasticsearch-7.16.1.jar:7.16.1]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:166) ~[elasticsearch-7.16.1.jar:7.16.1]
	... 6 more

删除自己添加的 /usr/share/elasticsearch/lib/log4j-api-2.16.0.jar 后，重启出现新错误：

[2021-12-14T10:14:44,477][ERROR][o.e.b.Bootstrap          ] [log] Exception
java.lang.IllegalStateException: jar hell!
class: org.apache.logging.log4j.core.appender.rewrite.MapRewritePolicy$Mode
jar1: /usr/share/elasticsearch/lib/elasticsearch-log4j-7.16.1.jar
jar2: /usr/share/elasticsearch/lib/log4j-core-2.16.0.jar
	at org.elasticsearch.jdk.JarHell.checkClass(JarHell.java:297) ~[elasticsearch-core-7.16.1.jar:7.16.1]
	at org.elasticsearch.jdk.JarHell.checkJarHell(JarHell.java:192) ~[elasticsearch-core-7.16.1.jar:7.16.1]
	at org.elasticsearch.jdk.JarHell.checkJarHell(JarHell.java:75) ~[elasticsearch-core-7.16.1.jar:7.16.1]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:219) ~[elasticsearch-7.16.1.jar:7.16.1]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:434) [elasticsearch-7.16.1.jar:7.16.1]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:166) [elasticsearch-7.16.1.jar:7.16.1]
	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:157) [elasticsearch-7.16.1.jar:7.16.1]
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:77) [elasticsearch-7.16.1.jar:7.16.1]
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:112) [elasticsearch-cli-7.16.1.jar:7.16.1]
	at org.elasticsearch.cli.Command.main(Command.java:77) [elasticsearch-cli-7.16.1.jar:7.16.1]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:122) [elasticsearch-7.16.1.jar:7.16.1]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:80) [elasticsearch-7.16.1.jar:7.16.1]
[2021-12-14T10:14:44,488][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [log] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: jar hell!
class: org.apache.logging.log4j.core.appender.rewrite.MapRewritePolicy$Mode
jar1: /usr/share/elasticsearch/lib/elasticsearch-log4j-7.16.1.jar
jar2: /usr/share/elasticsearch/lib/log4j-core-2.16.0.jar
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.16.1.jar:7.16.1]
	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:157) ~[elasticsearch-7.16.1.jar:7.16.1]
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:77) ~[elasticsearch-7.16.1.jar:7.16.1]
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:112) ~[elasticsearch-cli-7.16.1.jar:7.16.1]
	at org.elasticsearch.cli.Command.main(Command.java:77) ~[elasticsearch-cli-7.16.1.jar:7.16.1]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:122) ~[elasticsearch-7.16.1.jar:7.16.1]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:80) ~[elasticsearch-7.16.1.jar:7.16.1]
Caused by: java.lang.IllegalStateException: jar hell!
class: org.apache.logging.log4j.core.appender.rewrite.MapRewritePolicy$Mode
jar1: /usr/share/elasticsearch/lib/elasticsearch-log4j-7.16.1.jar
jar2: /usr/share/elasticsearch/lib/log4j-core-2.16.0.jar
	at org.elasticsearch.jdk.JarHell.checkClass(JarHell.java:297) ~[elasticsearch-core-7.16.1.jar:7.16.1]
	at org.elasticsearch.jdk.JarHell.checkJarHell(JarHell.java:192) ~[elasticsearch-core-7.16.1.jar:7.16.1]
	at org.elasticsearch.jdk.JarHell.checkJarHell(JarHell.java:75) ~[elasticsearch-core-7.16.1.jar:7.16.1]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:219) ~[elasticsearch-7.16.1.jar:7.16.1]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:434) ~[elasticsearch-7.16.1.jar:7.16.1]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:166) ~[elasticsearch-7.16.1.jar:7.16.1]
	... 6 more

删除自己添加的 /usr/share/elasticsearch/lib/log4j-core-2.16.0.jar 后重启正常。

通过对上述报错与 /usr/share/elasticsearch/lib/ 文件夹分析可知，elasticsearch-7.16.1 （CentOS 使用 yum 安装，并从 7.13.3 升级至 7.16.1）使用 log4j-api-2.11.1.jar，同楼上回复一致。并且 elasticsearch-7.16.1 使用了 elasticsearch-log4j-7.16.1.jar 替换了 log4j-core。

下图是阿里云对 elasticsearch-7.13.3 扫描的结果，仅提示了 log4j-core-2.11.1.jar 存在漏洞，并未说明 log4j-api-2.11.1.jar 是否有问题。

如果大家觉得 log4j-api-2.11.1.jar 可能存在风险，可使用 log4j-api-2.16.0.jar 将其替换（经过测试，暂未发现问题）。

另外说明：
在同时升级 elasticsearch、filebeat、kibana 至 7.16.1 后，需要将 kibana 在停止的状态下执行 curl -XDELETE http://localhost:9200/.kibana*，否则访问 kibana 后，页面仅显示 Kibana server is not ready yet

julichan · 2021-12-14T09:11:33Z

Hello, may you consider this bug incompletely resolved as 6.8.21 Docker version sill contain log4j-core-2.11.1.jar

MHenn1g · 2021-12-14T16:27:14Z

Hi @julichan indeed Log4j has not been removed in total.
They removed the vulnerable JndiLookup class from the Log4j package.

See this satement:
As of December 13, 2021, we have released Elasticsearch 6.8.21 and 7.16.1 which set the JVM option identified below and remove the vulnerable JndiLookup class from Log4j out of an abundance of caution.
(https://discuss.elastic.co/t/apache-log4j2-remote-code-execution-rce-vulnerability-cve-2021-44228-esa-2021-31/291476)

So if you looked inside the jar in 6.8.20 and earlier...
jar tvf /usr/share/elasticsearch/lib/log4j-core-2.11.1.jar | grep JndiLookup.class

This class would show up
2937 Sun Jul 22 20:45:20 CEST 2018 org/apache/logging/log4j/core/lookup/JndiLookup.class

But now, there should be nothing to find

julichan · 2021-12-14T18:31:19Z

Thanks for your quick reply

vishallalwani · 2021-12-14T18:33:48Z

As per the latest thread https://lists.apache.org/thread/83y7dx5xvn3h5290q1twn16tltolv88f, setting log4j.formatMsgNoLookups property to true does not mitigate the issue.

julichan · 2021-12-14T18:36:59Z

@vishallalwani , the variable log4j.formatMsgNoLookups alone does not mitigate it. But if the faulty class JNDILookup is removed from the jar, it does!

fishjam · 2021-12-15T02:02:45Z

I have try following method for my es 6.2.2:

find all log4j jar files in your es folder ( include plugins and other folders)

  $ find ./ -type f | grep log4j.*jar$
  ./lib/log4j-core-2.9.1.jar
  ./lib/log4j-1.2-api-2.9.1.jar
  ./lib/log4j-api-2.9.1.jar
  ./plugins/search-guard-6/log4j-slf4j-impl-2.9.1.jar

download log4j 2.16.0 jar files , and you can find all other files on apache

  wget https://repo.maven.apache.org/maven2/org/apache/logging/log4j/log4j-core/2.16.0/log4j-core-2.16.0.jar
  wget https://repo.maven.apache.org/maven2/org/apache/logging/log4j/log4j-api/2.16.0/log4j-api-2.16.0.jar
  wget https://repo.maven.apache.org/maven2/org/apache/logging/log4j/log4j-1.2-api/2.16.0/log4j-1.2-api-2.16.0.jar
  wget https://repo.maven.apache.org/maven2/org/apache/logging/log4j/log4j-slf4j-impl/2.16.0/log4j-slf4j-impl-2.16.0.jar

stop the es and backup your old log4j files

  mkdir ../es_bak_libs && find ./ -type f | grep log4j.*jar$ | xargs -i{} mv {} ../es_bak_libs/

mv the log4j 2.16.0 jar to the right folder, then restart es , and it works
lsof to confirm it, it use only log4j 2.16.0

Now I want to know:
- is this workable? can I fix the security issue by this method? how can I try to verify it?
- because I don't want to upgrade my ES version, and want to solve this problem fundamentally without introducing new problems
Consider ES_CLASSPATH="$ES_HOME/lib/*" in elasticsearch-env , and es load all jar from it, it's version independent if it can load & run the function successful

[fishjam bin]$fgrep ES_CLASSPATH elasticsearch-env
ES_CLASSPATH="$ES_HOME/lib/*"
"$JAVA" -cp "$ES_CLASSPATH" org.elasticsearch.tools.launchers.JavaVersionChecker

@kimchy seems you are es team, could you ask some body to check this solution? thank you very much.

sandeepk-veritas · 2021-12-15T08:48:14Z

I tried solution suggested by @fishjam above for 7.14.1, but it is not working, ES does not startup properly.

[2021-12-15T14:02:22,454][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [SK-FS-2K19] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "getClassLoader")
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75) ~[elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116) ~[elasticsearch-cli-7.14.1.jar:7.14.1]
at org.elasticsearch.cli.Command.main(Command.java:79) ~[elasticsearch-cli-7.14.1.jar:7.14.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81) ~[elasticsearch-7.14.1.jar:7.14.1]
Caused by: java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "getClassLoader")
at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472) ~[?:?]
at java.security.AccessController.checkPermission(AccessController.java:1036) ~[?:?]
at java.lang.SecurityManager.checkPermission(SecurityManager.java:408) ~[?:?]
at java.lang.ClassLoader.checkClassLoaderPermission(ClassLoader.java:2049) ~[?:?]
at java.lang.ClassLoader.getParent(ClassLoader.java:1799) ~[?:?]
at org.apache.logging.log4j.util.LoaderUtil.getClassLoaders(LoaderUtil.java:113) ~[log4j-api-2.16.0.jar:2.16.0]
at org.apache.logging.log4j.core.util.WatchManager.getEventServices(WatchManager.java:160) ~[log4j-core-2.16.0.jar:2.16.0]
at org.apache.logging.log4j.core.util.WatchManager.(WatchManager.java:137) ~[log4j-core-2.16.0.jar:2.16.0]
at org.apache.logging.log4j.core.config.AbstractConfiguration.(AbstractConfiguration.java:137) ~[log4j-core-2.16.0.jar:2.16.0]
at org.apache.logging.log4j.core.config.DefaultConfiguration.(DefaultConfiguration.java:46) ~[log4j-core-2.16.0.jar:2.16.0]
at org.apache.logging.log4j.core.layout.PatternLayout$Builder.build(PatternLayout.java:768) ~[log4j-core-2.16.0.jar:2.16.0]
at org.elasticsearch.xpack.deprecation.logging.EcsJsonLayout.(EcsJsonLayout.java:47) ~[?:?]
at org.elasticsearch.xpack.deprecation.logging.EcsJsonLayout.createLayout(EcsJsonLayout.java:124) ~[?:?]
at org.elasticsearch.xpack.deprecation.logging.EcsJsonLayout$Builder.build(EcsJsonLayout.java:150) ~[?:?]
at org.elasticsearch.xpack.deprecation.logging.DeprecationIndexingComponent.(DeprecationIndexingComponent.java:70) ~[?:?]
at org.elasticsearch.xpack.deprecation.Deprecation.createComponents(Deprecation.java:83) ~[?:?]
at org.elasticsearch.node.Node.lambda$new$18(Node.java:615) ~[elasticsearch-7.14.1.jar:7.14.1]
at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:273) ~[?:?]
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625) ~[?:?]
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) ~[?:?]
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) ~[?:?]
at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913) ~[?:?]
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682) ~[?:?]
at org.elasticsearch.node.Node.(Node.java:619) ~[elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.node.Node.(Node.java:281) ~[elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:219) ~[elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:219) ~[elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:399) ~[elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.14.1.jar:7.14.1]
... 6 more

sandeepk-veritas · 2021-12-15T09:39:54Z

To add to my above comment, the same exact steps do work with 7.12.0 it seems. Unfortunately that is not an option for us because our application bundles 7.14.1 only and it is already out in market.
Seems like some issue with transient dependencies.

fishjam · 2021-12-15T10:25:33Z

reproduce the exception as sandeepk-veritas for 7.14.1.
After compare log4j-core 2.11 and 2.16, the WatchManager.getEventServices function is new added function, and maybe it's forbidden by xpack.
And my solution is not common solution for all ES version, so maybe you have to use the temporary measures and wait for official solution.

sandeepk-veritas · 2021-12-15T10:29:15Z

@fishjam thanks a lot for quick response. Yes, we will wait for official solution.

BillLucky · 2021-12-15T13:29:51Z

I have try following method for my es 6.2.2:

find all log4j jar files in your es folder ( include plugins and other folders)
  $ find ./ -type f | grep log4j.*jar$
  ./lib/log4j-core-2.9.1.jar
  ./lib/log4j-1.2-api-2.9.1.jar
  ./lib/log4j-api-2.9.1.jar
  ./plugins/search-guard-6/log4j-slf4j-impl-2.9.1.jar
download log4j 2.16.0 jar files , and you can find all other files on apache
  wget https://repo.maven.apache.org/maven2/org/apache/logging/log4j/log4j-core/2.16.0/log4j-core-2.16.0.jar
  wget https://repo.maven.apache.org/maven2/org/apache/logging/log4j/log4j-api/2.16.0/log4j-api-2.16.0.jar
  wget https://repo.maven.apache.org/maven2/org/apache/logging/log4j/log4j-1.2-api/2.16.0/log4j-1.2-api-2.16.0.jar
  wget https://repo.maven.apache.org/maven2/org/apache/logging/log4j/log4j-slf4j-impl/2.16.0/log4j-slf4j-impl-2.16.0.jar
stop the es and backup your old log4j files
  mkdir ../es_bak_libs && find ./ -type f | grep log4j.*jar$ | xargs -i{} mv {} ../es_bak_libs/
mv the log4j 2.16.0 jar to the right folder, then restart es , and it works

lsof to confirm it, it use only log4j 2.16.0

Now I want to know:

is this workable? can I fix the security issue by this method? how can I try to verify it?

because I don't want to upgrade my ES version, and want to solve this problem fundamentally without introducing new problems

Consider ES_CLASSPATH="$ES_HOME/lib/*" in elasticsearch-env , and es load all jar from it, it's version independent if it can load & run the function successful
[fishjam bin]$fgrep ES_CLASSPATH elasticsearch-env
ES_CLASSPATH="$ES_HOME/lib/*"
"$JAVA" -cp "$ES_CLASSPATH" org.elasticsearch.tools.launchers.JavaVersionChecker
@kimchy seems you are es team, could you ask some body to check this solution? thank you very much.

We also plan to use this solution to fix the CVE. Is it an official recommendation?

xeraa · 2021-12-16T00:14:11Z

There is an advisory for fixes on 5.0.0-5.6.10 and 6.0.0-6.3.2: https://discuss.elastic.co/t/elasticsearch-5-0-0-5-6-10-and-6-0-0-6-3-2-log4j-cve-2021-44228-cve-2021-45046-remediation/292054

PS: Don't drop in the latest Log4j JAR — it's not that simple as you can see in #47298. But we're working on a bigger fix to move to the latest version.

fishjam · 2021-12-16T01:36:14Z

after analyze the org.apache.logging.log4j.core.util.WatchManager.getEventServices function, consider there is already try...catch for each ServiceLoader , and the AccessControlException is foreseeable error scenarios , So I modify the code as following , and build & replace the WatchManager.class in log4j-core-2.16.0.jar, then it works.

Steps:

1. add try catch for org.apache.logging.log4j.core.util.WatchManager.getEventServices and build the WatchManager.class

replace the WatchManager.class in es home path

   mkdir -p org/apache/logging/log4j/core/util
   cp WatchManager.class org/apache/logging/log4j/core/util/
   zip -u lib/log4j-core-2.16.0.jar org/apache/logging/log4j/core/util/WatchManager.class

1. start es (7.14.1) and check, it's successful , but I'm not sure is there any other issue, you need do more check for you case.
```
$ curl -X GET http://localhost:9200/_cat/health
1639617900 01:25:00 elasticsearch green 1 1 1 1 0 0 0 0 - 100.0%
```

log4j-core-2.16.0.jar.zip

In my opinion, the log4j security issue in #47298 should be fixed in log4j. I will try to create a PR for it.
but I'm not sure whethere is it acceptable and when they accept it.

sandeepk-veritas · 2021-12-16T07:07:53Z

With 7.14.1 and log4j2 2.16.0 jars in place, one option that I tried today is remove x-pack-deprecation and try to see if we ES is able to startup or not.
Found that it does work but not sure whether that is a possible solution to use. @xeraa any comments on it?

jianchen06 · 2021-12-16T22:25:35Z

There is an advisory for fixes on 5.0.0-5.6.10 and 6.0.0-6.3.2: https://discuss.elastic.co/t/elasticsearch-5-0-0-5-6-10-and-6-0-0-6-3-2-log4j-cve-2021-44228-cve-2021-45046-remediation/292054

PS: Don't drop in the latest Log4j JAR — it's not that simple as you can see in #47298. But we're working on a bigger fix to move to the latest version.

Hi @xeraa , I have a version 6.8.1 running on Linux. I don't see instructions in the official advisory on that version, other than upgarde ES. I've tried @fishjam 's method to replace the 2.11.1 jars with 2.16.0. Restart the elasticsearch service and it runs fine without any error. Should I go with that approach if I don't want to upgrade ES right now? Thanks.

fishjam · 2021-12-17T02:36:29Z

there is already LOG4J2-3236 for "java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "getClassLoader")" yesterday.

xeraa · 2021-12-20T02:58:24Z

To close the loop here: Elasticsearch 7.16.2 and 6.8.22 are out and they use Log4j 2.17.0. This is the version you really want :)

zdenkers · 2021-12-20T21:12:04Z

does anyone know when 7.16.2 will be published to docker hub?

xeraa · 2021-12-20T22:25:29Z

Looks like it has been merged (docker-library/elasticsearch@53dedd3), but I'm not sure what it will take now to appear.

In the meantime, you can always pull it from the Elastic repo: docker pull docker.elastic.co/elasticsearch/elasticsearch:7.16.2

aSapien added >bug needs:triage Requires assignment of a team area label labels Dec 10, 2021

bernd mentioned this issue Dec 10, 2021

Fixes for log4j CVE-2021-44228 Graylog2/graylog2-server#11786

Merged

rdnm closed this as completed Dec 13, 2021

zahirtezcan-bugs mentioned this issue Dec 14, 2021

log4j JNDI vulnerability fluent/fluent-logger-java#96

Closed

0-day in log4j package #81620

0-day in log4j package #81620

Comments

aSapien commented Dec 10, 2021

ghost commented Dec 10, 2021

JerryGuos commented Dec 10, 2021

jaywilliams commented Dec 10, 2021

akx commented Dec 10, 2021

mehta-ankit commented Dec 10, 2021 • edited Loading

jkowall commented Dec 10, 2021

p-ob commented Dec 10, 2021 • edited Loading

enoch85 commented Dec 11, 2021

tanguofu commented Dec 12, 2021

t0klian commented Dec 12, 2021 • edited Loading

aSapien commented Dec 12, 2021

t0klian commented Dec 12, 2021

aSapien commented Dec 12, 2021

xuxiaowei-com-cn commented Dec 13, 2021 • edited Loading

sevenJustFine commented Dec 13, 2021

xuxiaowei-com-cn commented Dec 13, 2021

vCassius commented Dec 13, 2021

sevenJustFine commented Dec 13, 2021

xuxiaowei-com-cn commented Dec 13, 2021 • edited Loading

vCassius commented Dec 13, 2021

cuishuaigit commented Dec 13, 2021 • edited Loading

Ark74 commented Dec 13, 2021

gastonsaid-ey commented Dec 13, 2021

xuxiaowei-com-cn commented Dec 13, 2021

FuviTeam commented Dec 13, 2021

jheiselman commented Dec 13, 2021

rdnm commented Dec 13, 2021

xuxiaowei-com-cn commented Dec 14, 2021 • edited Loading

julichan commented Dec 14, 2021

MHenn1g commented Dec 14, 2021

julichan commented Dec 14, 2021

vishallalwani commented Dec 14, 2021

julichan commented Dec 14, 2021

fishjam commented Dec 15, 2021 • edited Loading

sandeepk-veritas commented Dec 15, 2021

sandeepk-veritas commented Dec 15, 2021

fishjam commented Dec 15, 2021

sandeepk-veritas commented Dec 15, 2021

BillLucky commented Dec 15, 2021

xeraa commented Dec 16, 2021

fishjam commented Dec 16, 2021

sandeepk-veritas commented Dec 16, 2021

jianchen06 commented Dec 16, 2021 • edited Loading

fishjam commented Dec 17, 2021

xeraa commented Dec 20, 2021

zdenkers commented Dec 20, 2021

xeraa commented Dec 20, 2021

mehta-ankit commented Dec 10, 2021 •

edited

Loading

p-ob commented Dec 10, 2021 •

edited

Loading

t0klian commented Dec 12, 2021 •

edited

Loading

xuxiaowei-com-cn commented Dec 13, 2021 •

edited

Loading

xuxiaowei-com-cn commented Dec 13, 2021 •

edited

Loading

cuishuaigit commented Dec 13, 2021 •

edited

Loading

xuxiaowei-com-cn commented Dec 14, 2021 •

edited

Loading

fishjam commented Dec 15, 2021 •

edited

Loading

jianchen06 commented Dec 16, 2021 •

edited

Loading