Elasticsearch 8.14.1 (including ELK stack.) #324787
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
github-actions
bot
added
6.topic: nixos
messemar
force-pushed
the
elasticsearch
branch
2 times, most recently
from
321cf49 to
b21791c
Compare
ofborg
bot
added
8.has: clean-up
8.has: package (new)
messemar
commented
Jul 8, 2024
| ]; | ||
| license = licenses.elastic20; | ||
| platforms = platforms.unix; | ||
| maintainers = with maintainers; [ apeschar basvandijk ]; |
There was a problem hiding this comment.
@apeschar @basvandijk Is it okay for you to be the maintainer for elastic 8 as well ?
…t check for server availability
…n to re use tests from elasticsearch7
|
This could be interesting now that 8.16+ will be available as AGPL |
|
great job @messemar! Since kibana now is under APGL 3.0 elastic/kibana#192025 perhaps you can bring it back as a part of this PR. It has been removed in #264358 EDIT: fixed link to PR where kibana was dropped |
|
@messemar for me tests one # [ 16.525575] elasticsearch[891]: Sep 18, 2024 1:58:46 AM sun.util.locale.provider.LocaleProviderAdapter <clinit>
one # [ 16.533498] elasticsearch[891]: WARNING: COMPAT locale provider will be removed in a future release
one # [ 17.225194] elasticsearch[1111]: [2024-09-18T01:58:47,159][WARN ][o.e.c.l.LogConfigurator ] [one] Some logging configurations have %marker but don't have %node_name. We will automatically add %node_name to the pattern to ease the migration for users who customize log4j2.properties but will stop this behavior in 7.0. You should manually replace `%node_name` with `[%node_name]%marker ` in these locations:
one # [ 17.228262] elasticsearch[1111]: /var/lib/elasticsearch/config/log4j2.properties
one # [ 18.202800] elasticsearch[1111]: [2024-09-18T01:58:48,130][WARN ][o.e.n.NativeAccess ] [one] Unable to load native provider. Native methods will be disabled.
one # [ 18.204104] elasticsearch[1111]: java.lang.UnsatisfiedLinkError: Could not find libsystemd in java.library.path: /var/lib/elasticsearch/lib/platform/linux-x64:/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib
one # [ 18.205680] elasticsearch[1111]: at org.elasticsearch.nativeaccess.jdk.JdkSystemdLibrary.<clinit>(JdkSystemdLibrary.java:43) ~[elasticsearch-native-8.14.1.jar:?]
one # [ 18.206874] elasticsearch[1111]: at org.elasticsearch.nativeaccess.lib.NativeLibraryProvider.getLibrary(NativeLibraryProvider.java:58) ~[elasticsearch-native-8.14.1.jar:?]
one # [ 18.209105] elasticsearch[1111]: at org.elasticsearch.nativeaccess.LinuxNativeAccess.<init>(LinuxNativeAccess.java:20) ~[elasticsearch-native-8.14.1.jar:?]
one # [ 18.212242] elasticsearch[1111]: at org.elasticsearch.nativeaccess.NativeAccessHolder.<clinit>(NativeAccessHolder.java:28) ~[elasticsearch-native-8.14.1.jar:?]
one # [ 18.213466] elasticsearch[1111]: at org.elasticsearch.nativeaccess.NativeAccess.instance(NativeAccess.java:22) ~[elasticsearch-native-8.14.1.jar:?]
one # [ 18.214708] elasticsearch[1111]: at org.elasticsearch.bootstrap.Elasticsearch.initializeNatives(Elasticsearch.java:285) ~[elasticsearch-8.14.1.jar:?]
one # [ 18.215934] elasticsearch[1111]: at org.elasticsearch.bootstrap.Elasticsearch.initPhase2(Elasticsearch.java:169) ~[elasticsearch-8.14.1.jar:?]
one # [ 18.217794] elasticsearch[1111]: at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:74) ~[elasticsearch-8.14.1.jar:?]
one # [ 18.223146] elasticsearch[1111]: [2024-09-18T01:58:48,156][WARN ][o.e.n.NativeAccess ] [one] Cannot check if running as root because native access is not available
one # [ 19.736861] elasticsearch[1111]: [2024-09-18T01:58:49,674][INFO ][o.a.l.i.v.PanamaVectorizationProvider] [one] Java vector incubator API enabled; uses preferredBitSize=256; FMA enabled
one # [ 22.580244] elasticsearch[1111]: [2024-09-18T01:58:52,514][INFO ][o.e.n.Node ] [one] version[8.14.1], pid[1111], build[tar/93a57a1a76f556d8aee6a90d1a95b06187501310/2024-06-10T23:35:17.114581191Z], OS[Linux/6.6.51/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/21.0.3/21.0.3+9-nixos]
one # [ 22.582204] elasticsearch[1111]: [2024-09-18T01:58:52,518][INFO ][o.e.n.Node ] [one] JVM home [/nix/store/z0hakcf7s1r4wfmwkk6z44wx9b7qfgp3-openjdk-headless-21.0.3+9/lib/openjdk], using bundled JDK [false]
one # [ 22.585388] elasticsearch[1111]: [2024-09-18T01:58:52,518][INFO ][o.e.n.Node ] [one] JVM arguments [-Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -Djava.security.manager=allow, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j2.formatMsgNoLookups=true, -Djava.locale.providers=SPI,COMPAT, --add-opens=java.base/java.io=org.elasticsearch.preallocate, --add-opens=org.apache.lucene.core/org.apache.lucene.store=org.elasticsearch.vec, --enable-native-access=org.elasticsearch.nativeaccess, -XX:ReplayDataFile=logs/replay_pid%p.log, -Djava.library.path=/var/lib/elasticsearch/lib/platform/linux-x64:/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib, -Djna.library.path=/var/lib/elasticsearch/lib/platform/linux-x64:/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib, -Des.distribution.type=tar, -XX:+UseG1GC, -Djava.io.tmpdir=/tmp/elasticsearch-15066551684426462688, --add-modules=jdk.incubator.vector, -XX:+HeapDumpOnOutOfMemoryError, -XX:+ExitOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/lib/elasticsearch/logs/gc.log:utctime,level,pid,tags:filecount=32,filesize=64m, -Xms1453m, -Xmx1453m, -XX:MaxDirectMemorySize=762314752, -XX:G1HeapRegionSize=4m, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=15, --module-path=/var/lib/elasticsearch/lib, --add-modules=jdk.net, --add-modules=ALL-MODULE-PATH, -Djdk.module.main=org.elasticsearch.server]
one # [ 22.597711] elasticsearch[1111]: [2024-09-18T01:58:52,518][INFO ][o.e.n.Node ] [one] Default Locale [en_US]
one # [ 26.286190] elasticsearch[1111]: [2024-09-18T01:58:56,214][WARN ][c.a.a.p.i.BasicProfileConfigFileLoader] [one] Unable to load config file null
one # [ 26.287275] elasticsearch[1111]: java.security.AccessControlException: access denied ("java.io.FilePermission" "/var/lib/elasticsearch/.aws/config" "read")
one # [ 26.289202] elasticsearch[1111]: at java.security.AccessControlContext.checkPermission(AccessControlContext.java:488) ~[?:?]
one # [ 26.290169] elasticsearch[1111]: at java.security.AccessController.checkPermission(AccessController.java:1071) ~[?:?]
one # [ 26.292061] elasticsearch[1111]: at java.lang.SecurityManager.checkPermission(SecurityManager.java:411) ~[?:?]
one # [ 26.292932] elasticsearch[1111]: at java.lang.SecurityManager.checkRead(SecurityManager.java:742) ~[?:?]
one # [ 26.293833] elasticsearch[1111]: at java.io.File.exists(File.java:831) ~[?:?]
one # [ 26.295497] elasticsearch[1111]: at com.amazonaws.profile.path.config.SharedConfigDefaultLocationProvider.getLocation(SharedConfigDefaultLocationProvider.java:36) ~[aws-java-sdk-core-1.12.270.jar:?]
one # [ 26.296885] elasticsearch[1111]: at com.amazonaws.profile.path.AwsProfileFileLocationProviderChain.getLocation(AwsProfileFileLocationProviderChain.java:41) ~[aws-java-sdk-core-1.12.270.jar:?]
one # [ 26.300358] elasticsearch[1111]: at com.amazonaws.auth.profile.internal.BasicProfileConfigFileLoader.getProfilesConfigFile(BasicProfileConfigFileLoader.java:69) ~[aws-java-sdk-core-1.12.270.jar:?]
one # [ 26.301967] elasticsearch[1111]: at com.amazonaws.auth.profile.internal.BasicProfileConfigFileLoader.getProfile(BasicProfileConfigFileLoader.java:55) ~[aws-java-sdk-core-1.12.270.jar:?]
one # [ 26.303596] elasticsearch[1111]: at com.amazonaws.retry.internal.RetryModeResolver.profile(RetryModeResolver.java:103) ~[aws-java-sdk-core-1.12.270.jar:?]
one # [ 26.306309] elasticsearch[1111]: at com.amazonaws.retry.internal.RetryModeResolver.resolveRetryMode(RetryModeResolver.java:89) ~[aws-java-sdk-core-1.12.270.jar:?]
one # [ 26.307570] elasticsearch[1111]: at com.amazonaws.retry.internal.RetryModeResolver.<init>(RetryModeResolver.java:55) ~[aws-java-sdk-core-1.12.270.jar:?]
one # [ 26.309249] elasticsearch[1111]: at com.amazonaws.retry.internal.RetryModeResolver.<init>(RetryModeResolver.java:48) ~[aws-java-sdk-core-1.12.270.jar:?]
one # [ 26.313180] elasticsearch[1111]: at com.amazonaws.retry.RetryPolicy.<clinit>(RetryPolicy.java:35) ~[aws-java-sdk-core-1.12.270.jar:?]
one # [ 26.314204] elasticsearch[1111]: at com.amazonaws.retry.PredefinedRetryPolicies.<clinit>(PredefinedRetryPolicies.java:32) ~[aws-java-sdk-core-1.12.270.jar:?]
one # [ 26.315329] elasticsearch[1111]: at com.amazonaws.ClientConfiguration.<clinit>(ClientConfiguration.java:89) ~[aws-java-sdk-core-1.12.270.jar:?]
one # [ 26.316346] elasticsearch[1111]: at java.lang.Class.forName0(Native Method) ~[?:?]
one # [ 26.317102] elasticsearch[1111]: at java.lang.Class.forName(Class.java:421) ~[?:?]
one # [ 26.317912] elasticsearch[1111]: at java.lang.Class.forName(Class.java:412) ~[?:?]
one # [ 26.319468] elasticsearch[1111]: at org.elasticsearch.repositories.s3.S3RepositoryPlugin.lambda$static$0(S3RepositoryPlugin.java:53) ~[repository-s3-8.14.1.jar:8.14.1]
one # [ 26.320653] elasticsearch[1111]: at java.security.AccessController.doPrivileged(AccessController.java:319) ~[?:?]
one # [ 26.321576] elasticsearch[1111]: at org.elasticsearch.repositories.s3.S3RepositoryPlugin.<clinit>(S3RepositoryPlugin.java:47) ~[repository-s3-8.14.1.jar:8.14.1]
one # [ 26.323263] elasticsearch[1111]: at jdk.internal.misc.Unsafe.ensureClassInitialized0(Native Method) ~[?:?]
one # [ 26.326240] elasticsearch[1111]: at jdk.internal.misc.Unsafe.ensureClassInitialized(Unsafe.java:1160) ~[?:?]
one # [ 26.327617] elasticsearch[1111]: at jdk.internal.reflect.MethodHandleAccessorFactory.ensureClassInitialized(MethodHandleAccessorFactory.java:300) ~[?:?]
one # [ 26.329181] elasticsearch[1111]: at jdk.internal.reflect.MethodHandleAccessorFactory.newConstructorAccessor(MethodHandleAccessorFactory.java:103) ~[?:?]
one # [ 26.332273] elasticsearch[1111]: at jdk.internal.reflect.ReflectionFactory.newConstructorAccessor(ReflectionFactory.java:200) ~[?:?]
one # [ 26.333382] elasticsearch[1111]: at java.lang.reflect.Constructor.acquireConstructorAccessor(Constructor.java:549) ~[?:?]
one # [ 26.334564] elasticsearch[1111]: at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499) ~[?:?]
one # [ 26.336261] elasticsearch[1111]: at java.lang.reflect.Constructor.newInstance(Constructor.java:486) ~[?:?]
one # [ 26.337572] elasticsearch[1111]: at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:681) ~[elasticsearch-8.14.1.jar:?]
one # [ 26.338554] elasticsearch[1111]: at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:531) ~[elasticsearch-8.14.1.jar:?]
one # [ 26.339630] elasticsearch[1111]: at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:295) ~[elasticsearch-8.14.1.jar:?]
one # [ 26.342082] elasticsearch[1111]: at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:165) ~[elasticsearch-8.14.1.jar:?]
one # [ 26.343247] elasticsearch[1111]: at org.elasticsearch.node.NodeServiceProvider.newPluginService(NodeServiceProvider.java:54) ~[elasticsearch-8.14.1.jar:?]
one # [ 26.344618] elasticsearch[1111]: at org.elasticsearch.node.NodeConstruction.createEnvironment(NodeConstruction.java:451) ~[elasticsearch-8.14.1.jar:?]
one # [ 26.346254] elasticsearch[1111]: at org.elasticsearch.node.NodeConstruction.prepareConstruction(NodeConstruction.java:252) ~[elasticsearch-8.14.1.jar:?]
one # [ 26.348318] elasticsearch[1111]: at org.elasticsearch.node.Node.<init>(Node.java:192) ~[elasticsearch-8.14.1.jar:?]
one # [ 26.350246] elasticsearch[1111]: at org.elasticsearch.bootstrap.Elasticsearch$2.<init>(Elasticsearch.java:240) ~[elasticsearch-8.14.1.jar:?]
one # [ 26.351474] elasticsearch[1111]: at org.elasticsearch.bootstrap.Elasticsearch.initPhase3(Elasticsearch.java:240) ~[elasticsearch-8.14.1.jar:?]
one # [ 26.354196] elasticsearch[1111]: at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:75) ~[elasticsearch-8.14.1.jar:?]I managed to find a solution here 534cbba From 534cbba6435a392c04e4ef9e03532aa39df6b739 Mon Sep 17 00:00:00 2001
From: Pavel Anpin <pavel@anpin.fyi>
Date: Tue, 17 Sep 2024 23:21:01 -0300
Subject: [PATCH] elasticsearch8: fix missing native dependencies
---
pkgs/by-name/el/elasticsearch8/package.nix | 7 +++++--
1 file changed, 5 insertions(+), 2 deletions(-)
diff --git a/pkgs/by-name/el/elasticsearch8/package.nix b/pkgs/by-name/el/elasticsearch8/package.nix
index a68f48423ae29c7..2ac146997e73992 100644
--- a/pkgs/by-name/el/elasticsearch8/package.nix
+++ b/pkgs/by-name/el/elasticsearch8/package.nix
@@ -9,6 +9,8 @@
, coreutils
, autoPatchelfHook
, zlib
+, systemdLibs
+, zstd
}:
with lib;
let
@@ -45,7 +47,7 @@ stdenv.mkDerivation rec {
nativeBuildInputs = [ makeWrapper ]
++ lib.optional (!stdenv.hostPlatform.isDarwin) autoPatchelfHook;
- buildInputs = [ jre_headless util-linux zlib ];
+ buildInputs = [ jre_headless util-linux zlib systemdLibs zstd ];
runtimeDependencies = [ zlib jre_headless ];
@@ -61,7 +63,8 @@ stdenv.mkDerivation rec {
wrapProgram $out/bin/elasticsearch \
--prefix PATH : "${makeBinPath [ util-linux coreutils gnugrep ]}" \
--set ES_JAVA_HOME "${jre_headless}" \
- --set SCRIPT_NAME "$out/bin/elasticsearch"
+ --set SCRIPT_NAME "$out/bin/elasticsearch" \
+ --prefix ES_JAVA_OPTS : '-Djava.library.path="${lib.makeLibraryPath buildInputs}:$out/lib"'
esbin=$out/bin/elasticsearch
|
wegank
added
the
2.status: merge conflict
|
What is the status and how can i contribute to speed up the PR? We need to bring back the ELK stack into nix. |
Unfortunately, i have no time to finish it at the moment. I am Happy, if you can help. |
What are the current todo's and what is the dod? |
There was a request to pull in kibana as well. |
You can either fork or even check, if a push to the branch is possible. If it is not possible, i will manage the rights accordingly Maybe I find some time to continue on it in the next two weeks as well. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of changes
This PR adds elasticsearch8 to the code base. It also updates the ELK stack to support elasticsearch8 by adding new packages as well.
The
services.elasticsearchNixOS module is adopted to work with elasticsearch 7 and 8. Also, the tests for elasticsearch are adopted to execute with both, elasticsearch 7 and 8.Things done
nix.conf? (See Nix manual)sandbox = relaxedsandbox = truenix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage./result/bin/)Add a 👍 reaction to pull requests you find important.