Kibana plugin install stalls at "Optimizing and caching browser bundles..."

[bradvido]

Kibana version:
6.2.4
Elasticsearch version:
6.2.4
Server OS version:
Windows Server 2016

Original install method (e.g. download page, yum, from source, etc.):
zip download from elastic.co

Describe the bug:
I executed the install command here about 16 hours ago, and it still hasn't gotten past this step. As you can see node is using ~8 GB of memory which seems excessive.

Note that per recommendations in other issues, I did set the --max-old-space-size to 8000

Steps to reproduce:

1. Install ES/Kibana on Server 2016
2. Execute command seen in screenshot (happens for any plugin install, not just the one pictured)
3. Wait

Expected behavior:
Plugin install should finish quickly

[tylersmalley]

Related: https://discuss.elastic.co/t/install-problem-kibana-plugin-appears-to-hang/134355/3

[tylersmalley]

Related: #19633

[bradvido]

Still an issue for me in 6.3.0

[akhilvatts]

+1 in 6.4.0

[derentis]

+1 in 6.3.1 on Ubuntu 18.0.4.1 LTS

just sits there with processes open but never actually goes anywhere. That's been running for 24hrs now and it isn't doing anything.

[zukeru]

+1 here I have the same issue with sentinl plugin. It will get caught in the optimizing part. If I kill it and then run it again it says its installed and configured. This doesn't happen with searchguard however.

[Mrfly-ca]

I am having the same issue running the install for wazuhapp-3.6.1_6.4.2.zip

Everything else is the same. (Server, OS, mem etc)

[n4ll3ec]

+1 , same issue occured on ELK Stack6.4.1,Ubuntu16.04 LTS

[akhilvatts]

+1 ELK 6.5.0

[memelet]

I get a hang with wazuh 3.7.1 on kibana 6.5.1

[kellervater]

Same here with 6.5.1 and the logtrail plugin of the same version.

[q2dg]

What a mess...The same with Kibana 6.5.4 trying to install Sentinl plugin...

[anottrott]

Same issue with Logtrail 6.5.1 on Ubuntu 18.04, CPU and memory are maxed out. It seems like this is a common issue with the kibana build when installing plugins

After some time the installation crashed with the following error message... Seems like NodeJS running out of memory?

Attempting to transfer from https://github.com/sivasamyk/logtrail/releases/download/v0.1.30/logtrail-6.5.2-0.1.30.zip
Transferring 2096312 bytes....................
Transfer complete
Retrieving metadata from plugin archive
Extracting plugin archive
Extraction complete
Optimizing and caching browser bundles...

Plugin installation was unsuccessful due to error "Command failed: /elasticsearch/kibana-6.5.2-linux-x86_64/node/bin/node /elasticsearch/kibana-6.5.2-linux-x86_64/src/cli --env.name=production --optimize.useBundleCache=false --server.autoListen=false --plugins.initialize=false
(node:658) [DEP0022] DeprecationWarning: os.tmpDir() is deprecated. Use os.tmpdir() instead.
FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
1: node::Abort() [/elasticsearch/kibana-6.5.2-linux-x86_64/node/bin/node]
2: 0x8cce9c [/elasticsearch/kibana-6.5.2-linux-x86_64/node/bin/node]
3: v8::Utils::ReportOOMFailure(char const*, bool) [/elasticsearch/kibana-6.5.2-linux-x86_64/node/bin/node]
4: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [/elasticsearch/kibana-6.5.2-linux-x86_64/node/bin/node]
5: v8::internal::Factory::NewRawTwoByteString(int, v8::internal::PretenureFlag) [/elasticsearch/kibana-6.5.2-linux-x86_64/node/bin/node]
6: v8::internal::String::SlowFlatten(v8::internal::Handlev8::internal::ConsString, v8::internal::PretenureFlag) [/elasticsearch/kibana-6.5.2-linux-x86_64/node/bin/node]
7: v8::internal::String::Flatten(v8::internal::Handlev8::internal::String, v8::internal::PretenureFlag) [/elasticsearch/kibana-6.5.2-linux-x86_64/node/bin/node]
8: v8::internal::String::LastIndexOf(v8::internal::Isolate*, v8::internal::Handlev8::internal::Object, v8::internal::Handlev8::internal::Object, v8::internal::Handlev8::internal::Object) [/elasticsearch/kibana-6.5.2-linux-x86_64/node/bin/node]
9: v8::internal::Builtin_StringPrototypeLastIndexOf(int, v8::internal::Object**, v8::internal::Isolate*) [/elasticsearch/kibana-6.5.2-linux-x86_64/node/bin/node]
10: 0x235c5098697d

{"type":"log","@timestamp":"2019-01-14T18:30:37Z","tags":["info","optimize"],"pid":658,"message":"Optimizing and caching bundles for ml, logtrail, stateSessionStorageRedirect, status_page, timelion, graph, monitoring, space_selector, login, logout, dashboardViewer, apm, canvas, infra and kibana. This may take a few minutes"}

<--- Last few GCs --->

[658:0x38a9170] 995416 ms: Mark-sweep 1310.6 (1457.1) -> 1310.5 (1458.6) MB, 1777.2 / 0.0 ms allocation failure GC in old space requested
[658:0x38a9170] 997310 ms: Mark-sweep 1310.5 (1458.6) -> 1310.5 (1427.6) MB, 1893.5 / 0.0 ms last resort GC in old space requested
[658:0x38a9170] 999434 ms: Mark-sweep 1310.5 (1427.6) -> 1310.5 (1427.6) MB, 2124.7 / 0.0 ms last resort GC in old space requested

<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x1c668eca58b9
0: builtin exit frame: lastIndexOf(this=0x3deac35b02c9 <Very long string[1106657]>,0x289b9bd120e9 <String[1]: \n>)

1: has_nlb(aka has_nlb) [0x13dff58022d1 <undefined>:5970] [bytecode=0x37933e0febd9 offset=15](this=0x13dff58022d1 <undefined>)
2: /* anonymous */(aka /* anonymous */) [0x13dff58022d1 <undefined>:6070] [bytecode=0x132e668035b1 offset=60](this=0x13dff58022d1 <undefined>,...

"

[CannibalKush]

Exact same problem with kibana 6.4.2 and searchguard 6.4.2-17, any fix on the horizon?

[minhle2994]

+1 in Kibana 6.5.4

[fabrei]

same with 6.5.1 and the plugin network_vis 5.5.0-1
there are already some issues #25849 and #21794 which are dealing with this topic. #21809 should resolve this, but it seems it doesn't.

my workaround is:

1. stop kibana (maybe elastic too)
2. install the plugin until Optimizing and caching browser bundles... is shown
3. stop installation
4. run rm -rf /usr/share/kibana/optimize/bundles
5. in /usr/share/kibana/bin/kibana add --max-old-space-size=3072 to $NODE_OPTIONS
6. start kibana normally
7. kibana starts the optimization and after 3 to 5 minutes kibana is up..

seems that the optimization at startup is quite different to the optimization after a plugin installation!?

[mistic]

We have made several improvements on this matter that will come along by 6.6.x and 6.7.x.
Meanwhile I believe the best workaround to deal with this situation would be:

1. Stop kibana.
2. Set the current directory to the Kibana installation dir. (For example cd /usr/share/kibana)
3. rm -rf optimize/bundles
4. NODE_OPTIONS="--max-old-space-size=4096" ./bin/kibana
5. Kibana should optimize and start normally in some minutes

[minhle2994]

Sometime installing process hang at Optimizing and caching browser bundles but plugin still install successfully

[mistic]

@minhle2994 can u try what I suggest on my previous comment please?

[Beginner77]

+1 in Kibana 6.7.1. It just stalls at the optimizing and caching browser bundle points. when checked inside the container, this command - " /usr/share/kibana/node/bin/node /usr/share/kibana/src/cli --env.name=production --optimize.useBundleCache=false --server.autoListen=false --plugins.initialize=false" was running for a very long time.Killed the process and tried to run this manually and it just gets stuck there do nothing before I eventually kill it.

[g3rhard]

+1 same thing in Kibana 6.7.1
Debian 9

NODE_OPTIONS="--max-old-space-size=8192" /usr/share/kibana/bin/kibana-plugin install https://github.com/bitsensor/elastalert-kibana-plugin/releases/download/1.0.3/elastalert-kibana-plugin-1.0.3-6.7.1.zip
Attempting to transfer from https://github.com/bitsensor/elastalert-kibana-plugin/releases/download/1.0.3/elastalert-kibana-plugin-1.0.3-6.7.1.zip
Transferring 22930973 bytes....................
Transfer complete
Retrieving metadata from plugin archive
Extracting plugin archive
Extraction complete
Optimizing and caching browser bundles...

In kibana.yml:

...
xpack.monitoring.enabled: false
xpack.graph.enabled: false
xpack.ml.enabled: false
xpack.security.enabled: false
xpack.watcher.enabled: false

[mistic]

@Beginner77 @g3rhard that is strange, can you try to run the steps I mention above on #19678 (comment)

[OpakAlex]

Same issue for kubernetes and kibana. After add some xpack options, i see only Optimizing and caching browser bundles, maybe it calculates moon orbita?)))))))))))) or mars?)

[g3rhard]

I checked user perms on folder and files in kibana directory and forget about problem. I think best way - to run install plugins with sudo (sudo -u kibana, as example), because folders and files has owner and group kibana (kibana:kibana). Perhaps I could be wrong.

[OpakAlex]

@g3rhard but plugins installed. Did you have this error or you fixed it? Or you just write a comment ?)))

[andsens]

Here you go, a Dockerfile that creates a kibana container with preoptimized bundles for you.

[Willsms]

+1 in Kibana 6.5.2, I installed the plugin successfully. but I failed to start Kibana.
@mistic your solution is effective for Kibana 6.5.2. Thank you.
If you are installing the plugin failed. stop here: Optimizing and caching browser bundles....
1.Clean up your plugin, bin/kibana-plugin remove your plugin
2.download your plugin.zip
3.export NODE_OPTIONS="--max-old-space-size=4096"
4. bin/kibana-plugin install file:///your plugin.zip
5.Wait a minute
If installed the plugin successfully. but failed to start kibana and stop here: Browserslist: caniuse-lite is outdated. Please run next command npm update caniuse-lite browserslist
run the steps @mistic mention above on #19678 (comment）and mem must > 6g.

[remmeier]

is there any progress on this, in the forum it was mentioned that the entire thing may get eliminated (which would be great):

tylersmalley, Elastic Team Member, Jun '18

Unfortunately, the optimizing process is very CPU intensive and I have seen 
it take up to 10 minutes to complete. How long have you let it run for? We 
hope to remove the need for this process entirely in the coming months.

a year passed since that post.

[bradvido]

I've had it running for 8 days now, and has made no progress, so it's obviously not just due to "CPU Intense operations":

[Willsms]

@bradvido You need to check if your memory is large enough. If it is less than a certain threshold, the installation is not feasible. I tried, if the available memory is less than 5g, the installation failed. I recommend it is greater than 6g or more. My current available memory is 10g, probably waiting for 4-6 minutes.
This is just a temporary solution.
If the plugin supports it, you can try version 6.7.2, which is relatively stable.

[bradvido]

That doesn’t seem to be the issue because the box has 32 gb and the elasticseearch service was shut down during the attempted install so at least 25 were available. 🤷🏻‍♂️

…

________________________________ From: Willsms <notifications@github.com> Sent: Thursday, July 18, 2019 8:16:09 PM To: elastic/kibana Cc: Brady Vidovic; Mention Subject: Re: [elastic/kibana] Kibana plugin install stalls at "Optimizing and caching browser bundles..." (#19678) @bradvido<https://github.com/bradvido> You need to check if your memory is large enough. If it is less than a certain threshold, the installation is not feasible. I tried, if the available memory is less than 5g, the installation failed. I recommend it is greater than 6g or more. My current available memory is 10g, probably waiting for 4-6 minutes. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#19678?email_source=notifications&email_token=AAGX7A7UVQV6QDEHYGPFK4DQAEIVTA5CNFSM4FDMI7MKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD2KIVYI#issuecomment-513051361>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AAGX7A5DZ2T7OGGIBJRMJM3QAEIVTANCNFSM4FDMI7MA>.

[rca0]

This command line works for me

NODE_OPTIONS="--max-old-space-size=4096" /usr/share/kibana/bin/kibana-plugin install https://github.com/bitsensor/elastalert-kibana-plugin/releases/download/1.0.1/elastalert-kibana-plugin-1.0.1-6.5.3.zip

[Sagesh]

I am using Kibana 6.5, I have installed it as a docker service. None of the above solutions are working for me.

Anyone has other options?

Thanks,
Sagesh

[BenderRodrigez]

Same issue for me in k8s.
Set:

- name: XPACK_SECURITY_ENABLED
    value: "false"

And got this problem.
Elasticsearch works just fine with the same options. Both, Kibana and Elasticsearch have version 6.2.4.

[fabrei]

We have made several improvements on this matter that will come along by 6.6.x and 6.7.x.
Meanwhile I believe the best workaround to deal with this situation would be:

1. Stop kibana.

2. Set the current directory to the Kibana installation dir. (For example `cd /usr/share/kibana`)

3. `rm -rf optimize/bundles`

4. `NODE_OPTIONS="--max-old-space-size=4096" ./bin/kibana`

5. Kibana should optimize and start normally in some minutes

After an update to v.7.4.2, it works in an acceptable time. I also have to remove the bundle files, otherwise the changes of the plugin do not have an effect. But it take not one minute and kibana is alive and can be used :) Before it was so much more time for me..

[tylersmalley]

Closing as there is no longer an optimization step in production.