Kibana Optimizing and caching bundles for 28 hours #41726

hackdefendr · 2019-07-23T00:16:01Z

Kibana version: 7.2.0

Elasticsearch version: 7.2.0

Server OS version: Debian Buster (10)

Browser version: Google Chrome Version 75.0.3770.142

Original install method (e.g. download page, yum, from source, etc.): APT

Describe the bug:
Kibana optimization is taking a long time to complete. I decided to start it from the command line with the --optimize flag to see what happens. Nothing happens, seriously for more than 24hrs. This all started when I tried to install the Sentinl plugin, which is now removed because of obvious reasons. Now Kibana is optimizing the core plugins for the past 28 hrs.

STDOUT LOG

Steps to reproduce:

Install Elastic, Kibana, Logstash normally.
Install any compatible Kibana plugin
Start Kibana

Expected behavior:
If optimization is needed, I highly doubt every plugin requires it. I suppose that I expect that optimization would not take 28+ hours to complete.

Any additional context:
I'd like to see more documentation around this mysterious and seemingly required optimization function. I like to know what is going on during this optimization process.

legrego · 2019-07-23T15:13:36Z

@hackdefendr,

The optimization process is responsible for generating JS bundles for all of the installed plugins. Behind the scenes, it's using webpack to perform this work.

28 hours is a really long time...as the message suggests, it should only take a few minutes. How much memory (RAM) is available on your system? Are you able to increase the amount of available RAM and retry?

elasticmachine · 2019-07-23T15:13:53Z

Pinging @elastic/kibana-operations

hackdefendr · 2019-07-23T23:28:17Z

@legrego Thanks for replying.

I have 32Gb in the system, I think I can spare some :)

I will test and retry.

hackdefendr · 2019-07-23T23:38:51Z

OK assuming NODE_OPTIONS is still the proper way to increase allocated memory, this is what I did:

NODE_OPTIONS="--max_old_space_size=8192"

Yeah I just gave it 8Gb and so far I am seeing the same results. Its only been a few minutes thus far, but it feels the same.

EDIT (2 hrs later)

It is still optimizing. I'm thinking that its time to remove and reinstall, but want to troubleshoot if there is anything left to try.

legrego · 2019-07-25T11:22:10Z

@mistic do you have any suggestions here?

mistic · 2019-07-25T12:43:45Z

@legrego @hackdefendr could you try to delete the optimize directory and restart the optimization process?

hackdefendr · 2019-07-25T17:36:03Z

Yep I will try that this evening as this is in my lab at home.

hackdefendr · 2019-07-27T14:49:10Z

Sorry for the delay. OK deleting the optimize directory and then running the same command seems to have worked. I am back to where I was before I started messing with things. Optimization was quick and then Kibana started right up.

hackdefendr · 2019-07-27T22:56:52Z

I was wrong. Only this is being created in the optimize folder:

~# ll /usr/share/kibana/optimize/bundles/
total 80
drwxr-xr-x 1 kibana kibana 512 Jul 27 17:47 .
drwxrwxr-x 1 kibana kibana 512 Jul 27 17:18 ..
-rw-r--r-- 1 kibana kibana 2411 Jul 27 17:47 apm.entry.js
-rw-r--r-- 1 kibana kibana 0 Jul 27 17:47 apm.style.css
-rw-r--r-- 1 kibana kibana 2412 Jul 27 17:47 canvas.entry.js
-rw-r--r-- 1 kibana kibana 0 Jul 27 17:47 canvas.style.css
-rw-r--r-- 1 kibana kibana 2410 Jul 27 17:47 code.entry.js
-rw-r--r-- 1 kibana kibana 0 Jul 27 17:47 code.style.css
-rw-r--r-- 1 kibana kibana 2433 Jul 27 17:47 dashboardViewer.entry.js
-rw-r--r-- 1 kibana kibana 0 Jul 27 17:47 dashboardViewer.style.css
-rw-r--r-- 1 kibana kibana 2411 Jul 27 17:47 graph.entry.js
-rw-r--r-- 1 kibana kibana 0 Jul 27 17:47 graph.style.css
-rw-r--r-- 1 kibana kibana 2411 Jul 27 17:47 infra.entry.js
-rw-r--r-- 1 kibana kibana 0 Jul 27 17:47 infra.style.css
-rw-r--r-- 1 kibana kibana 2415 Jul 27 17:47 kibana.entry.js
-rw-r--r-- 1 kibana kibana 0 Jul 27 17:47 kibana.style.css
-rw-r--r-- 1 kibana kibana 2427 Jul 27 17:47 logged_out.entry.js
-rw-r--r-- 1 kibana kibana 0 Jul 27 17:47 logged_out.style.css
-rw-r--r-- 1 kibana kibana 2422 Jul 27 17:47 login.entry.js
-rw-r--r-- 1 kibana kibana 0 Jul 27 17:47 login.style.css
-rw-r--r-- 1 kibana kibana 2423 Jul 27 17:47 logout.entry.js
-rw-r--r-- 1 kibana kibana 0 Jul 27 17:47 logout.style.css
-rw-r--r-- 1 kibana kibana 2412 Jul 27 17:47 maps.entry.js
-rw-r--r-- 1 kibana kibana 0 Jul 27 17:47 maps.style.css
-rw-r--r-- 1 kibana kibana 2408 Jul 27 17:47 ml.entry.js
-rw-r--r-- 1 kibana kibana 0 Jul 27 17:47 ml.style.css
-rw-r--r-- 1 kibana kibana 2423 Jul 27 17:47 monitoring.entry.js
-rw-r--r-- 1 kibana kibana 0 Jul 27 17:47 monitoring.style.css
-rw-r--r-- 1 kibana kibana 2436 Jul 27 17:47 overwritten_session.entry.js
-rw-r--r-- 1 kibana kibana 0 Jul 27 17:47 overwritten_session.style.css
-rw-r--r-- 1 kibana kibana 2410 Jul 27 17:47 siem.entry.js
-rw-r--r-- 1 kibana kibana 0 Jul 27 17:47 siem.style.css
-rw-r--r-- 1 kibana kibana 2429 Jul 27 17:47 space_selector.entry.js
-rw-r--r-- 1 kibana kibana 0 Jul 27 17:47 space_selector.style.css
-rw-r--r-- 1 kibana kibana 2432 Jul 27 17:47 stateSessionStorageRedirect.entry.js
-rw-r--r-- 1 kibana kibana 0 Jul 27 17:47 stateSessionStorageRedirect.style.css
-rw-r--r-- 1 kibana kibana 2425 Jul 27 17:47 status_page.entry.js
-rw-r--r-- 1 kibana kibana 0 Jul 27 17:47 status_page.style.css
-rw-r--r-- 1 kibana kibana 2414 Jul 27 17:47 timelion.entry.js
-rw-r--r-- 1 kibana kibana 0 Jul 27 17:47 timelion.style.css
-rw-r--r-- 1 kibana kibana 2412 Jul 27 17:47 uptime.entry.js
-rw-r--r-- 1 kibana kibana 0 Jul 27 17:47 uptime.style.css

No bundles.

Does the process of creating bundles depend on the file system type? Just curious, I mean it shouldn't matter that this is Debian Buster under Windows Subsystem for Linux. I mean Linux is Linux right?

If I uninstall completely, including purging all configuration files, then reinstall...Kibana starts up without issue and continues to work without issue until I try to add a plugin. I've tested apps and visualizations and got the same results.

I don't know, I guess I could test an actual Debian Buster install on a legit EXT4 file system and see if I can install a plugin under Kibana.

hackdefendr · 2019-07-28T22:18:26Z

Reproduced the same behavior in Debian 10 (Buster) on actual hardware and in VM.

Note the babelcache file is currently 44 MB, which I find odd.

-rw-rw-r-- 1 kibana kibana 44M Jul 28 18:36 .babelcache.json

Oh and it seems to loop. Deleting from the optimize/bundles folder and rebuilding. I figure this is where the problem lies. The answer might be in the ,babelcache.json file.

hassanmehmud · 2019-09-18T14:45:42Z

any one found solution for this ? Permanent one?

hackdefendr · 2019-09-29T19:19:57Z

I'm starting to wonder if this is a file system related bug. Running this in Windows Subsystem for Linux (WSL) is the only time I encounter issues adding plugins to kibana. When running identical versions in an actual Linux VM on an EXT4 filesystem I do not have this problem with most basic plugins.

So the question I have is, does the optimization process rely in anyway on the file system type?

tylersmalley · 2019-09-30T17:41:40Z

@hackdefendr, I currently running Kibana in dev under WSL2 and was using WSL1 prior without any issues.

@mistic do you have any thoughts on the cause for these folks or something they can test to debug?

hackdefendr · 2019-10-01T01:31:36Z

@tylersmalley have you attempted to install a plugin like country-flag-fieldformatters to kibana? Basically any visualization I try to install causes kibana optimization to go on forever. Only under WSL though.

mistic · 2019-10-01T14:15:54Z

@tylersmalley in case I'm remembering correctly WSL could have some problems with some multi threading features of webpack/some webpack plugins.

One plugin that was causing those problems was terser-webpack-plugin with the parallel option enabled. I was not able to reproduce the problem but I think it is worth it to try it.

@hackdefendr do you think you could upgrade the terser-webpack-plugin present in the kibana version you are using to the version 2.1.2 , then delete the optimize dir and try to start kibana again?

hackdefendr · 2019-10-01T15:04:20Z

Yes I can try that when I get home later since this is in my lab at home. Um, any tips on how to install this other than npm install terser-webpack-plugin --save-dev

How do I install this into Kibana?

mistic · 2019-10-01T16:21:03Z

@hackdefendr I would say you need to have the correct yarn version installed in order to match the kibana needs (I think it should be 1.10.1) and then just run yarn upgrade terser-webpack-plugin@2.1.2 at the root kibana directory

hackdefendr · 2019-10-01T16:24:16Z

Perfect. That I can do...I will update this thread again a bit later.

jbudz · 2019-10-01T21:21:56Z

Related #19678

hackdefendr · 2019-10-02T00:32:46Z

OK I used yarn 1.19.1 and I don't think it worked.

from /usr/share/kibana, I ran:

$ yarn upgrade terser-webpack-plugin@2.1.2
yarn upgrade v1.19.0
warning package.json: License should be a valid SPDX license expression
error No lockfile in this directory. Run `yarn install` to generate one.
info Visit https://yarnpkg.com/en/docs/cli/upgrade for documentation about this command.

I did not run yarn install as it suggested.

Also, I see that 7.4.0 dropped. Should I upgrade before we go any further?

mistic · 2019-10-02T01:15:08Z

@hackdefendr I forgot we did some operations to the package.json we ship within our distributable so the above operation would not work. I think in order to test this maybe the best thing would be checkout kibana from source, change the terser-webpack-plugin version directly on the package.json, run yarn kbn bootstrap and then node scripts/build --skip-os-packages --no-oss in order to generate a new distributable with the intended node_modules version. @hackdefendr Do you wanna me to do it and share with you a zip so you can try it or do you prefer to do it yourself?

hackdefendr · 2019-10-02T01:22:52Z

I think I can do it, but it will be this weekend before I will have the time to do this. I have all of my logs backed up in json format so we can use my WSL as a test. It will be fun to.

hackdefendr · 2019-10-03T01:32:14Z

OK that was interesting. Compiling Kibana under WSL.

Everything built successfully including v2.1.2 terser-webpack-plugin.

Kibana 7.4.1-SNAPSHOT

I will install this version under /opt to keep it separate, then will compile/install a simple plugin like that country flags one I like.

hackdefendr · 2019-10-03T02:15:55Z

Alright Kibana snapshot running successfully and connected to my elasticsearch. I will install the plugin tomorrow, after letting this Kibana run for some time.

hackdefendr · 2019-10-03T16:38:33Z

Ok I woke up this morning and my Kibana SNAPSHOT was still running perfectly. So I went ahead and compiled the country flags plugin and optimization only took about 10 minutes, but did complete successfully. Kibana started right back up, and I was able to change the GEOIP Country Code 2 field to show the Country Flag.

Success!

mistic · 2019-10-03T17:49:37Z

@hackdefendr awesome news. I will proceed and put up a PR with the fix for Kibana. Thanks for the help

hackdefendr · 2019-10-03T17:53:24Z

Not a problem. Like I said, it was fun to do.

legrego added Team:Operations Team label for Operations Team triage_needed labels Jul 23, 2019

mistic mentioned this issue Oct 3, 2019

Fix optimizer under wsl environment with patched terser webpack plugin version #47250

Merged

hackdefendr closed this as completed Oct 4, 2019

dbbaughe mentioned this issue Nov 14, 2019

kibana plugin can not install opendistro-for-elasticsearch/alerting-kibana-plugin#107

Closed

tylersmalley mentioned this issue Feb 19, 2020

[discuss] Require WSL2 for development on Windows #58069

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Kibana Optimizing and caching bundles for 28 hours #41726

Kibana Optimizing and caching bundles for 28 hours #41726

hackdefendr commented Jul 23, 2019

legrego commented Jul 23, 2019

elasticmachine commented Jul 23, 2019

hackdefendr commented Jul 23, 2019

hackdefendr commented Jul 23, 2019 •

edited

Loading

legrego commented Jul 25, 2019

mistic commented Jul 25, 2019

hackdefendr commented Jul 25, 2019

hackdefendr commented Jul 27, 2019

hackdefendr commented Jul 27, 2019 •

edited

Loading

hackdefendr commented Jul 28, 2019 •

edited

Loading

hassanmehmud commented Sep 18, 2019

hackdefendr commented Sep 29, 2019

tylersmalley commented Sep 30, 2019

hackdefendr commented Oct 1, 2019

mistic commented Oct 1, 2019

hackdefendr commented Oct 1, 2019 •

edited

Loading

mistic commented Oct 1, 2019

hackdefendr commented Oct 1, 2019

jbudz commented Oct 1, 2019

hackdefendr commented Oct 2, 2019

mistic commented Oct 2, 2019

hackdefendr commented Oct 2, 2019

hackdefendr commented Oct 3, 2019

hackdefendr commented Oct 3, 2019

hackdefendr commented Oct 3, 2019

mistic commented Oct 3, 2019

hackdefendr commented Oct 3, 2019

Kibana Optimizing and caching bundles for 28 hours #41726

Kibana Optimizing and caching bundles for 28 hours #41726

Comments

hackdefendr commented Jul 23, 2019

legrego commented Jul 23, 2019

elasticmachine commented Jul 23, 2019

hackdefendr commented Jul 23, 2019

hackdefendr commented Jul 23, 2019 • edited Loading

legrego commented Jul 25, 2019

mistic commented Jul 25, 2019

hackdefendr commented Jul 25, 2019

hackdefendr commented Jul 27, 2019

hackdefendr commented Jul 27, 2019 • edited Loading

hackdefendr commented Jul 28, 2019 • edited Loading

hassanmehmud commented Sep 18, 2019

hackdefendr commented Sep 29, 2019

tylersmalley commented Sep 30, 2019

hackdefendr commented Oct 1, 2019

mistic commented Oct 1, 2019

hackdefendr commented Oct 1, 2019 • edited Loading

mistic commented Oct 1, 2019

hackdefendr commented Oct 1, 2019

jbudz commented Oct 1, 2019

hackdefendr commented Oct 2, 2019

mistic commented Oct 2, 2019

hackdefendr commented Oct 2, 2019

hackdefendr commented Oct 3, 2019

hackdefendr commented Oct 3, 2019

hackdefendr commented Oct 3, 2019

mistic commented Oct 3, 2019

hackdefendr commented Oct 3, 2019

hackdefendr commented Jul 23, 2019 •

edited

Loading

hackdefendr commented Jul 27, 2019 •

edited

Loading

hackdefendr commented Jul 28, 2019 •

edited

Loading

hackdefendr commented Oct 1, 2019 •

edited

Loading