Not able to set NODE_OPTIONS in sysconfig for systemd environment #36671

VarunSrinivasa · 2019-05-20T10:37:24Z

**Kibana version:6.6.1

**Elasticsearch version:6.6.1

**Server OS version: Rhel 7.6

**Browser version: 74.0.3729.157 [Chrome]

**Browser OS version: Windows 10 64 bit

**Original install method (e.g. download page, yum, from source, etc.): yum

**Describe the bug:- Not able to set the Node js property --max-http-header-size in Kibana Service file /etc/systemd/system/kibana.service

Steps to reproduce:-

In the file /etc/systemd/system/kibana.service we are setting the property and value Environment=NODE_OPTIONS=--max-old-space-size=1024 --max-http-header-size=65550. But after restarting kibana, The values for the properties --max-old-space-size, --max-http-header-size are not reflecting.
-bash-4.2# ps -eaf|grep kibana
kibana 5422 1 0 10:15 ? 00:00:06 /usr/share/kibana/bin/../node/bin/node --no-warnings --max-http-header-size=65536 /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml
root 6026 4223 0 10:27 pts/1 00:00:00 grep --color=auto kibana

From the above output it is clear that kibana is not reading the file /etc/systemd/system/kibana.service

Also we tried to set using direct environment variables, But kibana is not taking the environment variables.
Ex:- export NODE_OPTIONS=--max-old-space-size=1024 --max-http-header-size=65550
Then restart kibana, Once Kibana comes up changes are not reflecting.
Same thing we tried in K8s environment , But we are not sure whether values are getting reflected.
Ex:- env:
- name: "NODE_OPTIONS"
value: "--max-http-header-size=10000000"
Once Container is restarted, Changes are not reflecting.

mistic · 2019-05-20T15:49:24Z

@VarunSrinivasa could u use the default provided /etc/systemd/system/kibana.service file and then define your env vars inside /etc/sysconfig/kibana as it should be our env var file for rpm based os?
You just need to add the next line to that file

`export NODE_OPTIONS="--max-old-space-size=1024 --max-http-header-size=65550"`

Then run systemctl daemon-reload and restart Kibana, everything should work.

elasticmachine · 2019-05-20T15:49:48Z

Pinging @elastic/kibana-operations

VarunSrinivasa · 2019-05-24T09:14:06Z

@mistic ,
Thanks for your quick response. I will check and update you.

VarunSrinivasa · 2019-05-24T13:07:48Z

Hi @mistic ,

I tried configuring the value export NODE_OPTIONS="--max-old-space-size=1024 --max-http-header-size=65550" in the file /etc/sysconfig/kibana . **But i am getting below error in journal logs **.

May 24 12:44:19 vm-10-99-26-157 systemd[1]: Stopping Kibana...
May 24 12:44:19 vm-10-99-26-157 systemd[1]: Stopped Kibana.
May 24 12:44:26 vm-10-99-26-157 systemd[1]: [/etc/systemd/system/kibana.service:3] Unknown lvalue 'StartLimitIntervalSec' in section 'Unit'
May 24 12:44:26 vm-10-99-26-157 systemd[1]: [/etc/systemd/system/kibana.service:4] Unknown lvalue 'StartLimitBurst' in section 'Unit'
May 24 12:44:26 vm-10-99-26-157 systemd[1]: [/etc/systemd/system/kibana.service:16] Invalid environment assignment, ignoring: NODE_OPTIONS=--max-http-he
May 24 12:44:33 vm-10-99-26-157 systemd[1]: Started Kibana.
May 24 12:44:33 vm-10-99-26-157 kibana[7708]: --max-http-header-size=12000

Also i observe in the file /usr/share/kibana/bin/kibana , The value for --max-http-header-size is hard coded to 65536.
When i am trying to export --max-old-space-size and -max-http-header-size in the file /etc/systemd/system/kibana.service. It is not taking both the properties. But when i am exporting any one property, It is reflecting after restarting kibana.

Ex:- Environment=NODE_OPTIONS=--max-http-header-size=65555
bash-4.2# systemctl status kibana -l
● kibana.service - Kibana
Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: disabled)
Active: active (running) since Fri 2019-05-24 12:53:18 UTC; 4s ago
Main PID: 8281 (node)
CGroup: /system.slice/kibana.service
└─8281 /usr/share/kibana/bin/../node/bin/node --no-warnings --max-http-header-size=65536 --max-http-header-size=65555 /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml

Ex:- Environment=NODE_OPTIONS=--max-http-header-size=65555 --max-old-space-size=712

bash-4.2# systemctl status kibana -l
● kibana.service - Kibana
Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: disabled)
Active: active (running) since Fri 2019-05-24 12:53:18 UTC; 4s ago
Main PID: 8281 (node)
CGroup: /system.slice/kibana.service
└─8281 /usr/share/kibana/bin/../node/bin/node --no-warnings --max-http-header-size=65536 --max-old-space-size=712 /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml

mistic · 2019-05-24T13:32:53Z

@VarunSrinivasa can you try to edit the /etc/sysconfig/kibana and remove the export and the quotes so just use NODE_OPTIONS=--max-old-space-size=1024 --max-http-header-size=65550 ?

VarunSrinivasa · 2019-05-27T06:24:40Z

@mistic ,

Will check and update you.

VarunSrinivasa · 2019-05-27T07:12:06Z

Hi @mistic

I have provided the value NODE_OPTIONS=--max-http-header-size=65550 in the file /etc/sysconfig/kibana. Please find the contents of file /etc/systemd/system/kibana.service

_[_Unit]
Description=Kibana
StartLimitIntervalSec=30
StartLimitBurst=3

[Service]
Type=simple
User=kibana
Group=root
#Load env vars from /etc/default/ and /etc/sysconfig/ if they exist.Prefixing the path with '-' makes it try #to load, but if the file doesn't
#exist, it continues onward.
#EnvironmentFile=-/etc/default/kibana
EnvironmentFile=-/etc/sysconfig/kibana
ExecStart=/usr/share/kibana/bin/kibana "-c /etc/kibana/kibana.yml"
Restart=always
WorkingDirectory=/

[Install]
WantedBy=multi-user.target__

Once i reload the configuration and start the kibana , I am getting below output.

[_root@vm-10-99-26-157 system]# systemctl status kibana -l
● kibana.service - Kibana
Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: disabled)
Active: active (running) since Mon 2019-05-27 06:41:40 UTC; 28min ago
Main PID: 11000 (node)
CGroup: /system.slice/kibana.service
└─11000 /usr/share/kibana/bin/../node/bin/node --no-warnings --max-http-header-size=65536 --max-http-header-size=65550 /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml

May 27 06:41:40 vm-10-99-26-157 systemd[1]: Started Kibana.
[root@vm-10-99-26-157 system]#_

Since there are two values 65536 and 65550 for the property--max-http-header-size .Can you please confirm which value will be used for the property --max-http-header-size?

mistic · 2019-05-27T10:37:01Z

@VarunSrinivasa it will use the last defined value

VarunSrinivasa · 2019-05-27T12:48:11Z

Hi @mistic ,

Thanks for the quick response will test and update you.

aggarwalShivani · 2019-06-11T16:10:09Z

Hi @mistic ,
How can I add this property in docker environment? I am using helm to install kibana docker -
Can the property --max-http-header-size be set an environment variable to the container?

mistic · 2019-06-11T18:22:59Z

@aggarwalShivani I believe you can just set an environment variable to the container docker run -e "NODE_OPTIONS=--max-old-space-size=2048" .....

aggarwalShivani · 2019-06-12T15:38:10Z

Hi @mistic ,
I have set the max-http-header-length as an env NODE_OPTIONS=--max-http-header-length=1048576 in my kibana container.

I use kibana with searchguard and the authentication mode is keycloak(openid). The flow is like -

On accessing kibana UI, it redirects to keycloak for user authentication
If user is authenticated, keycloak sends a jwt token to kibana backend
kibana uses this id-token in the header to interact with elasticsearch and loads kibana UI.

I observe that when my token's size is around 1-2KB, i am able to login to kibana. But, when the token size is larger (say 9kb) , the login fails.
Is there some other module used by kibana (node-module like http, wreck?) that could be imposing the limit of 8kb header length? If yes, can you tell me how can I configure that?

mistic · 2019-06-13T01:02:59Z

@jbudz I know you have done something in the past related to max-http-header-length, do you have any thoughts or can you think about someone that could provide any help here?

jbudz · 2019-06-13T13:25:53Z

I want to dig into this further when I get a moment, so will reopen. For now, if you want to just make this work you can modify these values in /usr/share/kibana/bin/kibana on the last line.

Sorry for the terse reply will update further when I get a chance!

tylersmalley · 2020-03-18T18:37:52Z

@jbudz any update here?

tylersmalley · 2020-07-22T16:18:02Z

I think we can close this now that we have a node.options file in the config directory which should be used instead.

#62468

of "/etc/default/kibana". This is a standards difference. rpm based distributions generally use /etc/sysconfig. In production we're reading from both, with sysconfig second to override any historical differences. Related to elastic#36671, elastic#6423

mistic closed this as completed May 20, 2019

mistic added the Team:Operations Team label for Operations Team label May 20, 2019

jbudz reopened this Jun 13, 2019

tylersmalley changed the title ~~Not able to set the Node js property --max-http-header-size in Kibana~~ Not able to set NODE_OPTIONS in sysconfig for systemd environment Feb 19, 2020

tylersmalley closed this as completed Jul 22, 2020

kunisen mentioned this issue Aug 4, 2020

Kibana NODE_OPTIONS for --max_old_space_size doesn't work #74213

Closed

jbudz mentioned this issue Oct 30, 2020

[rpm] Create default environment file at "/etc/sysconfig/kibana" #82144

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Not able to set NODE_OPTIONS in sysconfig for systemd environment #36671

Not able to set NODE_OPTIONS in sysconfig for systemd environment #36671

VarunSrinivasa commented May 20, 2019

mistic commented May 20, 2019

elasticmachine commented May 20, 2019

VarunSrinivasa commented May 24, 2019

VarunSrinivasa commented May 24, 2019

mistic commented May 24, 2019

VarunSrinivasa commented May 27, 2019

VarunSrinivasa commented May 27, 2019 •

edited

Loading

mistic commented May 27, 2019

VarunSrinivasa commented May 27, 2019

aggarwalShivani commented Jun 11, 2019

mistic commented Jun 11, 2019

aggarwalShivani commented Jun 12, 2019

mistic commented Jun 13, 2019

jbudz commented Jun 13, 2019 •

edited

Loading

tylersmalley commented Mar 18, 2020

tylersmalley commented Jul 22, 2020

Not able to set NODE_OPTIONS in sysconfig for systemd environment #36671

Not able to set NODE_OPTIONS in sysconfig for systemd environment #36671

Comments

VarunSrinivasa commented May 20, 2019

mistic commented May 20, 2019

elasticmachine commented May 20, 2019

VarunSrinivasa commented May 24, 2019

VarunSrinivasa commented May 24, 2019

mistic commented May 24, 2019

VarunSrinivasa commented May 27, 2019

VarunSrinivasa commented May 27, 2019 • edited Loading

mistic commented May 27, 2019

VarunSrinivasa commented May 27, 2019

aggarwalShivani commented Jun 11, 2019

mistic commented Jun 11, 2019

aggarwalShivani commented Jun 12, 2019

mistic commented Jun 13, 2019

jbudz commented Jun 13, 2019 • edited Loading

tylersmalley commented Mar 18, 2020

tylersmalley commented Jul 22, 2020

VarunSrinivasa commented May 27, 2019 •

edited

Loading

jbudz commented Jun 13, 2019 •

edited

Loading