Ubuntu 18.04 kibana.service Unknown lvalue 'StartLimitIntervalSec' in section 'Unit' #38842

LeeDr · 2019-06-12T21:16:40Z

Kibana version: 7.2.0

Elasticsearch version: 7.2.0

Server OS version: Ubuntu 18.04
Linux vagrant 4.4.0-31-generic #50-Ubuntu SMP Wed Jul 13 00:07:12 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

Browser version: Chrome

Browser OS version: Windows 10

Original install method (e.g. download page, yum, from source, etc.): .deb packages default distribution

Describe the bug: If there's some bug that prevents Kibana from starting properly or if Kibana crashes, the /etc/systemd/system/kibana.service file causes it to attempt to restart. But apparently a setting has changed between version of systemd where our settings are not valid on this version.

Steps to reproduce:

Install the kibana deb package on Ubuntu 18.04
create a problem in your kibana.yml file like having an unknown setting or duplicate setting (I had a duplicate of elasticsearch.hosts:)
start kibana service
check journalctl -u kibana to see logs

Expected behavior:

Screenshots (if relevant):

Errors in browser console (if relevant):

Provide logs and/or server output (if relevant):
This Unknown lvalue 'StartLimitIntervalSec' in section 'Unit' shows up at the beginning and the end of this log;

root@vagrant:~# journalctl -u kibana
-- Logs begin at Wed 2019-06-12 18:14:57 UTC, end at Wed 2019-06-12 20:19:38 UTC. --
Jun 12 18:23:41 vagrant systemd[1]: [/etc/systemd/system/kibana.service:3] Unknown lvalue 'StartLimitIntervalSec' in section 'Unit'
Jun 12 18:23:41 vagrant systemd[1]: Started Kibana.
Jun 12 18:23:42 vagrant kibana[7747]: FATAL CLI ERROR YAMLException: duplicated mapping key at line 128, column 1:
Jun 12 18:23:42 vagrant kibana[7747]:     elasticsearch.hosts: https://ela ...
Jun 12 18:23:42 vagrant kibana[7747]:     ^
Jun 12 18:23:42 vagrant kibana[7747]:     at generateError (/usr/share/kibana/node_modules/js-yaml/lib/js-yaml/loader.js:167:10)
Jun 12 18:23:42 vagrant kibana[7747]:     at throwError (/usr/share/kibana/node_modules/js-yaml/lib/js-yaml/loader.js:173:9)
Jun 12 18:23:42 vagrant kibana[7747]:     at storeMappingPair (/usr/share/kibana/node_modules/js-yaml/lib/js-yaml/loader.js:335:7)
Jun 12 18:23:42 vagrant kibana[7747]:     at readBlockMapping (/usr/share/kibana/node_modules/js-yaml/lib/js-yaml/loader.js:1098:9)
Jun 12 18:23:42 vagrant kibana[7747]:     at composeNode (/usr/share/kibana/node_modules/js-yaml/lib/js-yaml/loader.js:1359:12)
Jun 12 18:23:42 vagrant kibana[7747]:     at readDocument (/usr/share/kibana/node_modules/js-yaml/lib/js-yaml/loader.js:1519:3)
Jun 12 18:23:42 vagrant kibana[7747]:     at loadDocuments (/usr/share/kibana/node_modules/js-yaml/lib/js-yaml/loader.js:1575:5)
Jun 12 18:23:42 vagrant kibana[7747]:     at load (/usr/share/kibana/node_modules/js-yaml/lib/js-yaml/loader.js:1596:19)
Jun 12 18:23:42 vagrant kibana[7747]:     at safeLoad (/usr/share/kibana/node_modules/js-yaml/lib/js-yaml/loader.js:1618:10)
Jun 12 18:23:42 vagrant kibana[7747]:     at readYaml (/usr/share/kibana/src/core/server/config/read_config.js:34:47)
Jun 12 18:23:42 vagrant systemd[1]: kibana.service: Main process exited, code=exited, status=1/FAILURE
Jun 12 18:23:42 vagrant systemd[1]: kibana.service: Unit entered failed state.
Jun 12 18:23:42 vagrant systemd[1]: kibana.service: Failed with result 'exit-code'.
Jun 12 18:23:43 vagrant systemd[1]: kibana.service: Service hold-off time over, scheduling restart.
Jun 12 18:23:43 vagrant systemd[1]: Stopped Kibana.
Jun 12 18:23:43 vagrant systemd[1]: Started Kibana.
Jun 12 18:23:44 vagrant kibana[7766]: FATAL CLI ERROR YAMLException: duplicated mapping key at line 128, column 1:
Jun 12 18:23:44 vagrant kibana[7766]:     elasticsearch.hosts: https://ela ...
Jun 12 18:23:44 vagrant kibana[7766]:     ^
Jun 12 18:23:44 vagrant kibana[7766]:     at generateError (/usr/share/kibana/node_modules/js-yaml/lib/js-yaml/loader.js:167:10)
Jun 12 18:23:44 vagrant kibana[7766]:     at throwError (/usr/share/kibana/node_modules/js-yaml/lib/js-yaml/loader.js:173:9)
Jun 12 18:23:44 vagrant kibana[7766]:     at storeMappingPair (/usr/share/kibana/node_modules/js-yaml/lib/js-yaml/loader.js:335:7)
Jun 12 18:23:44 vagrant kibana[7766]:     at readBlockMapping (/usr/share/kibana/node_modules/js-yaml/lib/js-yaml/loader.js:1098:9)
Jun 12 18:23:44 vagrant kibana[7766]:     at composeNode (/usr/share/kibana/node_modules/js-yaml/lib/js-yaml/loader.js:1359:12)
Jun 12 18:23:44 vagrant kibana[7766]:     at readDocument (/usr/share/kibana/node_modules/js-yaml/lib/js-yaml/loader.js:1519:3)
Jun 12 18:23:44 vagrant kibana[7766]:     at loadDocuments (/usr/share/kibana/node_modules/js-yaml/lib/js-yaml/loader.js:1575:5)
Jun 12 18:23:44 vagrant systemd[1]: kibana.service: Main process exited, code=exited, status=1/FAILURE
Jun 12 18:23:44 vagrant systemd[1]: kibana.service: Unit entered failed state.
Jun 12 18:23:44 vagrant systemd[1]: kibana.service: Failed with result 'exit-code'.
Jun 12 18:23:44 vagrant systemd[1]: kibana.service: Service hold-off time over, scheduling restart.
Jun 12 18:23:44 vagrant systemd[1]: Stopped Kibana.
Jun 12 18:23:44 vagrant systemd[1]: Started Kibana.
Jun 12 18:23:45 vagrant kibana[7776]: FATAL CLI ERROR YAMLException: duplicated mapping key at line 128, column 1:
Jun 12 18:23:45 vagrant kibana[7776]:     elasticsearch.hosts: https://ela ...
Jun 12 18:23:45 vagrant kibana[7776]:     ^
Jun 12 18:23:45 vagrant kibana[7776]:     at generateError (/usr/share/kibana/node_modules/js-yaml/lib/js-yaml/loader.js:167:10)
Jun 12 18:23:45 vagrant kibana[7776]:     at throwError (/usr/share/kibana/node_modules/js-yaml/lib/js-yaml/loader.js:173:9)
Jun 12 18:23:45 vagrant kibana[7776]:     at storeMappingPair (/usr/share/kibana/node_modules/js-yaml/lib/js-yaml/loader.js:335:7)
Jun 12 18:23:45 vagrant kibana[7776]:     at readBlockMapping (/usr/share/kibana/node_modules/js-yaml/lib/js-yaml/loader.js:1098:9)
Jun 12 18:23:45 vagrant kibana[7776]:     at composeNode (/usr/share/kibana/node_modules/js-yaml/lib/js-yaml/loader.js:1359:12)
Jun 12 18:23:45 vagrant kibana[7776]:     at readDocument (/usr/share/kibana/node_modules/js-yaml/lib/js-yaml/loader.js:1519:3)
Jun 12 18:23:45 vagrant kibana[7776]:     at loadDocuments (/usr/share/kibana/node_modules/js-yaml/lib/js-yaml/loader.js:1575:5)
Jun 12 18:23:45 vagrant kibana[7776]:     at load (/usr/share/kibana/node_modules/js-yaml/lib/js-yaml/loader.js:1596:19)
Jun 12 18:23:45 vagrant kibana[7776]:     at safeLoad (/usr/share/kibana/node_modules/js-yaml/lib/js-yaml/loader.js:1618:10)
Jun 12 18:23:45 vagrant systemd[1]: kibana.service: Main process exited, code=exited, status=1/FAILURE
Jun 12 18:23:45 vagrant systemd[1]: kibana.service: Unit entered failed state.
Jun 12 18:23:45 vagrant systemd[1]: kibana.service: Failed with result 'exit-code'.
Jun 12 18:23:45 vagrant systemd[1]: kibana.service: Service hold-off time over, scheduling restart.
Jun 12 18:23:45 vagrant systemd[1]: Stopped Kibana.
Jun 12 18:23:45 vagrant systemd[1]: kibana.service: Start request repeated too quickly.
Jun 12 18:23:45 vagrant systemd[1]: Failed to start Kibana.
Jun 12 20:19:38 vagrant systemd[1]: [/etc/systemd/system/kibana.service:3] Unknown lvalue 'StartLimitIntervalSec' in section 'Unit'

Any additional context:

Here's the /etc/systemd/system/kibana.service from my install;

root@vagrant:~# cat /etc/systemd/system/kibana.service
[Unit]
Description=Kibana
StartLimitIntervalSec=30
StartLimitBurst=3

[Service]
Type=simple
User=kibana
Group=kibana
# Load env vars from /etc/default/ and /etc/sysconfig/ if they exist.
# Prefixing the path with '-' makes it try to load, but if the file doesn't
# exist, it continues onward.
EnvironmentFile=-/etc/default/kibana
EnvironmentFile=-/etc/sysconfig/kibana
ExecStart=/usr/share/kibana/bin/kibana "-c /etc/kibana/kibana.yml"
Restart=always
WorkingDirectory=/

[Install]
WantedBy=multi-user.target

The text was updated successfully, but these errors were encountered:

elasticmachine · 2019-06-12T21:16:42Z

Pinging @elastic/kibana-platform

elasticmachine · 2019-06-13T14:22:50Z

Pinging @elastic/kibana-operations

maxhallhatesm3 · 2019-09-09T19:29:38Z

Saw the same log error on Centos 7.

Kibana version: 7.3.1
Elasticsearch version: 7.3.1
Server OS version: CentOS Linux release 7.5.1804 (Core)

Browser version: Safari
Browser OS version: OSX 10.14.5

Original install method: yum

Steps to reproduce:

Install the kibana yum package on Centos 7 Core
start kibana service

Sep 09 15:11:02 comp1 systemd[1]: [/etc/systemd/system/kibana.service:3] Unknown lvalue 'StartLimitIntervalSec' in section 'Unit'

Sep 09 15:11:02 comp1 systemd[1]: [/etc/systemd/system/kibana.service:4] Unknown lvalue 'StartLimitBurst' in section 'Unit'

dixon1e · 2019-09-11T14:06:56Z

Did this change to or from 'StartLimitInterval' ? See here:
NixOS/nixpkgs#45785

Atem18 · 2019-09-18T12:12:49Z

@dixon1e It changes from StartLimitInterval to StartLimitIntervalSec in systemd 230. But since Centos 7 still uses Systemd 219, then the update of .service file from Elastic devs is preventing Elastic to start...

mkauf · 2019-10-03T12:12:05Z

"StartLimitIntervalSec" and "StartLimitBurst" are in the section "[Unit]". ~~That's wrong, they must be moved to the section "[Service]".~~

Atem18 · 2019-10-03T13:01:18Z

@mkauf no, you are wrong : https://lists.freedesktop.org/archives/systemd-devel/2017-July/039255.html

StartLimitInterval was moved from [Service] to [Unit] section in
6bf0f408e4833152197fb38fb10a9989c89f3a59, but the old location was
still accepted for compatibility. The new StartLimitIntervalSec name
is valid only in [Unit].

mkauf · 2019-10-03T14:27:48Z

@Atem18 ok, thanks for pointing this out.

but the old location was still accepted for compatibility

So maybe Kibana could also use the old location to improve the compatibility with systems that don't know about the new location yet (e. g. CentOS 7).

Atem18 · 2019-10-03T15:06:21Z

@mkauf Well, the big issue here is that they use both the new name and the new location. CentOS 7 know neither of them. So technically, they would have to maintain two files and according to the OS, copy the right one.

mkauf · 2019-10-07T09:44:25Z

@Atem18 It's possible to use the old name "StartLimitInterval" instead of "StartLimitIntervalSec".

I have tested this:

[Service]
StartLimitInterval=30s
StartLimitBurst=3

It works on CentOS 7 (systemd 219) and it also works on Fedora 30 (systemd 241). The "Unknown lvalue" message does not appear anymore.

Atem18 · 2019-10-07T11:26:58Z

@mkauf Then that should be the solution.

torrycrass · 2019-10-09T16:37:51Z

Confirmed this is also a problem on Ubuntu 16.04.6 LTS with Kibana 7.4.0

lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 16.04.6 LTS
Release:	16.04
Codename:	xenial

Updating the file as follows allowed Kibana to start.

[Unit]
Description=Kibana

[Service]
StartLimitIntervalSec=30
StartLimitBurst=3
Type=simple
User=kibana
Group=kibana
# Load env vars from /etc/default/ and /etc/sysconfig/ if they exist.
# Prefixing the path with '-' makes it try to load, but if the file doesn't
# exist, it continues onward.
EnvironmentFile=-/etc/default/kibana
EnvironmentFile=-/etc/sysconfig/kibana
ExecStart=/usr/share/kibana/bin/kibana "-c /etc/kibana/kibana.yml"
Restart=always
WorkingDirectory=/

[Install]
WantedBy=multi-user.target

systemctl status kibana.service 
● kibana.service - Kibana
   Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2019-10-09 12:29:49 EDT; 6min ago
 Main PID: 9925 (node)
   CGroup: /system.slice/kibana.service
           └─9925 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml

tylersmalley · 2020-03-18T18:36:40Z

This was resolved in #47909 starting with 7.5.0.

LeeDr added bug Fixes for quality problems that affect the customer experience Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc labels Jun 12, 2019

jbudz added Team:Operations Team label for Operations Team and removed Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc labels Jun 13, 2019

tylersmalley closed this as completed Mar 18, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Ubuntu 18.04 kibana.service Unknown lvalue 'StartLimitIntervalSec' in section 'Unit' #38842

Ubuntu 18.04 kibana.service Unknown lvalue 'StartLimitIntervalSec' in section 'Unit' #38842

LeeDr commented Jun 12, 2019

elasticmachine commented Jun 12, 2019

elasticmachine commented Jun 13, 2019

maxhallhatesm3 commented Sep 9, 2019

dixon1e commented Sep 11, 2019 •

edited

Loading

Atem18 commented Sep 18, 2019

mkauf commented Oct 3, 2019 •

edited

Loading

Atem18 commented Oct 3, 2019

mkauf commented Oct 3, 2019

Atem18 commented Oct 3, 2019

mkauf commented Oct 7, 2019

Atem18 commented Oct 7, 2019

torrycrass commented Oct 9, 2019

tylersmalley commented Mar 18, 2020

Ubuntu 18.04 kibana.service Unknown lvalue 'StartLimitIntervalSec' in section 'Unit' #38842

Ubuntu 18.04 kibana.service Unknown lvalue 'StartLimitIntervalSec' in section 'Unit' #38842

Comments

LeeDr commented Jun 12, 2019

elasticmachine commented Jun 12, 2019

elasticmachine commented Jun 13, 2019

maxhallhatesm3 commented Sep 9, 2019

dixon1e commented Sep 11, 2019 • edited Loading

Atem18 commented Sep 18, 2019

mkauf commented Oct 3, 2019 • edited Loading

Atem18 commented Oct 3, 2019

mkauf commented Oct 3, 2019

Atem18 commented Oct 3, 2019

mkauf commented Oct 7, 2019

Atem18 commented Oct 7, 2019

torrycrass commented Oct 9, 2019

tylersmalley commented Mar 18, 2020

dixon1e commented Sep 11, 2019 •

edited

Loading

mkauf commented Oct 3, 2019 •

edited

Loading