Upgrade complains due to userdel ad already running process

[markwalkom]

Running an upgrade on Ubuntu 14.04 to 4.5;

# apt-get dist-upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
The following packages will be upgraded:
  kibana logstash oracle-java8-installer
3 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
31 not fully installed or removed.
Need to get 108 MB of archives.
After this operation, 4,444 kB of additional disk space will be used.
Do you want to continue? [Y/n]

And then;

Unpacking kibana (4.5.0) over (4.4.1) ...
userdel: user kibana is currently used by process 1024
dpkg: warning: subprocess old post-removal script returned error exit status 8
dpkg: trying script from the new package instead ...
userdel: user kibana is currently used by process 1024
dpkg: error processing archive /var/cache/apt/archives/kibana_4.5.0_amd64.deb (--unpack):
 subprocess new post-removal script returned error exit status 8
userdel: user kibana is currently used by process 1024
dpkg: error while cleaning up:
 subprocess new post-removal script returned error exit status 8
Errors were encountered while processing:
 /var/cache/apt/archives/kibana_4.5.0_amd64.deb
E: Sub-process /usr/bin/dpkg returned an error code (1)

There's no reason for us to call userdel, we don't do it during an ES install upgrade for eg.

I couldn't find the code that handles this to comment directly there sorry.

[GoodMirek]

Upgrade 4.4.1 --> 4.4.2 is also affected.

[ccurvey]

and 4.5.0 --> 4.5.1

[rlueckl]

+1

I had the same issue while updating from 4.5.0 to 4.5.1

Proposed solution: add a "service kibana stop" to the post-remove (or pre-remove) script.

[jbudz]

Fixed in #7308

[william0901]

4.5.1 to 4.5.4 is still broken.

[jbudz]

@williamelastic do you have some more info on steps to reproduce? This should be fixed in debs. rpm has a slightly different script ordering and will take one more release to clear out old package scripts, the fix itself is in.

ubuntu@kbdeb64:~$ sudo dpkg -i kibana_4.5.1_amd64.deb
Selecting previously unselected package kibana.
(Reading database ... 55400 files and directories currently installed.)
Preparing to unpack kibana_4.5.1_amd64.deb ...
Unpacking kibana (4.5.1) ...
Setting up kibana (4.5.1) ...
Processing triggers for systemd (229-4ubuntu6) ...
Processing triggers for ureadahead (0.100.0-19) ...
ubuntu@kbdeb64:~$ sudo service kibana start
ubuntu@kbdeb64:~$ sudo service kibana status
● kibana.service - no description given
   Loaded: loaded (/lib/systemd/system/kibana.service; disabled; vendor preset: enabled)
   Active: active (running) since Thu 2016-08-04 14:01:43 UTC; 11s ago
 Main PID: 6661 (node)
    Tasks: 9
   Memory: 91.1M
      CPU: 1.644s
   CGroup: /system.slice/kibana.service
           └─6661 /opt/kibana/bin/../node/bin/node /opt/kibana/bin/../src/cli

Aug 04 14:01:45 kbdeb64 kibana[6661]: {"type":"log","@timestamp":"2016-08-04T14:01:45+00:00","tags":["status","plugin:spyModes","info"]
Aug 04 14:01:45 kbdeb64 kibana[6661]: {"type":"log","@timestamp":"2016-08-04T14:01:45+00:00","tags":["status","plugin:statusPage","info
Aug 04 14:01:45 kbdeb64 kibana[6661]: {"type":"log","@timestamp":"2016-08-04T14:01:45+00:00","tags":["status","plugin:table_vis","info"
Aug 04 14:01:45 kbdeb64 kibana[6661]: {"type":"log","@timestamp":"2016-08-04T14:01:45+00:00","tags":["listening","info"],"pid":6661,"me
Aug 04 14:01:47 kbdeb64 kibana[6661]: {"type":"log","@timestamp":"2016-08-04T14:01:47+00:00","tags":["warning","elasticsearch"],"pid":6
Aug 04 14:01:47 kbdeb64 kibana[6661]: {"type":"log","@timestamp":"2016-08-04T14:01:47+00:00","tags":["warning","elasticsearch"],"pid":6
Aug 04 14:01:50 kbdeb64 kibana[6661]: {"type":"log","@timestamp":"2016-08-04T14:01:50+00:00","tags":["warning","elasticsearch"],"pid":6
Aug 04 14:01:50 kbdeb64 kibana[6661]: {"type":"log","@timestamp":"2016-08-04T14:01:50+00:00","tags":["warning","elasticsearch"],"pid":6
Aug 04 14:01:52 kbdeb64 kibana[6661]: {"type":"log","@timestamp":"2016-08-04T14:01:52+00:00","tags":["warning","elasticsearch"],"pid":6
Aug 04 14:01:52 kbdeb64 kibana[6661]: {"type":"log","@timestamp":"2016-08-04T14:01:52+00:00","tags":["warning","elasticsearch"],"pid":6
ubuntu@kbdeb64:~$ sudo dpkg -i kibana_4.5.4_amd64.deb
(Reading database ... 72551 files and directories currently installed.)
Preparing to unpack kibana_4.5.4_amd64.deb ...
Unpacking kibana (4.5.4) over (4.5.1) ...
Setting up kibana (4.5.4) ...
Installing new version of config file /etc/default/kibana ...
Processing triggers for systemd (229-4ubuntu6) ...
Processing triggers for ureadahead (0.100.0-19) ...

[william0901]

Here are the steps to reproduce

CentOS 7.2/rpm package

# yum install kibana-4.5.1-1.x86_64.rpm 
# tail -3 /etc/passwd
elasticsearch:x:996:994:elasticsearch user:/home/elasticsearch:/sbin/nologin
logstash:x:995:993:logstash:/opt/logstash:/sbin/nologin
kibana:x:994:992::/home/kibana:/bin/bash  <---- kibana userid is created
# rpm -Uvh kibana-4.5.4-1.x86_64.rpm 
Preparing...                          ################################# [100%]
Updating / installing...
   1:kibana-4.5.4-1                   ################################# [ 50%]
Cleaning up / removing...
   2:kibana-4.5.1-1                   ################################# [100%]
[root@localhost ~]# tail -3 /etc/passwd
sshd:x:74:74:Privilege-separated SSH:/var/empty/sshd:/sbin/nologin
elasticsearch:x:996:994:elasticsearch user:/home/elasticsearch:/sbin/nologin
logstash:x:995:993:logstash:/opt/logstash:/sbin/nologin
<--- kibana id is removed.

You can see kibana userid was removed after the upgrade.

[ksbarnt]

Not sure how much help this will be, but on my CentOS 6 box upgrading from 4.5.3 to 4.5.4 (via yum using the elastic repo) removed the kibana user, but downgrading back to 4.5.3 recreated it.

[epixa]

@jbudz I'm not sure I understand the situation. Are you saying that this is not going to be an issue going forward? If users get 4.5.4 installed (either through a purge/install or through an upgrade/manual fix up on permissions), will they have an issue upgrading from that point onward? If so, under what circumstances? Is there anything anyone can do right now to help ease the upgrade path to 4.5.4?

[jbudz]

Correct, this will not be an issue after an installation of 4.5.4+. 4.5.4->4.5.5, 4.5.4->4.x, alpha4+ will just work.

For rpms the last script ran is post remove of the old package which in 4.5.{0,1,2,3} blindly deletes the kibana user. debs aren't running into this because the last script ran is post install of the new package, which ensures the kibana user exists. 4.5.4 only removes the user on package removal, not upgrades, fixing this for future releases.

A potential fix is adding a post transaction to rpms that will re-run the new post installation script at the very end of an upgrade/install. This wasn't done originally because it adds technical debt, and this isn't a new regression in 4.5.4 - upgrades have never worked previously. The manual fix is adding a kibana user and group and running chown -R kibana:kibana /opt/kibana/optimize, or to remove and then install. If this is manually fixed when upgrading to 4.5.4, there should not be an issue from then on, the root cause is the old package removal scripts.

[epixa]

That's really helpful, thanks.

[jbudz]

Like mentioned above, this is fixed in 4.5.4+, and there are docs on upgrading from older versions: https://www.elastic.co/guide/en/kibana/current/upgrading-kibana.html. Closing this one now.