Missing systemd scripts in 6.2.4

[robgil]

• Version: 6.2.4
• Operating System: CentOS 7.4.1708
• Config File (if you have sensitive info, please remove it): NA
• Sample Data: NA
• Steps to Reproduce:

Yum Repo

[elasticsearch-6.x]
name=Elasticsearch repository for 6.x packages
baseurl=https://artifacts.elastic.co/packages/6.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md

Missing Systemd

# rpm -ql logstash|grep /usr/lib/systemd/system/
#

yum info

# yum info logstash
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: mirror.23media.de
 * epel: mirror.de.leaseweb.net
 * extras: ftp.plusline.de
 * updates: ftp.rz.uni-frankfurt.de
Installed Packages
Name        : logstash
Arch        : noarch
Epoch       : 1
Version     : 6.2.4
Release     : 1
Size        : 237 M
Repo        : installed
From repo   : elasticsearch-6.x
Summary     : An extensible logging pipeline
URL         : http://www.elasticsearch.org/overview/logstash/
License     : ASL 2.0
Description : An extensible logging pipeline

RPM Info

# rpm -qi logstash
Name        : logstash
Epoch       : 1
Version     : 6.2.4
Release     : 1
Architecture: noarch
Install Date: Wed 18 Apr 2018 05:55:18 PM UTC
Group       : default
Size        : 248557285
License     : ASL 2.0
Signature   : RSA/SHA512, Thu 12 Apr 2018 10:46:14 PM UTC, Key ID d27d666cd88e42b4
Source RPM  : logstash-6.2.4-1.src.rpm
Build Date  : Thu 12 Apr 2018 10:31:05 PM UTC
Build Host  : packer-virtualbox-iso-1522340558
Relocations : / 
Packager    : <vagrant@packer-virtualbox-iso-1522340558>
Vendor      : Elasticsearch
URL         : http://www.elasticsearch.org/overview/logstash/
Summary     : An extensible logging pipeline
Description :
An extensible logging pipeline

Unit Files

# systemctl list-unit-files -a|grep logstash
#

[robgil]

Quick testing shows this impacts all 6.x versions. Testing 5.x next.

[jordansissel]

Hmm, the systemd scripts should be generated at package install time.

% sudo yum install logstash
...
Installing:
 logstash                         noarch                         1:6.2.3-1                           elasticsearch-6.x                         133 M

...
Is this ok [y/N]: y
...
Running transaction
  Preparing        :                                                                                                                             1/1
  Running scriptlet: logstash-1:6.2.3-1.noarch                                                                                                   1/1
  Installing       : logstash-1:6.2.3-1.noarch                                                                                                   1/1
  Running scriptlet: logstash-1:6.2.3-1.noarch                                                                                                   1/1
Using provided startup.options file: /etc/logstash/startup.options
Successfully created system startup script for Logstash
  Verifying        : logstash-1:6.2.3-1.noarch                                                                                                   1/1

Installed:
  logstash.noarch 1:6.2.3-1

% systemctl list-unit-files | grep logstash
logstash.service                            disabled

% sudo yum remove logstash
...
% systemctl list-unit-files | grep -c logstash
0

[robgil]

This is due to java not being installed. Since listing java as a dep would be hard (sun jdk, ibm jdk, openjdk, etc).

Warning: RPMDB altered outside of yum.
  Installing : 1:logstash-5.6.9-1.noarch                                                                             1/1 
Could not find any executable java binary. Please install java in your PATH or set JAVA_HOME.
warning: %post(logstash-1:5.6.9-1.noarch) scriptlet failed, exit status 1
Non-fatal POSTIN scriptlet failure in rpm package 1:logstash-5.6.9-1.noarch
  Verifying  : 1:logstash-5.6.9-1.noarch                                                                             1/1 

Installed:
  logstash.noarch 1:5.6.9-1                                                                                              

[robgil]

This fails because java isn't installed, yet is not defined as a dependency. Java is used also for system discovery (os version, systemd/upstart/sysv). A few options that might help make this less weird.

• Fail hard if java is not installed. Currently returns 0
• Avoid generating files in the packages
  • Offer distro specific packages for OSs that we support only (my preference - we'll need to move this route when we implement apparmor, selinux, FIPS)
    • RHEL / CentOS 6/7, Ubuntu
  • Create secondary packages with the specific scripts (logstash-systemd, logstash-sysv, logstash-upstart)
• Distribute our own version of java in our repo and make it a dep (version lock to the java we support). Users can change JAVA_HOME or alternatives configs if they choose to use something else.

As is, this gives a false positive that installation was successful.

[d34dh0r53]

This is affecting logstash-5.6.9-1 on Ubuntu as well. At the very least the failure to install should not be silent.

[andrewvc]

This is an interesting issue given the java dependency issue. We should, at the very least, make that error be fatal instead of non-fatal.

@d34dh0r53 I assume you're seeing the same thing?

We should investigate debian, where I believe it is possible to list any Java as a valid Java dep.

[jordansissel]

where I believe it is possible to list any Java as a valid Java dep.

There's an issue I can't find right now where we did that research and showed it was not possible (Oracle JDK is not available as a Deb, for example, only OpenJDK, and last I checked, the web-upd8 PPA uses a naming scheme which makes it not quite compatible with the upstream debian openjdk ones)

That said, if we only supported OpenJDK from Debian/Ubuntu directly, this would be possible. (In the past, one driver was rejecting this dependency because OpenJDK 8 wasnot available on supported Debian/Ubuntu releases, but this is not true anymore, but we'll have the same problem with Java 11 is released...)

[andrewvc]

So it seems like we can solve this by having a pre install script fail if no java is found, right? That should show an error and prevent the package install from succeeding.

[robgil]

@andrewvc yup

[jordansissel]

warning: %post(logstash-1:5.6.9-1.noarch) scriptlet failed, exit status 1
Non-fatal POSTIN scriptlet failure in rpm package 1:logstash-5.6.9-1.noarch

I think the exit status needs to be some special number (that isn't 1) that tells yum/dnf/rpm to consider it fatal.

I glanced around the rpm docs on scriptlets and wasn't able to find much firm detail on what specific exit codes mean.

[georgexsh]

@jordansissel agreed, system-install should not eat exit code to fail silently.
another issue I encountered is due to #7725, JRuby/JVM cannot be started on a small memory machine, got some error message, but RPM doesn't complain at all.

[a-nldisr]

Issue found in latest version of logstash: 6.4.0, both in the rpm binary as in the repo installation.

[roman-kutlak]

Just a note here -- I have also encountered this problem when my system run out of memory:

Running transaction
  Installing : 1:logstash-6.5.4-1.noarch                                                                                                                                                         1/1 
Using provided startup.options file: /etc/logstash/startup.options
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00000000c5330000, 986513408, 0) failed; error='Cannot allocate memory' (errno=12)
/usr/share/logstash/bin/system-install: line 88: #: command not found
Unable to install system startup script for Logstash.
chmod: cannot access ‘/etc/default/logstash’: No such file or directory
warning: %post(logstash-1:6.5.4-1.noarch) scriptlet failed, exit status 1
Non-fatal POSTIN scriptlet failure in rpm package 1:logstash-6.5.4-1.noarch
  Verifying  : 1:logstash-6.5.4-1.noarch                                                                                                                                                         1/1 

Installed:
  logstash.noarch 1:6.5.4-1      

I guess normally you would see the error but when installing remotely using Ansible, it just disappears in the log.

[CODE-REaD]

I recently experienced the problem of logstash installing successfully under CentOS 7 but without configuring its systemctl files. As I recall a nonfatal error occurred regarding java at the time. I since installed java and found that after I yum eraseed logstash and then yum installed it, the systemctl files now installed correctly. So, thank you for this tip, and be advised the java dependency problem apparently lives on.

[edward2a]

The logstash RPM does not have java dependency indication, so in a system that does not have java installed, the installation does not create the init scripts, it does not generate the bin files (bin/logstash is not linked to a system path) and the things just falls apart.

This is still the case for latest version.
Please, fix the dependency.

[rafzei]

Correct, there should be at least requirements page so you can easily compare it to your environment.