icinga2 pki new-cert only generates private key on CentOS 7.* (getrandom() not implemented in SolusVM host kernels)

[sharkyzz]

Describe the bug

We are using the icinga2-agent-kickstart script to deploy on CentOS hosts. One step in the script is:

if ! "$ICINGA2_BIN" pki new-cert --cn "${ICINGA2_NODENAME}" \ --cert "${ICINGA2_SSLDIR}/${ICINGA2_NODENAME}.crt" \ --csr "${ICINGA2_SSLDIR}/${ICINGA2_NODENAME}.csr" \ --key "${ICINGA2_SSLDIR}/${ICINGA2_NODENAME}.key"

This should result in a csr, crt and key file. However on CentOS only the key is generated. When running this command on eg. Debian it works as expected.

To Reproduce

icinga2 pki new-cert --cn --key /var/lib/icinga2/certs/.key --cert /var/lib/icinga2/certs/.crt --csr /var/lib/icinga2/certs/.csr

Your Environment

Include as many relevant details about the environment you experienced the problem in

• Version used (icinga2 --version): 2.11.2-1
• Operating System and version: CentOS Linux release 7.7.1908 (Core)
• Enabled features (icinga2 feature list): api checker mainlog notification
• Icinga Web 2 version and modules (System - About): 2.7.3

[dnsmichi]

Tests

docker run -ti centos:7 bash

yum -y install https://packages.icinga.com/epel/icinga-rpm-release-7-latest.noarch.rpm
yum -y install epel-release
yum -y install icinga2

ICINGA2_NODENAME=`hostname -f`
ICINGA2_SSLDIR=/tmp

icinga2 pki new-cert --cn "${ICINGA2_NODENAME}" \
--cert "${ICINGA2_SSLDIR}/${ICINGA2_NODENAME}.crt" \
--csr "${ICINGA2_SSLDIR}/${ICINGA2_NODENAME}.csr" \
--key "${ICINGA2_SSLDIR}/${ICINGA2_NODENAME}.key"

Result

information/base: Writing private key to '/tmp/73ad57409cd9.key'.
information/base: Writing X509 certificate to '/tmp/73ad57409cd9.crt'.
information/base: Writing certificate signing request to '/tmp/73ad57409cd9.csr'.

[root@73ad57409cd9 /]# ls -lahtr /tmp/ | grep 73
-rw------- 1 icinga icinga 3.2K Dec  3 13:02 73ad57409cd9.key
-rw-r--r-- 1 icinga icinga 1.8K Dec  3 13:02 73ad57409cd9.crt
-rw-r--r-- 1 icinga icinga 1.7K Dec  3 13:02 73ad57409cd9.csr

Not able to reproduce this.

Likely the script lacks sanity checks with e.g. the ICINGA2_NODENAME variable not being initialized. Or SELinux prevents creating these files.

[sharkyzz]

Here it doesn't work. Putting the variables aside:

[root@U16194 ~]# rpm -q centos-release 
centos-release-7-7.1908.0.el7.centos.x86_64
[root@U16194 ~]# icinga2 pki new-cert --cn test --key /tmp/test.key --cert /tmp/test.crt --csr /tmp/test.csr
information/base: Writing private key to '/tmp/test.key'.

[dnsmichi]

Full output of icinga2 --version and getenforce please.

[sharkyzz]

[root@U16194 ~]# icinga2 --version
icinga2 - The Icinga 2 network monitoring daemon (version: 2.11.2-1)

Copyright (c) 2012-2019 Icinga GmbH (https://icinga.com/)
License GPLv2+: GNU GPL version 2 or later <http://gnu.org/licenses/gpl2.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

System information:
  Platform: CentOS Linux
  Platform version: 7 (Core)
  Kernel: Linux
  Kernel version: 3.10.0-229.1.2.el7.centos.onapp.x86_64
  Architecture: x86_64

Build information:
  Compiler: GNU 4.8.5
  Build host: runner-LTrJQZ9N-project-322-concurrent-0

Application information:

General paths:
  Config directory: /etc/icinga2
  Data directory: /var/lib/icinga2
  Log directory: /var/log/icinga2
  Cache directory: /var/cache/icinga2
  Spool directory: /var/spool/icinga2
  Run directory: /run/icinga2

Old paths (deprecated):
  Installation root: /usr
  Sysconf directory: /etc
  Run directory (base): /run
  Local state directory: /var

Internal paths:
  Package data directory: /usr/share/icinga2
  State path: /var/lib/icinga2/icinga2.state
  Modified attributes path: /var/lib/icinga2/modified-attributes.conf
  Objects path: /var/cache/icinga2/icinga2.debug
  Vars path: /var/cache/icinga2/icinga2.vars
  PID path: /run/icinga2/icinga2.pid


[root@U16194 ~]# getenforce 
Disabled

[dnsmichi]

onapp is a specific Kernel build, is that a container/VM running on a cloud service?

Also, try re-running the script with strace to see whether it tries to actually create the file handles.

strace -f <command>

[sharkyzz]

Yes, it is a VM on a SolusVM platform.

Strace gives a ton of 'Bad file descriptor' messages

See strace output: https://www.dropbox.com/transfer/AAAAAJEIsMitz6Paoo8FgN3D4sI35_sCQJXELltuqHIuAqfV2rTpfLU

[dnsmichi]

Ah, I forgot. icinga2 is a shell wrapper for the binary. In order to see its syscalls, run the full binary path.

strace -f "/usr/lib64/icinga2/sbin/icinga2", "--no-stack-rlimit", "pki", "new-cert", "--cn", "test", "--key", "/tmp/test.key", "--cert", "/tmp/test.crt", "--csr", "/tmp/test.csr"

[sharkyzz]

I couldn't get your command above to work, so ended up with this. Hope it is a valid strace now.

strace -o /tmp/icinga_strace -f /usr/lib64/icinga2/sbin/icinga2 --no-stack-rlimit pki new-cert --cn test --key /tmp/test.key --cert /tmp/test.crt --csr /tmp/test.csr

https://www.dropbox.com/transfer/AAAAALSU4pL4tVm75yoQaUmCXwygpIUaAJszPvmwDWY0ir_TYecQFys

[dnsmichi]

I see. This is what I suspected already - getrandom() syscall is not implemented. This is used by the OpenSSL API resp. Boost UUID.

6570  chmod("/tmp/test.key", 0600 <unfinished ...>
6586  <... close resumed> )             = -1 EBADF (Bad file descriptor)
6570  <... chmod resumed> )             = 0
6586  close(5193)                       = -1 EBADF (Bad file descriptor)
6586  close(5194)                       = -1 EBADF (Bad file descriptor)
6570  getrandom( <unfinished ...>
6586  close(5195 <unfinished ...>
6570  <... getrandom resumed> 0x7fffc19c1c00, 16, 0) = -1 ENOSYS (Function not implemented)

That's actually a problem with the host system running an older Kernel (likely el6) and the guest runs with el7+. Similar issue with OpenVZ, Xen, cPanel and other VM providers, we have discussed this in #7560.

https://bugzilla.redhat.com/show_bug.cgi?id=1330000
microsoft/onnxruntime#1471
boostorg/uuid#91 (comment)

Unfortunately we cannot fix this on the application layer, this needs a kernel update on the host system providing the syscall to the guest OS.

[sharkyzz]

Thanks a lot the clarification Michael! We will upgrade the host system (indeed el6)