BIGTOP-3356: Add CentOS 8 support to the Docker provisioner

[sekikn]

No description provided.

[iwasakims]

@sekikn I got the following error on provisioning with the patch and enable_local_repo = true. yum-config-manager seems to be replaced by yum config-manager on CentOS 8.

sudo: yum-config-manager: command not found

[iwasakims]

After I fixed the yum-config-manager part, I got the another error. Installing rng-tools seemed to be failed.

Failed to download metadata for repo 'AppStream'
Error: Failed to download metadata for repo 'AppStream'
sed: can't read /usr/lib/systemd/system/rngd.service: No such file or directory
Failed to start rngd.service: Unit rngd.service not found.

[iwasakims]

@sekikn Sorry, the cause of the error was that I did not update docker image by build.sh.

[iwasakims]

Hmm... starting services by systemctl does not work in docker container. Since it works outside container, I'm suspecting issue like docker/for-linux#835.

[iwasakims]

adding /sys/fs/cgroup to volumes worked for me to fix the systemctl issue.

diff --git a/provisioner/docker/docker-compose.yml b/provisioner/docker/docker-compose.yml
index cee849e9..b3bb1a42 100644
--- a/provisioner/docker/docker-compose.yml
+++ b/provisioner/docker/docker-compose.yml
@@ -24,3 +24,4 @@ bigtop:
     - ./config/hiera.yaml:/etc/puppet/hiera.yaml
     - ./config/hieradata:/etc/puppet/hieradata
     - ./config/hosts:/etc/hosts
+    - /sys/fs/cgroup:/sys/fs/cgroup:ro

[sekikn]

Great, will try it locally. Thanks!

[evans-ye]

adding /sys/fs/cgroup to volumes worked for me to fix the systemctl issue.

diff --git a/provisioner/docker/docker-compose.yml b/provisioner/docker/docker-compose.yml
index cee849e9..b3bb1a42 100644
--- a/provisioner/docker/docker-compose.yml
+++ b/provisioner/docker/docker-compose.yml
@@ -24,3 +24,4 @@ bigtop:
     - ./config/hiera.yaml:/etc/puppet/hiera.yaml
     - ./config/hieradata:/etc/puppet/hieradata
     - ./config/hosts:/etc/hosts
+    - /sys/fs/cgroup:/sys/fs/cgroup:ro

@iwasakims Can you share some technical reference to this. Thanks!

[iwasakims]

Can you share some technical reference to this.

@evans-ye The background is similar to BIGTOP-3302. systemd added check to see if service process belongs to the service as a fix for CVE-2018-16888. The check uses cgroups.

• RFE: stricter checks on the PID read from PIDFile= or sd_notify()'s MAINPID= systemd/systemd#6632
• Make MAINPID= and PIDFile= handling more restrictive (and other stuff) systemd/systemd#7816

If cgroups info is not available, init script fails with following message, indicating failure of checking added by the systemd/systemd#6632.

May 21 11:53:07 2eef6a902086 zookeeper-server[2620]: Starting zookeeper ... STARTED
May 21 11:53:07 2eef6a902086 runuser[2665]: pam_unix(runuser:session): session closed for user zookeeper
May 21 11:53:07 2eef6a902086 systemd[1]: zookeeper-server.service: New main PID 2714 does not belong to service, and PID file is not owned by root. Refusing.

[sekikn]

Thanks @iwasakims, I also confirmed your fix worked well. I updated the PR.

[iwasakims]

+1. ./docker-hadoop.sh -c 1 --smoke-tests worked on my local with following config.yaml (and fix of #642 for hadoop installation).

docker:
        memory_limit: "8g"
        image: "bigtop/puppet:trunk-centos-8"

distro: centos
components: [hdfs, yarn, mapreduce]
enable_local_repo: true
smoke_test_components: [hdfs, yarn, mapreduce]

Provisioning bigtop/puppet:trunk-centos-7 on CentOS 7 worked as before with this patch.

[sekikn]

Thanks! Merged into master.

[evans-ye]

The investigation is informative. Thanks @iwasakims.