Kernel Devel error when building Cortx in docker

[stx-hseow]

Attempt 1

Following this guide, I wanted to build Cortx using the cortx-build image. However, I encounter the following error:

...
 --> Already installed : lustre-client-devel-2.12.5-1.el7.x86_64
 --> Already installed : libuuid-devel-2.23.2-63.el7.x86_64
 --> Already installed : binutils-devel-2.27-43.base.el7.x86_64
 --> Already installed : python36-ply-3.9-2.el7.noarch
 --> Already installed : perl-autodie-2.16-2.el7.noarch
 --> Already installed : systemd-devel-219-73.el7.1.x86_64
 --> Already installed : python-devel-2.7.5-88.el7.x86_64
 --> Already installed : libedit-devel-3.0-12.20121213cvs.el7.x86_64
Error: No Package found for kernel-devel = 3.10.0-1062.el7
ERROR: cortx-motr source code is not available in cortx-workspace. Exiting...
make: *** [_cortx-motr_prereq] Error 1

Attempt 2

So I thought perhaps it's missing the kernel devel, and I opened an interactive shell
docker run --rm -v /var/artifacts:/var/artifacts -v /root/cortx:/cortx-workspace -it ghcr.io/seagate/cortx-build:centos-7.8.2003
Manually install a kernel devel before building

yum install -y https://buildlogs.centos.org/c7.1908.00.x86_64/kernel/20190808101829/3.10.0-1062.el7.x86_64/kernel-devel-3.10.0-1062.el7.x86_64.rpm
make clean build

It went past the initial stage, but then failed again (error_output.txt) at a later stage

...
checking whether malloc_size is declared... no
checking whether backtrace is declared... yes
checking systemd/sd-daemon.h usability... yes
checking systemd/sd-daemon.h presence... yes
checking for systemd/sd-daemon.h... yes
checking whether sd_notify is declared... yes
checking for library containing sd_notify... -lsystemd-daemon
checking for symbols file... configure: error: Failed to find symbols files in /lib/modules/3.10.0-1062.el7.x86_64/build
error: Bad exit status from /var/tmp/rpm-tmp.HxRlS2 (%build)


RPM build errors:
    Bad exit status from /var/tmp/rpm-tmp.HxRlS2 (%build)
make[2]: *** [__rpm_tests] Error 1
make[1]: *** [rpms] Error 2
make: *** [_cortx-motr_build] Error 2

Script to reproduce

yum install -y http://opensource.wandisco.com/centos/7/git/x86_64/wandisco-git-release-7-2.noarch.rpm
yum install -y git
yum install -y yum-utils
yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
yum install -y docker-ce docker-ce-cli containerd.io

cd /root
mkdir /var/artifacts
git clone --depth 1 --recursive --shallow-submodules https://github.com/Seagate/cortx
systemctl start docker
docker run --rm -v /var/artifacts:/var/artifacts -v /root/cortx:/cortx-workspace ghcr.io/seagate/cortx-build:centos-7.8.2003 make clean build

Using VMWare Workstation 16 Player (Windows 10) running CentOS 7.7.1908

[mukul-seagate11]

@shailesh-vaidya, please address the issue as reported?

[shailesh-vaidya]

@harrison-seow-seagate Thanks for trying out Docker image. To debug issue can you please post the output of following commands from the system on which you are executing docker run command.

cat /etc/redhat-release
uname -r
yum list kernel

[stx-hseow]

Hi @shailesh-vaidya, please refer the output below, thanks.

[harrison@localhost ~]$ cat /etc/redhat-release
CentOS Linux release 7.7.1908 (Core)
[harrison@localhost ~]$ uname -r
3.10.0-1062.el7.x86_64
[harrison@localhost ~]$ yum list kernel
Loaded plugins: fastestmirror, langpacks
Determining fastest mirrors
 * epel: hkg.mirror.rackspace.com
Installed Packages
kernel.x86_64                      3.10.0-1062.el7                            @anaconda
Available Packages
kernel.x86_64                      3.10.0-1160.21.1.el7                       updates  
[harrison@localhost ~]$ 

[shailesh-vaidya]

@harrison-seow-seagate Thanks for the command output's. Seems you are trying to generate Build on CentOS 7.7.1908 system. We need to use CentOS 7.8.2003 system to build CORTX. The same is documented as Step#1 of Procedure

You can use image TAG to identify OS Version for it. e.g. ghcr.io/seagate/cortx-build:centos-7.8.2003. We will be releasing subsequent OS version Docker Images.

Please try Build process on CentOS 7.8.2003 and let me know if any issues.

[stx-hseow]

I see, thanks @shailesh-vaidya, I will try it out again in a newer version.

[stale]

This issue/pull request has been marked as needs attention as it has been left pending without new activity for 4 days. Tagging @mukul-seagate11 for appropriate assignment. Sorry for the delay & Thank you for contributing to CORTX. We will get back to you as soon as possible.

[shailesh-vaidya]

@harrison-seow-seagate Please let us know if you are able to build CORTX packages on CentOS 7.8.2003 system. Let me know if any help needed.

[stx-hseow]

Hi @shailesh-vaidya, I have a rather slow Internet speed, so it will take me some time to download the new OS and rebuild Cortx.
Thanks for asking, I will post it up if I face any issue.

[stx-hseow]

Hi @shailesh-vaidya , I have just tested it out today and although the output is different but I'm still seeing a very similar RPM build error.

Output Snippet

Ran 9 tests in 19.150s

OK
make[4]: Leaving directory `/root/rpmbuild/BUILD/cortx-hare'
make[3]: Leaving directory `/root/rpmbuild/BUILD/cortx-hare'
make[3]: *** [build] Error 2


RPM build errors:
error: Bad exit status from /var/tmp/rpm-tmp.sgPZc0 (%build)
    Bad exit status from /var/tmp/rpm-tmp.sgPZc0 (%build)
make[2]: Leaving directory `/cortx-workspace/cortx-hare'
make[2]: *** [__rpm] Error 1
make[1]: *** [rpm] Error 2
make[1]: Leaving directory `/cortx-workspace/cortx-hare'
make: *** [_cortx-hare_build] Error 2

Machine Info

[root@localhost ~]# cat /etc/redhat-release
CentOS Linux release 7.8.2003 (Core)
[root@localhost ~]# uname -r
3.10.0-1127.el7.x86_64
[root@localhost ~]# yum list kernel
Loaded plugins: fastestmirror, langpacks
Loading mirror speeds from cached hostfile
 * base: centos.netonboard.com
 * extras: centos.netonboard.com
 * updates: centos.netonboard.com
Installed Packages
kernel.x86_64                                                 3.10.0-1127.el7                                                      @anaconda
Available Packages
kernel.x86_64                                                 3.10.0-1160.21.1.el7                                                 updates  
[root@localhost ~]# 

Script to Reproduce

su

yum install -y http://opensource.wandisco.com/centos/7/git/x86_64/wandisco-git-release-7-2.noarch.rpm
yum install -y git

yum install -y yum-utils
yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
yum install -y docker-ce docker-ce-cli containerd.io

cd /root
mkdir /var/artifacts
git clone --depth 1 --recursive --shallow-submodules https://github.com/Seagate/cortx
systemctl start docker
docker run --rm -v /var/artifacts:/var/artifacts -v /root/cortx:/cortx-workspace ghcr.io/seagate/cortx-build:centos-7.8.2003 make clean build

[shailesh-vaidya]

Hi @harrison-seow-seagate - Hare Build was failing due to a new dependency added. It's fixed now. Please pull the latest image and try to build again.
You can use below commands,

docker pull ghcr.io/seagate/cortx-build:centos-7.8.2003
docker run --rm -v /var/artifacts:/var/artifacts -v /root/cortx:/cortx-workspace ghcr.io/seagate/cortx-build:centos-7.8.2003 make clean build

[stx-hseow]

Thanks @shailesh-vaidya, I pulled the latest image and it worked. I'll close this issue as it's resolved.