The minimal version of 1.0 doesn't have bridge kernel module

[SergioFalcone]

The issue below was happened when I tried to install docker within Photon 1.0. Accidentally it has not bridge package installed and I wasn't able to build kernel with corresponding options included.
Here is what I observed:

root@sc2-rdops-vm08-dhcp-26-53 [ ~ ]# tdnf provides docker
Refreshing metadata for: 'VMware Photon Linux 1.0(x86_64)Updates'
Refreshing metadata for: 'VMware Photon Extras 1.0(x86_64)'
Refreshing metadata for: 'VMware Lightwave 1.0(x86_64)'
Refreshing metadata for: 'VMware Photon Linux 1.0(x86_64)'
docker-1.11.0-5.ph1.x86_64 : Docker
Repo     : photon
docker-17.06.0-3.ph1.x86_64 : Docker
Repo     : photon-updates
docker-17.06.0-2.ph1.x86_64 : Docker
Repo     : photon-updates
docker-17.06.0-1.ph1.x86_64 : Docker
Repo     : photon-updates
docker-1.13.1-4.ph1.x86_64 : Docker
Repo     : photon-updates
docker-1.13.1-3.ph1.x86_64 : Docker
Repo     : photon-updates
docker-1.13.1-1.ph1.x86_64 : Docker
Repo     : photon-updates
docker-1.12.6-1.ph1.x86_64 : Docker
Repo     : photon-updates
docker-1.12.1-1.ph1.x86_64 : Docker
Repo     : photon-updates
docker-1.11.2-1.ph1.x86_64 : Docker
Repo     : photon-updates
root@sc2-rdops-vm08-dhcp-26-53 [ ~ ]# tdnf install docker

Installing:
libltdl                                                                                       x86_64                             2.4.6-2.ph1                                                                37.82 k
libseccomp                                                                                    x86_64                             2.2.3-2.ph1                                                               453.62 k
docker                                                                                        x86_64                             17.06.0-3.ph1                                                             110.28 M

Total installed size: 110.76 M
Is this ok [y/N]:y

Downloading:
docker                                33070896    100%
libseccomp                              102057    100%
libltdl                                  20367    100%
Testing transaction
Running transaction

Complete!
root@sc2-rdops-vm08-dhcp-26-53 [ ~ ]# 
root@sc2-rdops-vm08-dhcp-26-53 [ ~ ]# 
root@sc2-rdops-vm08-dhcp-26-53 [ ~ ]# which docker
/usr/bin/docker
root@sc2-rdops-vm08-dhcp-26-53 [ ~ ]# systemctl enable docker
Created symlink from /etc/systemd/system/multi-user.target.wants/docker.service to /usr/lib/systemd/system/docker.service.
root@sc2-rdops-vm08-dhcp-26-53 [ ~ ]# systemctl start docker
Job for docker.service failed because the control process exited with error code. See "systemctl status docker.service" and "journalctl -xe" for details.
root@sc2-rdops-vm08-dhcp-26-53 [ ~ ]# 
root@sc2-rdops-vm08-dhcp-26-53 [ ~ ]# 
root@sc2-rdops-vm08-dhcp-26-53 [ ~ ]# 
root@sc2-rdops-vm08-dhcp-26-53 [ ~ ]# journalctl -u docker
-- Logs begin at Tue 2018-03-27 13:54:08 UTC, end at Wed 2018-04-04 15:08:10 UTC. --
Apr 04 15:08:06 sc2-rdops-vm08-dhcp-26-53.eng.vmware.com systemd[1]: Starting Docker Application Container Engine...
Apr 04 15:08:06 sc2-rdops-vm08-dhcp-26-53.eng.vmware.com dockerd[14654]: time="2018-04-04T15:08:06.246404758Z" level=info msg="libcontainerd: new containerd process, pid: 14681"
Apr 04 15:08:07 sc2-rdops-vm08-dhcp-26-53.eng.vmware.com dockerd[14654]: time="2018-04-04T15:08:07.250631394Z" level=warning msg="failed to rename /var/lib/docker/tmp for background deletion: rename /var/lib/docker/tmp /var/lib/docker/t
Apr 04 15:08:07 sc2-rdops-vm08-dhcp-26-53.eng.vmware.com dockerd[14654]: time="2018-04-04T15:08:07.251599981Z" level=error msg="Failed to built-in GetDriver graph aufs /var/lib/docker"
Apr 04 15:08:07 sc2-rdops-vm08-dhcp-26-53.eng.vmware.com dockerd[14654]: time="2018-04-04T15:08:07.338472666Z" level=info msg="Graph migration to content-addressability took 0.00 seconds"
Apr 04 15:08:07 sc2-rdops-vm08-dhcp-26-53.eng.vmware.com dockerd[14654]: time="2018-04-04T15:08:07.338814932Z" level=warning msg="Your kernel does not support cgroup rt period"
Apr 04 15:08:07 sc2-rdops-vm08-dhcp-26-53.eng.vmware.com dockerd[14654]: time="2018-04-04T15:08:07.338833512Z" level=warning msg="Your kernel does not support cgroup rt runtime"
Apr 04 15:08:07 sc2-rdops-vm08-dhcp-26-53.eng.vmware.com dockerd[14654]: time="2018-04-04T15:08:07.339272118Z" level=info msg="Loading containers: start."
Apr 04 15:08:07 sc2-rdops-vm08-dhcp-26-53.eng.vmware.com dockerd[14654]: time="2018-04-04T15:08:07.376578682Z" level=warning msg="Running modprobe bridge br_netfilter failed with message: modprobe: ERROR: Error running install command f
Apr 04 15:08:07 sc2-rdops-vm08-dhcp-26-53.eng.vmware.com dockerd[14654]: time="2018-04-04T15:08:07.547900609Z" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set
Apr 04 15:08:07 sc2-rdops-vm08-dhcp-26-53.eng.vmware.com dockerd[14654]: Error starting daemon: Error initializing network controller: Error creating default "bridge" network: package not installed
Apr 04 15:08:07 sc2-rdops-vm08-dhcp-26-53.eng.vmware.com systemd[1]: docker.service: Main process exited, code=exited, status=1/FAILURE
Apr 04 15:08:07 sc2-rdops-vm08-dhcp-26-53.eng.vmware.com systemd[1]: Failed to start Docker Application Container Engine.
Apr 04 15:08:07 sc2-rdops-vm08-dhcp-26-53.eng.vmware.com systemd[1]: docker.service: Unit entered failed state.
Apr 04 15:08:07 sc2-rdops-vm08-dhcp-26-53.eng.vmware.com systemd[1]: docker.service: Failed with result 'exit-code'.
Apr 04 15:08:07 sc2-rdops-vm08-dhcp-26-53.eng.vmware.com systemd[1]: docker.service: Service hold-off time over, scheduling restart.
Apr 04 15:08:07 sc2-rdops-vm08-dhcp-26-53.eng.vmware.com systemd[1]: Stopped Docker Application Container Engine.
Apr 04 15:08:07 sc2-rdops-vm08-dhcp-26-53.eng.vmware.com systemd[1]: Starting Docker Application Container Engine...
Apr 04 15:08:07 sc2-rdops-vm08-dhcp-26-53.eng.vmware.com dockerd[14777]: time="2018-04-04T15:08:07.953021094Z" level=info msg="libcontainerd: new containerd process, pid: 14785"
Apr 04 15:08:08 sc2-rdops-vm08-dhcp-26-53.eng.vmware.com dockerd[14777]: time="2018-04-04T15:08:08.957706096Z" level=info msg="[graphdriver] using prior storage driver: overlay2"
Apr 04 15:08:08 sc2-rdops-vm08-dhcp-26-53.eng.vmware.com dockerd[14777]: time="2018-04-04T15:08:08.960028932Z" level=info msg="Graph migration to content-addressability took 0.00 seconds"
Apr 04 15:08:08 sc2-rdops-vm08-dhcp-26-53.eng.vmware.com dockerd[14777]: time="2018-04-04T15:08:08.960281058Z" level=warning msg="Your kernel does not support cgroup rt period"
Apr 04 15:08:08 sc2-rdops-vm08-dhcp-26-53.eng.vmware.com dockerd[14777]: time="2018-04-04T15:08:08.960302818Z" level=warning msg="Your kernel does not support cgroup rt runtime"
Apr 04 15:08:08 sc2-rdops-vm08-dhcp-26-53.eng.vmware.com dockerd[14777]: time="2018-04-04T15:08:08.961067903Z" level=info msg="Loading containers: start."
Apr 04 15:08:08 sc2-rdops-vm08-dhcp-26-53.eng.vmware.com dockerd[14777]: time="2018-04-04T15:08:08.971250583Z" level=warning msg="Running modprobe bridge br_netfilter failed with message: modprobe: ERROR: Error running install command f
Apr 04 15:08:09 sc2-rdops-vm08-dhcp-26-53.eng.vmware.com dockerd[14777]: time="2018-04-04T15:08:09.025868948Z" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set
Apr 04 15:08:09 sc2-rdops-vm08-dhcp-26-53.eng.vmware.com dockerd[14777]: Error starting daemon: Error initializing network controller: Error creating default "bridge" network: package not installed
Apr 04 15:08:09 sc2-rdops-vm08-dhcp-26-53.eng.vmware.com systemd[1]: docker.service: Main process exited, code=exited, status=1/FAILURE
Apr 04 15:08:09 sc2-rdops-vm08-dhcp-26-53.eng.vmware.com systemd[1]: Failed to start Docker Application Container Engine.
Apr 04 15:08:09 sc2-rdops-vm08-dhcp-26-53.eng.vmware.com systemd[1]: docker.service: Unit entered failed state.
Apr 04 15:08:09 sc2-rdops-vm08-dhcp-26-53.eng.vmware.com systemd[1]: docker.service: Failed with result 'exit-code'.
Apr 04 15:08:09 sc2-rdops-vm08-dhcp-26-53.eng.vmware.com systemd[1]: docker.service: Service hold-off time over, scheduling restart.
Apr 04 15:08:09 sc2-rdops-vm08-dhcp-26-53.eng.vmware.com systemd[1]: Stopped Docker Application Container Engine.
Apr 04 15:08:09 sc2-rdops-vm08-dhcp-26-53.eng.vmware.com systemd[1]: Starting Docker Application Container Engine...
Apr 04 15:08:09 sc2-rdops-vm08-dhcp-26-53.eng.vmware.com dockerd[14843]: time="2018-04-04T15:08:09.201735513Z" level=info msg="libcontainerd: new containerd process, pid: 14851"
Apr 04 15:08:10 sc2-rdops-vm08-dhcp-26-53.eng.vmware.com dockerd[14843]: time="2018-04-04T15:08:10.206549729Z" level=info msg="[graphdriver] using prior storage driver: overlay2"
Apr 04 15:08:10 sc2-rdops-vm08-dhcp-26-53.eng.vmware.com dockerd[14843]: time="2018-04-04T15:08:10.209283700Z" level=info msg="Graph migration to content-addressability took 0.00 seconds"
Apr 04 15:08:10 sc2-rdops-vm08-dhcp-26-53.eng.vmware.com dockerd[14843]: time="2018-04-04T15:08:10.209668311Z" level=warning msg="Your kernel does not support cgroup rt period"
Apr 04 15:08:10 sc2-rdops-vm08-dhcp-26-53.eng.vmware.com dockerd[14843]: time="2018-04-04T15:08:10.209687194Z" level=warning msg="Your kernel does not support cgroup rt runtime"
Apr 04 15:08:10 sc2-rdops-vm08-dhcp-26-53.eng.vmware.com dockerd[14843]: time="2018-04-04T15:08:10.210066331Z" level=info msg="Loading containers: start."
Apr 04 15:08:10 sc2-rdops-vm08-dhcp-26-53.eng.vmware.com dockerd[14843]: time="2018-04-04T15:08:10.221547258Z" level=warning msg="Running modprobe bridge br_netfilter failed with message: modprobe: ERROR: Error running install command f
Apr 04 15:08:10 sc2-rdops-vm08-dhcp-26-53.eng.vmware.com dockerd[14843]: time="2018-04-04T15:08:10.282571030Z" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set
Apr 04 15:08:10 sc2-rdops-vm08-dhcp-26-53.eng.vmware.com dockerd[14843]: Error starting daemon: Error initializing network controller: **Error creating default "bridge" network: package not installed**
Apr 04 15:08:10 sc2-rdops-vm08-dhcp-26-53.eng.vmware.com systemd[1]: docker.service: Main process exited, code=exited, status=1/FAILURE
Apr 04 15:08:10 sc2-rdops-vm08-dhcp-26-53.eng.vmware.com systemd[1]: Failed to start Docker Application Container Engine.
Apr 04 15:08:10 sc2-rdops-vm08-dhcp-26-53.eng.vmware.com systemd[1]: docker.service: Unit entered failed state.
Apr 04 15:08:10 sc2-rdops-vm08-dhcp-26-53.eng.vmware.com systemd[1]: docker.service: Failed with result 'exit-code'.
Apr 04 15:08:10 sc2-rdops-vm08-dhcp-26-53.eng.vmware.com systemd[1]: docker.service: Service hold-off time over, scheduling restart.
Apr 04 15:08:10 sc2-rdops-vm08-dhcp-26-53.eng.vmware.com systemd[1]: Stopped Docker Application Container Engine.
Apr 04 15:08:10 sc2-rdops-vm08-dhcp-26-53.eng.vmware.com systemd[1]: docker.service: Start request repeated too quickly.
Apr 04 15:08:10 sc2-rdops-vm08-dhcp-26-53.eng.vmware.com systemd[1]: Failed to start Docker Application Container Engine.
Apr 04 15:08:10 sc2-rdops-vm08-dhcp-26-53.eng.vmware.com systemd[1]: docker.service: Unit entered failed state.
Apr 04 15:08:10 sc2-rdops-vm08-dhcp-26-53.eng.vmware.com systemd[1]: docker.service: Failed with result 'start-limit'.

And on a full version it's ok. Here is the lsmod on the same 1.0 Full.

root@photon-os1-full [ ~ ]# lsmod | grep bridge
bridge                118784  1 br_netfilter
stp                    16384  1 bridge
llc                    16384  2 stp,bridge

Could you please help me to correctly bootstrap system with this module included. Assume the system is already exists and could not be bootstrapped from zero state.

If it's not possible then please provide the correct image to perform all our setup from scratch.

Thanks,
Sergio

[iNode]

Can you try to execute insmod br_netfilter manually an show full error if it fails and then show output of command `dmesg | grep br_netfilter:' as well?

[SergioFalcone]

insmod br_netfilter
insmod: ERROR: could not load module br_netfilter: No such file or directory

lsmod | grep bridge on min OS shows zero results

Thanks

[SergioFalcone]

As I understand in some of configurations like this

https://github.com/vmware/photon/blob/743bd2598788918cd1b65c5bcf9796cd41925662/SPECS/linux/config

, the CONFIG_BRIDGE_NETFILTER=m wasn't correctly set

Is there a chance to smoothly upgrade kernel in-place? Please, provide steps would be helpful!

[srivatsabhat]

I tried this locally with a fresh install of Photon OS 1.0 minimal from ISO, but was not able to reproduce the issue.

Did you try running modprobe br_netfilter and see if the module loads (lsmod | grep br_netfilter)?

If that doesn't work, can you tell us which kernel flavor and version you are running? Please provide the output of uname -a

[SergioFalcone]

modprobe br_netfilter
modprobe: ERROR: Error running install command for bridge
modprobe: ERROR: could not insert 'br_netfilter': Operation not permitted

uname -a
Linux localhost.localdom 4.4.115-2.ph1 #1-photon SMP Wed Mar 14 12:19:21 UTC 2018 x86_64 GNU/Linux

find / -name "*.xz" | grep br_netfilter
/usr/lib/modules/4.4.115-2.ph1/kernel/net/bridge/br_netfilter.ko.xz

The package exists but seems corrupted or of inconsistent version.

Please, note that: "System was build-up with a custom tool"

[srivatsabhat]

Wait, what do you mean by your comment "System was build-up with a custom tool" ?

Do you see this issue on a minimal install from a Photon OS 1.0-rev2 ISO (perhaps after running tdnf -y distro-sync to get the most recent versions of the installed packages)?

[SergioFalcone]

The system base is as reported - just I've got it with pre-installed OVA, so seems the providing team runs their vahardening scripts, which are not configured to include bridge modules. I will update you as the issue will be gone.

Update:

In vahardening scripts there’s a script that has to be overridden for Docker based environments. The addition of “GEN003619” to OverriddenPDIs, fixed the situation.