Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

F33: CLHM continuously loops on boot causing system to not come up #631

Closed
dustymabe opened this issue Sep 24, 2020 · 6 comments
Closed

F33: CLHM continuously loops on boot causing system to not come up #631

dustymabe opened this issue Sep 24, 2020 · 6 comments

Comments

@dustymabe
Copy link
Member

This is part of the Fedora 33 enablement work in #611.

For some reason console-login-helper-messages infinitely loops on boot and the system won't come up. Here is a snippet from the serial console:

[  OK  ] Finished Rebuild Journal Catalog.                                                                                                                                                                                                   
[  OK  ] Finished Update UTMP about System Boot/Shutdown.                                                                                                                                                                                    
         Starting Update is Completed...                                                                                                                                                                                                     
[    6.515449] 9p: Installing v9fs 9p2000 file system support                                                                                                                                                                                
[    6.517487] FS-Cache: Netfs '9p' registered for caching                                                                                                                                                                                   
[  OK  ] Finished Update is Completed.                                                                                                                                                                                                       
[  OK  ] Reached target System Initialization.                                                                                                                                                                                               
[  OK  ] Started Monitor console-lo…nippets directory for changes.                                                                                                                                                                           
[  OK  ] Started Monitor console-lo…nippets directory for changes.                                                                                                                                                                           
[  OK  ] Started OSTree Monitor Staged Deployment.                                                                                                                                                                                           
[  OK  ] Started Daily rotation of log files.                                                                                                                                                                                                
[  OK  ] Started Weekly RAID setup health check.                                                                      
[  OK  ] Started Daily Cleanup of Temporary Directories.                                                              
[  OK  ] Reached target Boot Completion Check.                                                                        
[  OK  ] Reached target Paths.                                                                                        
[  OK  ] Reached target Timers.                                                                                       
[  OK  ] Listening on D-Bus System Message Bus Socket.                                                                
         Starting Docker Socket for the API.                                                                          
[  OK  ] Listening on Open-iSCSI iscsid Socket.                                                                       
[  OK  ] Listening on Open-iSCSI iscsiuio Socket.                                                                     
[  OK  ] Listening on Docker Socket for the API.                                                                      
[  OK  ] Reached target Sockets.                                                                                      
[    6.544193] FS-Cache: Duplicate cookie detected                                                                    
[    6.545293] FS-Cache: O-cookie c=0000000040480be1 [p=00000000b652f2d5 fl=222 nc=0 na=1]
[    6.546603] FS-Cache: O-cookie d=0000000097b77a4a n=000000008d181e25                                               
[    6.547690] FS-Cache: O-key=[10] '34323934363733373333'                                                            
[  OK  ] Mounted /var/mnt/workdir.                                                                                    
[    6.548854] FS-Cache: N-cookie c=00000000daba4414 [p=00000000b652f2d5 fl=2 nc=0 na=1]
[    6.550290] FS-Cache: N-cookie d=0000000097b77a4a n=00000000b76466d0                                               
[    6.551486] FS-Cache: N-key=[10] '34323934363733373333'                                                            
[  OK  ] Mounted /var/mnt/workdir-tmp.                                                                                
[  OK  ] Reached target Basic System.                                                                                 
         Starting Network Manager...                                                                                  
[  OK  ] Reached target Synchronize…s@.service template instances.                                        
         Starting NTP client/server...                                                                                
         Starting Generate os relea…nsole-login-helper-messages...             
         Starting Check if Ignition config is provided...                                                             
         Starting Check that ssh-ke…added by Afterburn/Ignition...                                                    
         Starting Mark boot complete...                                                                               
[  OK  ] Started irqbalance daemon.                                                                                   
[    6.581978] EXT4-fs (vda1): re-mounted. Opts:                                                                      
[    6.583500] ext4 filesystem being remounted at /boot supports timestamps until 2038 (0x7fffffff)
         Starting OpenSSH ecdsa Server Key Generation...                                                              
         Starting OpenSSH ed25519 Server Key Generation...                                                            
         Starting OpenSSH rsa Server Key Generation...                                                                
         Starting System Security Services Daemon...                                                                  
         Starting Home Area Manager...                                                                                
[  OK  ] Started NTP client/server.                                                                                   
[  OK  ] Started Network Name Resolution.                                                                             
[  OK  ] Finished Generate os relea…console-login-helper-messages.             
[  OK  ] Finished Mark boot complete.                                                                                 
[  OK  ] Finished OpenSSH ecdsa Server Key Generation.                                                                
[  OK  ] Finished OpenSSH ed25519 Server Key Generation.                                                              
[  OK  ] Reached target Host and Network Name Lookups.                                                                
         Starting Generate console-…elper-messages motd snippet...                                                    
         Starting D-Bus System Message Bus...                                                                         
[  OK  ] Finished Check if Ignition config is provided.                                                               
[  OK  ] Finished OpenSSH rsa Server Key Generation.                                                                  
[  OK  ] Finished Generate console-…-helper-messages motd snippet.                                                    
[  OK  ] Reached target sshd-keygen.target.                                                                           
         Starting Generate SSH keys…nsole-login-helper-messages...                                                    
         Starting Generate console-…lper-messages issue snippet...             
         Starting Generate console-…elper-messages motd snippet...                                                    
[  OK  ] Finished Check that ssh-ke…e added by Afterburn/Ignition.             
[  OK  ] Finished Generate console-…-helper-messages motd snippet.                                                    
         Starting Generate console-…elper-messages motd snippet...             
[  OK  ] Finished Generate console-…helper-messages issue snippet.                                                    
         Starting Generate console-…lper-messages issue snippet...             
[  OK  ] Finished Generate console-…-helper-messages motd snippet.                                                    
[  OK  ] Started D-Bus System Message Bus.                                                                            
[  OK  ] Finished Generate SSH keys…console-login-helper-messages.                                                    
         Starting Generate console-…elper-messages motd snippet...                                                    
[  OK  ] Started Home Area Manager.                                                                                   
[  OK  ] Finished Generate console-…helper-messages issue snippet.
[  OK  ] Finished Generate console-…-helper-messages motd snippet.
[  OK  ] Started Network Manager.                                                                                     
[  OK  ] Reached target Network.                                                                                      
         Starting Network Manager Wait Online...                                                                      
         Starting Generate console-…lper-messages issue snippet...
         Starting Generate console-…elper-messages motd snippet...
         Starting Logout off all iSCSI sessions on shutdown...
         Starting OpenSSH server daemon...                                                                            
[  OK  ] Started Zincati Update Agent.                                                                                
[  OK  ] Started System Security Services Daemon.                                                                     
[  OK  ] Finished Logout off all iSCSI sessions on shutdown.
[  OK  ] Reached target User and Group Name Lookups.                                                                  
         Starting Hostname Service...                                                                                 
         Starting User Login Management...                                                                            
[  OK  ] Started OpenSSH server daemon.                                                                               
[  OK  ] Finished Generate console-…-helper-messages motd snippet.
[FAILED] Failed to start Generate c…-helper-messages motd snippet.
See 'systemctl status console-login-hel…-messages-motdgen.service' for details.
[FAILED] Failed to start Generate c…-helper-messages motd snippet.
See 'systemctl status console-login-hel…-messages-motdgen.service' for details.
[FAILED] Failed to start Generate c…-helper-messages motd snippet.
See 'systemctl status console-login-hel…-messages-motdgen.service' for details.
[FAILED] Failed to start Generate c…-helper-messages motd snippet.
See 'systemctl status console-login-hel…-messages-motdgen.service' for details.
[FAILED] Failed to start Generate c…-helper-messages motd snippet.
See 'systemctl status console-login-hel…-messages-motdgen.service' for details.
[FAILED] Failed to start Generate c…-helper-messages motd snippet.
See 'systemctl status console-login-hel…-messages-motdgen.service' for details.
[FAILED] Failed to start Generate c…-helper-messages motd snippet.

If I remove the clhm packages from the compose then the system boots fine. I'm not sure what the issue is.

The version of the packages is console-login-helper-messages-0.19-2.fc33.noarch.
dustymabe added a commit to dustymabe/fedora-coreos-config that referenced this issue Sep 24, 2020
@dustymabe
manifests: fedora-coreos-base: disable CLHM for now
17584d2 
console-login-helper-messages is causing a boot loop in Fedora 33.
See coreos/fedora-coreos-tracker#631.

Let's disable for now.
dustymabe added a commit to dustymabe/fedora-coreos-config that referenced this issue Sep 24, 2020
@dustymabe
manifests: fedora-coreos-base: disable CLHM for now
f4bbd88 
console-login-helper-messages is causing a boot loop in Fedora 33.
See coreos/fedora-coreos-tracker#631.

Let's disable for now.
@kelvinfan001
Copy link
Member

kelvinfan001 commented Sep 24, 2020 •
edited
Loading

Looks like PathExistsGlob= in the .path units is what's causing the infinite looping. https://github.com/coreos/console-login-helper-messages/blob/c2730ddf206cd779549a9ff443974b01d5db7d88/usr/lib/systemd/system/console-login-helper-messages-issuegen.path#L12, for example.
Trying to figure out if we were using it incorrectly, but not sure why this was not an issue in f32, since we are using it in the exact same way.

@dustymabe
Copy link
Member Author

I would maybe look at the release notes for the past few systemd releases to see if there is any mention of behavior changing there.

@kelvinfan001
Copy link
Member

Ah okay. So this commit systemd/systemd@708961c, which is included in systemd v246, changed the behaviour of PathExistsGlob. We probably need a different solution to implement the behaviour in https://github.com/coreos/console-login-helper-messages/blob/c2730ddf206cd779549a9ff443974b01d5db7d88/usr/lib/systemd/system/console-login-helper-messages-issuegen.path#L10-L11.

@kelvinfan001
Copy link
Member

kelvinfan001 commented Sep 25, 2020 •
edited
Loading

PathExistsGlob was there to achieve the following:
"Before, with PathChanged= only, the issuegen/motdgen service
would only be activated if a file had been dropped into the
runtime snippet directory after the .path unit had started,
requiring other services generating snippets to do e.g.
After=console-login-helper-messages-issuegen.path. Now, this
is not required and no explicit dependency on the issuegen/motdgen
units need be set."
Since we can no longer use PathExistsGlob for this purpose, there are two ways to address this:

  1. Add the issuegen.service and motdgen.service presets back into FCOS config and specfile so the source directories are checked at least once at boot up. Additionally, we would need to add an After=console-login-helper-messages-{issuegen, motdgen}.path to the .service units. This way, if snippets were dropped into the source directories before .path units are started, they would still be concatenated into a final snippet by console-login-helper-messages-{issuegen,motdgen}.service.
  2. Require other services generating snippets to do e.g. After=console-login-helper-messages-issuegen.path again. We may need to update a variety of files, including https://github.com/coreos/fedora-coreos-config/blob/e0f8d5afd0a7db838db688a73fcf4cf21ddfc710/overlay.d/15fcos/usr/lib/systemd/system/coreos-check-ignition-config.service

I think Option 1 is more suitable because this would allow for users to drop snippets in the host config directory, and those snippets in the host config directory will be generated into a final concatenated issue/motd. Furthermore, with util-linux 2.36+, generating snippets from the host config directory will be the only purpose of the .service units, since issue/motd snippets generated at runtime will go directly to the public /run directories and the rest of the logic will be handled by util-linux.

@kelvinfan001 kelvinfan001 mentioned this issue Sep 25, 2020
Merged
kelvinfan001 added a commit to kelvinfan001/fedora-coreos-config that referenced this issue Sep 25, 2020
@kelvinfan001
overlay.d/05core: Update CLHM presets
6e77341 
Add (back) preset for CLHM-{issuegen,motdgen}.service. This is
required for CLHM v0.2. v0.2 is required for CLHM to work properly
on F33. coreos/fedora-coreos-tracker#631
kelvinfan001 added a commit to kelvinfan001/fedora-coreos-config that referenced this issue Sep 25, 2020
@kelvinfan001
overlay.d/05core: Update CLHM presets
c04f25d 
Add (back) preset for CLHM-{issuegen,motdgen}.service. This is
required for CLHM v0.2. v0.2 is required for CLHM to work properly
on F33. coreos/fedora-coreos-tracker#631
@kelvinfan001 kelvinfan001 mentioned this issue Sep 25, 2020
Merged
kelvinfan001 added a commit to coreos/fedora-coreos-config that referenced this issue Sep 28, 2020
@kelvinfan001
overlay.d/05core: Update CLHM presets
a3059c9 
Add (back) preset for CLHM-{issuegen,motdgen}.service. This is
required for CLHM v0.2. v0.2 is required for CLHM to work properly
on F33. coreos/fedora-coreos-tracker#631
@dustymabe
Copy link
Member Author

Looks like this is fixed with the new version of CLHM that's out and heading to stable f33 repos soon: https://bodhi.fedoraproject.org/updates/FEDORA-2020-70dc5a7573

@dustymabe dustymabe added the status/pending-next-release Fixed upstream. Waiting on a next release. label Sep 29, 2020
@dustymabe
Copy link
Member Author

The fix for this went into next stream release 33.20201006.1.0. Please try out the new release and report issues.

@dustymabe dustymabe removed the status/pending-next-release Fixed upstream. Waiting on a next release. label Oct 13, 2020
kelvinfan001 added a commit to coreos/fedora-coreos-config that referenced this issue Jan 7, 2021
@kelvinfan001
overlay.d/05core: Update CLHM presets
2e85a6e 
Add (back) preset for CLHM-{issuegen,motdgen}.service. This is
required for CLHM v0.2. v0.2 is required for CLHM to work properly
on F33. coreos/fedora-coreos-tracker#631
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@dustymabe @kelvinfan001