zsysd stuck in an infinite loop promoting

[silveraid]

System information

Type	Version/Name
Distribution Name	Ubuntu
Distribution Version	20.04
Linux Kernel	5.4.0-42-generic #46-Ubuntu SMP
Architecture	x86
ZFS Version	0.8.3-1ubuntu12.2
SPL Version	0.8.3-1ubuntu12.1

Describe the problem you're observing

zsysd process stuck in some work causing moderate CPU usage (about 50% of a core) until the process is restarted.

Describe how to reproduce the problem

Not sure how to reproduce this, but I usually shut down my workstation at the end of the day and it is happening on its own every 2nd day or so.

Include any warning/errors/backtraces from the system logs

I have a Ubuntu default unstallation, have not tweaked anything else except that I have an encrypted regular volume, created outside of the Ubuntu installer, I don't have full pool or full home encryption, just a volume.

I observed that zsysd uses about 50% CPU all the time, until the process is restarted.

/var/log/syslog has the following messages repeating itself:

Jul 29 08:07:12 mk1 zed: eid=86881 class=history_event pool_guid=0xAF6BD66727173007  
Jul 29 08:07:13 mk1 zed[1735]: Missed 1 events
Jul 29 08:07:13 mk1 zed: eid=86894 class=history_event pool_guid=0xAF6BD66727173007  
Jul 29 08:07:13 mk1 zed[1735]: Missed 1 events

zpool events shows that a new event is being created very frequently:

Jul 29 2020 08:08:40.214981815 sysevent.fs.zfs.history_event
Jul 29 2020 08:08:40.242983660 sysevent.fs.zfs.history_event
Jul 29 2020 08:08:40.274985770 sysevent.fs.zfs.history_event
Jul 29 2020 08:08:40.298987353 sysevent.fs.zfs.history_event
Jul 29 2020 08:08:40.326989199 sysevent.fs.zfs.history_event
Jul 29 2020 08:08:40.354991044 sysevent.fs.zfs.history_event
Jul 29 2020 08:08:40.378992627 sysevent.fs.zfs.history_event
Jul 29 2020 08:08:40.406994473 sysevent.fs.zfs.history_event
Jul 29 2020 08:08:40.434996320 sysevent.fs.zfs.history_event
Jul 29 2020 08:08:40.466998429 sysevent.fs.zfs.history_event
Jul 29 2020 08:08:40.491000011 sysevent.fs.zfs.history_event
Jul 29 2020 08:08:40.515001594 sysevent.fs.zfs.history_event
Jul 29 2020 08:08:40.547003704 sysevent.fs.zfs.history_event
Jul 29 2020 08:08:40.571005286 sysevent.fs.zfs.history_event
Jul 29 2020 08:08:40.599007132 sysevent.fs.zfs.history_event
Jul 29 2020 08:08:40.623008714 sysevent.fs.zfs.history_event
Jul 29 2020 08:08:40.687012933 sysevent.fs.zfs.history_event
Jul 29 2020 08:08:40.711014516 sysevent.fs.zfs.history_event
Jul 29 2020 08:08:40.831022427 sysevent.fs.zfs.history_event
Jul 29 2020 08:08:40.943029811 sysevent.fs.zfs.history_event
Jul 29 2020 08:08:40.987032713 sysevent.fs.zfs.history_event

root@mk1:~# zpool events -H | awk '{print $5}' | sort | uniq -c
1153 sysevent.fs.zfs.history_event

These events seems to repeat themselves in an alternating pattern:

Jul 29 2020 08:09:19.893602679 sysevent.fs.zfs.history_event
        version = 0x0
        class = "sysevent.fs.zfs.history_event"
        pool = "rpool"
        pool_guid = 0xaf6bd66727173007
        pool_state = 0x0
        pool_context = 0x0
        history_hostname = "mk1"
        history_dsname = "rpool/ROOT/ubuntu_p20qi9/var/lib/1a532bc28bfb58c0a1a39cd02644d1585cbe398293da9940a871d6ce7e3a0105"
        history_internal_str = ""
        history_internal_name = "promote"
        history_dsid = 0x9ee
        history_txg = 0x250e61
        history_time = 0x5f2166ef
        time = 0x5f2166ef 0x35434b77 
        eid = 0x1652f

Jul 29 2020 08:09:19.929605062 sysevent.fs.zfs.history_event
        version = 0x0
        class = "sysevent.fs.zfs.history_event"
        pool = "rpool"
        pool_guid = 0xaf6bd66727173007
        pool_state = 0x0
        pool_context = 0x0
        history_hostname = "mk1"
        history_dsname = "rpool/ROOT/ubuntu_p20qi9/var/lib/4903e1deed0ad255c11fb800258fdf0f1f57bdce496969d87e2f542b2fbc58bc"
        history_internal_str = ""
        history_internal_name = "promote"
        history_dsid = 0x965
        history_txg = 0x250e62
        history_time = 0x5f2166ef
        time = 0x5f2166ef 0x3768a5c6 
        eid = 0x16530

Jul 29 2020 08:09:19.953606650 sysevent.fs.zfs.history_event
        version = 0x0
        class = "sysevent.fs.zfs.history_event"
        pool = "rpool"
        pool_guid = 0xaf6bd66727173007
        pool_state = 0x0
        pool_context = 0x0
        history_hostname = "mk1"
        history_dsname = "rpool/ROOT/ubuntu_p20qi9/var/lib/1a532bc28bfb58c0a1a39cd02644d1585cbe398293da9940a871d6ce7e3a0105"
        history_internal_str = ""
        history_internal_name = "promote"
        history_dsid = 0x9ee
        history_txg = 0x250e63
        history_time = 0x5f2166ef
        time = 0x5f2166ef 0x38d6e1fa 
        eid = 0x16531

Jul 29 2020 08:09:19.989609032 sysevent.fs.zfs.history_event
        version = 0x0
        class = "sysevent.fs.zfs.history_event"
        pool = "rpool"
        pool_guid = 0xaf6bd66727173007
        pool_state = 0x0
        pool_context = 0x0
        history_hostname = "mk1"
        history_dsname = "rpool/ROOT/ubuntu_p20qi9/var/lib/4903e1deed0ad255c11fb800258fdf0f1f57bdce496969d87e2f542b2fbc58bc"
        history_internal_str = ""
        history_internal_name = "promote"
        history_dsid = 0x965
        history_txg = 0x250e64
        history_time = 0x5f2166ef
        time = 0x5f2166ef 0x3afc3c48 
        eid = 0x16532

[loconox]

I'm facing the same issue.

[behlendorf]

The zsysd package isn't maintained in the OpenZFS repository, but is an Ubuntu package. It sounds like you may have encountered this issue, https://bugs.launchpad.net/ubuntu/+source/zsys/+bug/1870461 , which was fixed in zsys 0.4.6.

[loconox]

Thanks, I'm using version 0.4.8. I will report my issue there. Can be close.

[behlendorf]

Closed. As always if the root cause ends up being on the OpenZFS side please do let us know.