[BUG] - Jobs drifting between execution

[zerosoul13]

Describe the bug

I'm trying to execute a set of various scripts at a set interval. I found that at times, jobs seem to execute at odd intervals. At first, I though the execution time of each of them would be the root cause but this was not the case. Execution takes less that 1 second.

To Reproduce

Steps to reproduce the behavior:

1. Create a new scheduler
2. Register Jobs
3. Start scheduler

resCh := make(chan Result)

s := gocron.NewScheduler(time.UTC)
for _, script := range scripts {

     // script
    // "Command": "/usr/local/nagios/libexec/check_linux_disk.pl -p / -w 80 -c 90"
    s.Scheduler.Every(seconds).Seconds().Do(script, resCh)
}
s.StartBlocking()

Version

github.com/go-co-op/gocron v1.13.0

Expected behavior

Consistent executions at set intervals

Additional context

Below an example of a job set to run every 60 seconds but skips executions

sh-4.2# ./mon-agent

INFO[2022-03-29 00:33:59] DISK SPACE OK::XXU00000::warn=80 %, crit=90 %::fs_name=/, fs_type=UNKNOWN (0x794c7630), used%=42, used=102134 mb, available=141750 mb, total=257007 mb   check=check_linux_disk_usage exectime=0.043842 perfdata="_=102134 available=141750 total=257007"

INFO[2022-03-29 00:35:59] DISK SPACE OK::XXU00000::warn=80 %, crit=90 %::fs_name=/, fs_type=UNKNOWN (0x794c7630), used%=42, used=102135 mb, available=141749 mb, total=257007 mb   check=check_linux_disk_usage exectime=0.0410761 perfdata="_=102135 available=141749 total=257007"

INFO[2022-03-29 00:37:59] DISK SPACE OK::XXU00000::warn=80 %, crit=90 %::fs_name=/, fs_type=UNKNOWN (0x794c7630), used%=42, used=102135 mb, available=141749 mb, total=257007 mb   check=check_linux_disk_usage exectime=0.0354278 perfdata="_=102135 available=141749 total=257007"

INFO[2022-03-29 00:40:00] DISK SPACE OK::XXU00000::warn=80 %, crit=90 %::fs_name=/, fs_type=UNKNOWN (0x794c7630), used%=42, used=102136 mb, available=141748 mb, total=257007 mb   check=check_linux_disk_usage exectime=0.0470769 perfdata="_=102136 available=141748 total=257007"

[JohnRoesler]

@zerosoul13 looking into this some, I completely agree there is currently some lag between jobs that can cause a job to occur slightly later than expected. Definitely something we should figure out how to handle better. It's tricky because what's causing the lag is the small amount of compute time between calculating when to run the job next and then running to that time. One idea, is if we had some sort of clock sync / skew check on every job to make sure it's not lagging and then somehow correct.

Just some quick data to confirm what you're seeing: analysis of job lag.pdf

[JohnRoesler]

@zerosoul13 could you test out the master branch when you have a chance and see if it's solved for you? I ran some jobs for several hours and had no drift.

[zerosoul13]

@JohnRoesler Hello, I'll be glad to test the master branch and provide feedback. Thank you for looking into this :D