Tests: Attempt to fix flaky test in reloader on directories changes

[matej-g]

Signed-off-by: Matej Gera matejgera@gmail.com

• I added CHANGELOG entry for this change.
• Change is not relevant to the end user.

Changes

We've been seeing some increase in failures for TestReloader_DirectoriesApply recently (see e.g. this CI run). Although there were other fixes for the test (#3798), it looks like this is not directly related to the previously discovered issues.

According to the output, the test seems to skip step 1 from among the steps required to be run to manipulate rule files, leading to incorrect number of watch events:

--- FAIL: TestReloader_DirectoriesApply (2.52s)
    reloader_test.go:278: Performing step number 0
    reloader_test.go:278: Performing step number 2
    reloader_test.go:278: Performing step number 3
    reloader_test.go:278: Performing step number 4
    reloader_test.go:278: Performing step number 5
    reloader_test.go:365: reloader_test.go:365:
        
        	exp: 6
        
        	got: 5
        
        
FAIL
FAIL	github.com/thanos-io/thanos/pkg/reloader	5.280s

After more investigation, I have established that this is due to a 'race' between the reloader and the goroutine executing the steps. On an occasion, the reloader can invoke the reload handler two times while the step executing goroutine did not have time to catch up with the reloader. This causes the step execution loop to skip ahead to get on the same number of reloads as the reloader and thus not executing the step in between. The suggestion in this PR is to have mechanism to check these skips and 'catch up' with the reloader if that happens.

Verification

Run a modified test locally with extra output confirming the skipped steps are executed, if such a skip occurs in the test run.

[onprem]

Looks good to me. Thank you for fixing this, it has caused a lot of pain.

Just one non-blocking comment.

[GiedriusS]

💪