test_ts_of_lsn_api flakyness

[koivunej]

I've seen this once: https://neon-github-public-dev.s3.amazonaws.com/reports/pr-5756/6730946357/index.html#suites/2568f9ab62eb9e71321fab6263eed23e/269955fd321927a9:

test_runner/regress/test_lsn_mapping.py:236: in test_ts_of_lsn_api
    assert timestamp >= before_timestamp, "before_timestamp before timestamp"
E   AssertionError: before_timestamp before timestamp
E   assert datetime.datetime(2023, 11, 2, 10, 13, 28, 106532, tzinfo=datetime.timezone.utc) >= datetime.datetime(2023, 11, 2, 10, 13, 28, 208426, tzinfo=datetime.timezone.utc)

It does sound like it is non-trivial to fix.

[arpad-m]

In the CI run of #5497 the tests were already so flaky that it was more or less impossible to merge the PR. So I lowered the flakiness by adding sleeps. I think one way to reduce the occurence of test failures is to increase the amount slept. See the "Make test more robust" commit in #5497.

[koivunej]

"Make test more robust" commit

f6946e9 -- difference between timestamps is longer than 5ms, almost 10ms, this could still be related to the sleep, did not check the test.

[jcsp]

In last 3 days this has failed 6 times

[problame]

Test is fundamentally timing-sensitive. Arpad will start thread in #team-storage

[jcsp]

Let's check again in 1 week for ongoing failures

[arpad-m]

14 days after the patch merged: not a single flaky occurence. Last one was at 2024-05-03 14:37:10.

There has been a large gap of no occurences between April 13 and April 26, but we are (slightly) past the size of that gap and therefore, I think this has been resolved. Will revisit if it starts becoming a problem again.