Server reload errors

[plocket]

[See previous discussions in #392.]

If any developer on the docassemble server saves a module or uploads a module or something similar, it will cause a server reload for everyone on the server. If a test is running when the server reloads, the test may fail by timing out. This causes false negatives, which encourages people to ignore test failures.

Can we do anything about this?

Ideas

1. Definitely possible, but utility is questionable

1. Us the API to test whether the server is reloading. Would any of those really help? How much? Our server takes about 30 seconds to reload. Does the utility/impact of this method increase as the reload time increases? Or as the maximum test timeout seconds decreases? We could do this at these intervals:
   1. before an operation
   2. during an operation
   3. after a timeout failure
   4. On a loop (some fraction of current max timeout)
2. Error more quickly on a development server, removing the "graceful reload" behavior. Requires admin intervention, possibly a lot of it.

2. Maybe not possible

3. Use websockets or some such to communicate from the server to the client when it's going to reload. Supposedly not really possible with this system. At the very least, individual admins would probably end up being the ones implementing the solution.
4. Give a different error on a development server when the server reloads. Requires admin intervention, if it is even possible.
5. For each test run, build a docassemble server in docker. Will take a lot longer for every test run (which will impact our development significantly). May not be feasible to replicate the developer's server configurations and setup properly and therefore may lead to inaccurate results. Though that last part hasn't been tested, I'm not sure how it could be otherwise. We could possibly require the developer to provide the most up-to-date image of their server, but I don't know how, or how to do it securely.

Right now we're mainly looking into option 1, but discussion is open for all of them.

[plocket]

Note: The following may be impossible in a GitHub action
If we end up making our own server on each test run, and we change da so it can send out an alert about a server reload to a webhook, this is a possible lead on how to allow da to send that to the GitHub server that's running the test. Though, as stated above, this may actually be impossible.

[plocket]

Possible way to handle this locally without da changes that may actually work! From Slack convo.

Ok, I have maybe actually for real got an idea for how to solve our server reload detection issue. I swear, there have to be solutions to this out there, I just don’t know how to find them. Here goes:

1. Poll the server every X ms to see if it’s up. Keep a list of statuses for the last Y seconds (let’s say 30)
2. Run functions in the test. If a page times out (let’s say after 30 secs), check the list of statuses from the present into the past for a bit over 30 secs.
3. If the server was unavailable anytime during that time, wait till the server status shows that it’s up again, with some kind of maximum timeout there too.
4. Once the server seems to be up, try re-doing the operation that timed out.
5. Only do that a limited number of times and with a reasonable pause in between each poll.

This only works because we happen to know that the servers will be down for more than 5 seconds or so (though usually it's much more than that). That lets us poll at a reasonable rate - maybe every 500 ms or even 100 ms.

Possible issue someone else brought up: A server’s latency period that is longer than its downtime period.
Latency: What is the amount of time that a request can expect a response? Can depend on Circumstances.
Downtime: What is the min amount of time that it takes a server to reload?

[Btw, we can handle that by overlapping our polling to the server. We wouldn’t wait for a response before polling again.]