Return success from webBackendUpdateConnection endpoint once connection changes are saved

[lmossman]

What

Currently, when a user makes a change to their connection configuration that requires a reset, the webBackendUpdateConnection endpoint is called. This endpoint does the following:

1. Saves the changes to the connection record in the db
2. Decides if a reset is required (if not it returns here)
3. If a reset is required, it submits the reset and then synchronously waits for the reset to complete
4. Lastly it kicks off a new manual sync before returning a successful API response.

This means that users may wait many seconds or even multiple minutes in the following loading screen after clicking Save, which is a pretty bad user experience (when taking this screenshot, I had to wait around 1 minute for the loading spinner to go away, which is pretty excruciatingly slow):

How

Instead of forcing the user to wait for a reset job to fully complete before letting them do anything else, we should instead tell them that their changes were saved successfully as soon as the database is consistent with the changes they made, since at that point forward they should see their new changes if they refresh the page.

Therefore, the webBackendUpdateConnection should return a successful API response as soon as we have saved the user's changes to their connection record in the database. The logic to kick off a reset and a sync after that should not block the UI from returning, but should still happen as a result of saving changes when necessary.

A couple of ideas of how to accomplish this:

1. Add a new flag or signal method to the connection manager workflow that indicates we want to perform a reset and then a sync immediately after the reset, without the usual waiting time. This new signal or flag can then be used in a non-synchronous resetConnection method in the TemporalClient/EventRunner, which the webBackendUpdateConnection can call after saving the connection and before returning to kick off the reset+sync, without having to actually wait for anything to complete. This endpoint can then just return as soon as the reset job is kicked off.
2. Use some other mechanism to kick off an asynchronous thread/process that will submit a reset, wait to complete, and then submit a sync. This process can be spawned by the webBackendUpdateConnection after saving the connection, allowing it to return as soon as that asynchronous process has been spawned.

[lmossman]

FYI @timroes @evantahler

[evantahler]

cc @salima-airbyte

[salima-airbyte]

Possible UI change, discuss w/ FE team (soon)

[salima-airbyte]

Once the form has been submitted invoking the endpoint. If no errors, API should return 200 code. Reset job will be done asynchronously and its status will be in "status" tab.