Stop all background workers for the duration of the upgrade

[cevian]

This commit stops bgws while upgrading the Promscale extension.
This prevents lock contention between jobs and upgrades and thus
seems safer.

[cevian]

This will address some of the potential slowness seen in #274

[JamesGuthrie]

The background workers should also be stopped at the beginning of promscale--0.0.0.sql.

[JamesGuthrie]

This doesn't work without timescaledb. I know that we're planning on not supporting installations without timescaledb, but that will also require improving our test infra at least a little. I would suggest the following:

+-- stop all background workers in the current db so they don't interfere
+-- with the upgrade (by e.g. holding locks).
+--
+-- Note: this stops the scheduler (and thus all jobs)
+-- and waits for the current txn (the one doing the upgrade) to finish
+-- before starting it up again. Thus, no jobs will be running during
+-- the upgrade.
+
+-- TimescaleDB itself does the same thing:
+-- https://github.com/timescale/timescaledb/blob/72d03e6f7d30cc4794c9263445f14199241e2eb5/sql/updates/pre-update.sql#L34
+DO
+$stop_bgw$
+DECLARE
+    _is_timescaledb_installed boolean = false;
+BEGIN
+    SELECT count(*) > 0
+    INTO STRICT _is_timescaledb_installed
+    FROM pg_extension
+    WHERE extname='timescaledb';
+
+    IF _is_timescaledb_installed THEN
+        SELECT _timescaledb_internal.restart_background_workers();
+    END IF;
+END;
+$stop_bgw$;

Missed some important points

[JamesGuthrie]

Implementation of #291

[pgnickb]

Sorry for a stupid question, but I failed to get the answer by just glancing through the code of ts_bgw_db_workers_restart.. Is it blocking or does it always return immediately upon sending the message? And more relevantly, do we want to set a statement_timeout for this operation in case it takes too long?

[JamesGuthrie]

Sorry for a stupid question, but I failed to get the answer by just glancing through the code of ts_bgw_db_workers_restart.. Is it blocking or does it always return immediately upon sending the message? And more relevantly, do we want to set a statement_timeout for this operation in case it takes too long?

It looks like it can be blocking, if the message queue to the background worker process is full. The comment on ts_bgw_message_send_and_wait (called by ts_bgw_db_workers_restart) says:

/*
 * Write element to queue, wait/error if queue is full
 * consumes message and deallocates
 */
extern bool
ts_bgw_message_send_and_wait(BgwMessageType message_type, Oid db_oid)

@cevian On another note, I tested this change out in my test environment, by adding the "stop bgw" snippet to the promscale--0.0.0.sql script. It stopped the background workers, and they were run again when the timeout was reached.

[JamesGuthrie]

I also added a commit to make CI pass. Feel free to squash it down.

[JamesGuthrie]

@cevian are you waiting on anything to merge this?

[cevian]

Connector test https://github.com/timescale/promscale/actions/runs/2457393370