Restart scheduler on error

[mkindahl]

If the scheduler receives an error, it will never restart again since bgw_restart_time is set to BGW_NEVER_RESTART, which will prevent all jobs from executing.

This commit adds the GUC timescaledb.bgw_scheduler_restart_time that can be set to the restart time for the scheduler. It defaults to 60 seconds, which is the default restart interval for background workers PostgreSQL defines.

It also adds timescaledb.debug_bgw_scheduler_exit_status to be able to shutdown the scheduler with a non-zero exit status, which allows the restart functionality to be tested.

It also ensures that backend_type is explicitly set up rather than copied from application_name and add some more information to application_name. It also updates the tests to use backend_type where applicable.

Disable-check: loader-change

[mkindahl]

Compared to #6195 this adds code to the start of the launcher that handles the status for any schedulers that existed prior to the restart by killing them. Reproducing it in a test is difficult because of how the schedulers are used in tests, but this short script demonstrates the issue:

create view tsdb_bgw as
       select datname, pid, backend_type, application_name from pg_stat_activity
       where backend_type like '%TimescaleDB%';

alter system set timescaledb.debug_bgw_scheduler_exit_status to 1;
alter system set timescaledb.bgw_scheduler_restart_time to 10;
select pg_reload_conf();

-- Kill the schedulers
select pid, pg_terminate_backend(pid) from pg_stat_activity where backend_type like '%Scheduler%';
select pg_sleep(1);
select datname, backend_type, count(*) from tsdb_bgw group by datname, backend_type;

-- Wait for the schedulers to restart.
select pg_sleep(10);
select datname, backend_type, count(*) from tsdb_bgw group by datname, backend_type;

-- Kill the launcher, it should restart immediately.
select pid, pg_terminate_backend(pid) from pg_stat_activity where backend_type like '%Launcher%';
select pg_sleep(1);

-- This should only show one scheduler for each database.
select datname, backend_type, count(*) from tsdb_bgw group by datname, backend_type;

[codecov]

Codecov Report

Attention: Patch coverage is 72.72727% with 9 lines in your changes missing coverage. Please review.

Project coverage is 82.28%. Comparing base (59f50f2) to head (7767c39).
Report is 689 commits behind head on main.

Files with missing lines	Patch %	Lines
src/loader/bgw_launcher.c	67.85%	5 Missing and 4 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #7527      +/-   ##
==========================================
+ Coverage   80.06%   82.28%   +2.21%     
==========================================
  Files         190      238      +48     
  Lines       37181    43701    +6520     
  Branches     9450    10966    +1516     
==========================================
+ Hits        29770    35959    +6189     
- Misses       2997     3406     +409     
+ Partials     4414     4336      -78     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[fabriziomello]

LGTM

[timescale-automation]

Automated backport to 2.17.x not done: cherry-pick failed.

Git status

HEAD detached at origin/2.17.x
You are currently cherry-picking commit 62e623591.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	new file:   .unreleased/pr_7527
	modified:   src/bgw/scheduler.c
	modified:   src/compat/compat.h
	modified:   src/guc.h
	modified:   src/loader/bgw_launcher.c
	modified:   src/loader/bgw_launcher.h
	modified:   src/loader/loader.c
	modified:   test/expected/bgw_launcher.out
	modified:   test/sql/bgw_launcher.sql
	modified:   test/sql/include/bgw_launcher_utils.sql
	modified:   tsl/test/expected/bgw_scheduler_control.out
	new file:   tsl/test/expected/bgw_scheduler_restart.out
	modified:   tsl/test/sql/CMakeLists.txt
	new file:   tsl/test/sql/bgw_scheduler_restart.sql

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   .github/gh_matrix_builder.py
	both modified:   src/guc.c
	both modified:   test/pgtest/CMakeLists.txt

Job log