chore(jobsdb): add support for schema migration changesets to run always

[atzoum]

Description

When running separate gw and processor instances we cannot control the order that they get upgraded or how schema migrations get executed. Additionally, we cannot guarantee that while a gw schema migration is running or after it completes, a processor running an older release will not create new tables using a previous schema.

Changesets that run always can help in such scenarios, by bringing non-migrated tables into a usable state during a version upgrade or after starting up following a panic caused by an incompatible jobsdb table schema encounter.

Additional items

• Removed all down sql migration changesets since we are never performing a down migration, so these scripts are not used and have never been tested.
• Using distributed locks to make sure that no other process will be able to add or remove tables during the schema migration happens, otherwise schema migration may panic and db will be left in a dirty state. An advisory lock, shared between schema migration, internal migration and add new ds loop is introduced.

Notion Ticket

Link

Security

• The code changed/added as part of this pull request won't create any security issues with how the software is being used.

[codecov]

Codecov Report

Base: 46.73% // Head: 46.90% // Increases project coverage by +0.17% 🎉

Coverage data is based on head (3562cd5) compared to base (bed58c5).
Patch coverage: 64.51% of modified lines in pull request are covered.

❗ Current head 3562cd5 differs from pull request most recent head ac1ad1b. Consider uploading reports for the commit ac1ad1b to get more accurate results

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #2746      +/-   ##
==========================================
+ Coverage   46.73%   46.90%   +0.17%     
==========================================
  Files         300      300              
  Lines       49123    49139      +16     
==========================================
+ Hits        22958    23051      +93     
+ Misses      24704    24619      -85     
- Partials     1461     1469       +8     

Impacted Files	Coverage Δ
services/sql-migrator/migrator.go	42.96% <20.00%> (-1.04%)	⬇️
jobsdb/migration.go	78.64% <58.49%> (+1.36%)	⬆️
jobsdb/jobsdb.go	73.14% <69.04%> (+0.29%)	⬆️
jobsdb/setup.go	87.50% <76.92%> (-3.20%)	⬇️
services/oauth/oauth.go	53.88% <0.00%> (-3.11%)	⬇️
warehouse/upload.go	18.45% <0.00%> (-0.92%)	⬇️
regulation-worker/internal/delete/api/api.go	67.53% <0.00%> (-0.19%)	⬇️
warehouse/slave.go	1.08% <0.00%> (+<0.01%)	⬆️
warehouse/identities.go	1.04% <0.00%> (+<0.01%)	⬆️
warehouse/warehouse.go	10.03% <0.00%> (+0.01%)	⬆️
... and 9 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.