Migrate NFT.Storage to Postgres

[dchoi27]

The base PR for this migration is #263

Tasks

• SQL Schema
• Endpoints using postgres client
• normalize cidv1 feat: normalized CID web3-storage/web3-schema#4
• Add docs and scripts for local database setup to run tests Feat/migrations-docker #425
• Add deals schema and logic feat: migration deals schema #418
• pins and uploads endpoints need deals data like the check endpoint #454
• change status type to enum #459
• Cronjobs using postgres client #461 feat: cron v1 #496
  • + setup foreign table materialized views refresh
• Migration pipeline using postgres client #473 Feat/461-473 #491
• metrics will query postgres directly from /metrics feat: add prom metrics endpoint generated from postgres data #495
• handle DB errors properly, define what to return and what to catch and send to sentry. feat: db error reporting #510
• pinning services api needs to send error to sentry, because there we dont use the normal throw error flow. feat: add PSA error handler #512
• add uploads deleted_at, update select to account for deleted_at and change the ON CONFLICT clause in upload_fn fix: dont delete uploads #551
• Dont delete auth_keys use deleted_at because uploads needs auth_key.id feat: do not delete auth keys #539
• Define max number of item returned in the list endpoints (nft list and pins list) fix: min, max and default limits for v1 list endpoints #493
• Improve the db ts types setup, where we actually setup with docker, run the schema and get the types.
• Rename accounts table to user #374 refactor: rename account table to user #485
• Add new enum value for service_type IpfsCluster2 feat: add new cluster enum value #494 / feat: add new cluster enum and nothing else #509
• Move v1 endpoints to root refactor: move v1 to root (minimal changes) #534

Diagram

https://dbdiagram.io/d/615d7356940c4c4eec87e2be

[Gozala]

Plan of action

• Define schema for nft.storage #377
• Migrate data from kv stores to Postgres  #385
• Rewire nft.storage API endpoints to use Postgres #386

[alanshaw]

Things to verify on staging:

• Test register/login via email
• Test register/login via github
• Test logging in with existing "email user" still works
• Test logging in with existing "github user" still works
• Test creating an auth key
• Test deleting an auth key (does not appear in list that it can no longer be used)
• Test uploading a blob
• Test uploading a directory
• Test uploading a car
• Test uploading partial cars
• Test uploading an NFT
• Test uploading with existing key (from before migration) and new key This fails as expected, because staging uses different salt for creating/validating it's keys.
• Test deleting an upload (it gets deleted_at set and does not appear in files listing)
• Test re-uploading the same file twice
• Test re-uploading a previously deleted file
• Test pinning a CID
• Test files listing