Support multiple sql dirs

[bplunkett-stripe]

Description

Support multiple schema dirs

Motivation

Fixes #149

Testing

Multiple:

 go run ./cmd/pg-schema-diff plan --schema-dir ~/stripe/temp/examplesql --dsn 'host=localhost user=bplunkett dbname=postgres' --schema-dir ~/stripe/temp/examplesql2

################################ Generated plan ################################
1. CREATE SCHEMA "some_new_schema";
        -- Statement Timeout: 3s

2. CREATE TABLE "public"."examplesql2_foobar" (
        "id" text COLLATE "pg_catalog"."default"
);
        -- Statement Timeout: 3s

3. CREATE TABLE "public"."foobar" (
        "id" integer,
        "some_other_column" character varying(255) COLLATE "pg_catalog"."default",
        "fizz" text COLLATE "pg_catalog"."default"
);
        -- Statement Timeout: 3s

4. ALTER TABLE "public"."foobar" ADD CONSTRAINT "some_constraint" CHECK((id > 0));
        -- Statement Timeout: 3s

5. ALTER TABLE "public"."foobar" REPLICA IDENTITY FULL;
        -- Statement Timeout: 3s

6. CREATE INDEX CONCURRENTLY some_idx ON public.foobar USING btree (id);
        -- Statement Timeout: 20m0s
        -- Lock Timeout: 3s

7. DROP FUNCTION "public"."update_version"();
        -- Statement Timeout: 3s
        -- Hazard HAS_UNTRACKABLE_DEPENDENCIES: Dependencies, i.e. other functions used in the function body, of non-sql functions cannot be tracked. As a result, we cannot guarantee that function dependencies are ordered properly relative to this statement. For drops, this means you need to ensure that all functions this function depends on are dropped after this statement.

8. DROP TABLE "public"."your_table";
        -- Statement Timeout: 20m0s
        -- Lock Timeout: 3s
        -- Hazard DELETES_DATA: Deletes all rows in the table (and the table itself)

9. DROP EXTENSION "pg_repack";
        -- Statement Timeout: 3s
        -- Hazard HAS_UNTRACKABLE_DEPENDENCIES: This extension may be in use by tables, indexes, functions, triggers, etc. Tihs statement will be ran last, so this may be OK.

Nothing provided:

 go run ./cmd/pg-schema-diff plan --dsn 'host=localhost user=bplunkett dbname=postgres'
Error: either --schema-dir or --schema-source-dsn must be set
Usage:
  pg-schema-diff plan [flags]

Aliases:
  plan, diff

Flags:
      --data-pack-new-tables                     If set, will data pack new tables in the plan to minimize table size (re-arranges columns). (default true)
      --disable-plan-validation                  If set, will disable plan validation. Plan validation runs the migration against a temporarydatabase with an identical schema to the original, asserting that the generated plan actually migrates the schema to the desired target.
      --dsn string                               Connection string for the database (DB password can be specified through PGPASSWORD environment variable)
      --exclude-schema stringArray               Exclude the specified schema in the plan
  -h, --help                                     help for plan
      --include-schema stringArray               Include the specified schema in the plan
  -s, --insert-statement stringArray             'index=<index> statement="<statement>" timeout=<duration> lock_timeout=<duration>' values. Will insert the statement at the index in the generated plan. This follows normal insert semantics. Example: -s 'index=1 statement="SELECT pg_sleep(5)" timeout=5s lock_timeout=1s'
  -l, --lock-timeout-modifier stringArray        list of 'pattern="<regex>" timeout=<duration>', where if a statement matches the regex, the statement will have the target lock timeout. If multiple regexes match, the latest regex will take priority. Example: -t 'pattern="CREATE TABLE" timeout=5m'
      --schema-dir stringArray                   Directory of .SQL files to use as the schema source (can be multiple). Use to generate a diff between the target database and the schema in this directory.
      --schema-source-dsn string                 DSN for the database to use as the schema source. Use to generate a diff between the target database and the schema in this database.
  -t, --statement-timeout-modifier stringArray   list of 'pattern="<regex>" timeout=<duration>', where if a statement matches the regex, the statement will have the target statement timeout. If multiple regexes match, the latest regex will take priority. Example: -t 'pattern="CREATE TABLE" timeout=5m'

exit status 1

Missing schema dir:

 go run ./cmd/pg-schema-diff plan --schema-dir ~/stripe/temp/examplesqldne --dsn 'host=localhost user=bplunkett dbname=postgres'
Error: getting DDL from path "/Users/bplunkett/stripe/temp/examplesqldne": open /Users/bplunkett/stripe/temp/examplesqldne: no such file or directory
Usage:
  pg-schema-diff plan [flags]

Aliases:
  plan, diff

Flags:
      --data-pack-new-tables                     If set, will data pack new tables in the plan to minimize table size (re-arranges columns). (default true)
      --disable-plan-validation                  If set, will disable plan validation. Plan validation runs the migration against a temporarydatabase with an identical schema to the original, asserting that the generated plan actually migrates the schema to the desired target.
      --dsn string                               Connection string for the database (DB password can be specified through PGPASSWORD environment variable)
      --exclude-schema stringArray               Exclude the specified schema in the plan
  -h, --help                                     help for plan
      --include-schema stringArray               Include the specified schema in the plan
  -s, --insert-statement stringArray             'index=<index> statement="<statement>" timeout=<duration> lock_timeout=<duration>' values. Will insert the statement at the index in the generated plan. This follows normal insert semantics. Example: -s 'index=1 statement="SELECT pg_sleep(5)" timeout=5s lock_timeout=1s'
  -l, --lock-timeout-modifier stringArray        list of 'pattern="<regex>" timeout=<duration>', where if a statement matches the regex, the statement will have the target lock timeout. If multiple regexes match, the latest regex will take priority. Example: -t 'pattern="CREATE TABLE" timeout=5m'
      --schema-dir stringArray                   Directory of .SQL files to use as the schema source (can be multiple). Use to generate a diff between the target database and the schema in this directory.
      --schema-source-dsn string                 DSN for the database to use as the schema source. Use to generate a diff between the target database and the schema in this database.
  -t, --statement-timeout-modifier stringArray   list of 'pattern="<regex>" timeout=<duration>', where if a statement matches the regex, the statement will have the target statement timeout. If multiple regexes match, the latest regex will take priority. Example: -t 'pattern="CREATE TABLE" timeout=5m'

exit status 1

[aleclarson]

This doesn't appear to work for me.

pg-schema-diff apply \
  --skip-confirm-prompt \
  --dsn postgres://postgres:postgres@localhost:5432/postgres \
  --schema-dir ~/pg-nano/demos/exhaustive/sql/schema \
  --schema-dir ~/pg-nano/demos/exhaustive/sql/functions \
  --allow-hazards HAS_UNTRACKABLE_DEPENDENCIES

Only the last --schema-dir path is used, leading to unexpected DROP statements.

[Navbryce]

@aleclarson I believe this is a versioning problem. go install ...@latest does not install the latest commit on main, and we haven't cut a release yet to include this.

I recommend you go install github.com/stripe/pg-schema-diff/cmd/pg-schema-diff@main and try again (installs using actual latest revision). You can also validate by running go version -m $(where pg-schema-diff) and ensure the version says something like v0.7.1.

Feel free to open an issue if the issue persists!