version all the files, update CI tests and vscode settings #77

dave-connors-3 · 2023-06-21T14:09:01Z

this PR

adds versioned schema files for 1.5 + 1.6
leaves the originals in place until we get the greenlight to deprecate them from @benmosher
updates CI and the vscode settings to point the right versions at separate test files in the tests directory.
adds dependencies.yml support in 1.6

benmosher · 2023-06-23T12:57:55Z

an idea: storing files as e.g. schemas/{version}/dbt_project-{version}.json would be the teensiest bit redundant, but as the versions pile up, you won't end up with one super-heavy folder fulla schemata.

joellabes

The ones I commented on are the only ones I looked at, assuming that the rest are renames and copy-pastes. If there's others that you'd like me to check please lmk!

I'm approving this since the suggested changes aren't mandatory but will make life easier in the future 🔮

.github/workflows/ci.yml

joellabes · 2023-06-26T05:36:48Z

tests/validate_plusless_properties.py

+PROJECT_SCHEMA_FILES = [
+    "./schemas/dbt_project.json",
+    "./schemas/dbt_project-1.5.json",
+    "./schemas/dbt_project-1.6.json"
+]
+


This might be better as a query that lists all files matching the pattern dbt_project[\-\d\.]*\.json or something, instead of a hardcoded list that we have to remember to keep up to date

joellabes · 2023-06-26T05:38:05Z

schemas/dependencies-1.6.json

@@ -0,0 +1,22 @@
+{


joellabes · 2023-06-26T05:38:45Z

tests/1.6/invalid/dependencies.yml

@@ -0,0 +1,2 @@
+projects:
+  - gnome: my_dbt_project


https://www.youtube.com/watch?v=6n3pFFPSlW4

joellabes · 2023-06-26T05:38:53Z

tests/1.6/valid/dependencies.yml

@@ -0,0 +1,2 @@
+projects:
+  - name: my_dbt_project


dave-connors-3 · 2023-06-27T17:00:01Z

@joellabes would love a quick re-review! I updated the folder paths per Ben's recommendation and added the matrix feature to make CI a lot DRYer. The only notable byproduct of that move was that schema.yml files in the tests directory are now named dbt_yml_files to match the name of the schema, so the matrix combos behave properly. Let me know if that is a dealbreaker for you in any way!

I also chose not to do the file regexing in validate_plusless_properties.py -- feels like a quarterly update of that variable as a part of the new version process is an OK amount of friction

joellabes

Magnificent!

joellabes · 2023-06-27T21:48:16Z

.github/workflows/ci.yml

+      - name: Validate dbt_project.yml sample files for dbt version ${{ matrix.dbt-version }}
        run: |
-          npx ajv-cli test -s schemas/dbt_project-1.5.json -d tests/1.5/valid/dbt_project.yml --valid
-          npx ajv-cli test -s schemas/dbt_project-1.5.json -d tests/1.5/invalid/dbt_project.yml --invalid
-      - name: Validate resource yml sample files for dbt version 1.5
-        run: |
-          npx ajv-cli test -s schemas/dbt_yml_files-1.5.json -d tests/1.5/valid/schema.yml --valid
-          npx ajv-cli test -s schemas/dbt_yml_files-1.5.json -d tests/1.5/invalid/schema.yml --invalid
-      - name: Validate packages.yml sample files for dbt version 1.5
-        run: |
-          npx ajv-cli test -s schemas/packages-1.5.json -d tests/1.5/valid/packages.yml --valid
-          npx ajv-cli test -s schemas/packages-1.5.json -d tests/1.5/invalid/packages.yml --invalid
-      - name: Validate selectors.yml sample files for dbt version 1.5
-        run: |
-          npx ajv-cli test -s schemas/selectors-1.5.json -d tests/1.5/valid/selectors.yml --valid
-          npx ajv-cli test -s schemas/selectors-1.5.json -d tests/1.5/invalid/selectors.yml --invalid
-
-      - name: Validate dbt_project.yml sample files for dbt version 1.6
-        run: |
-          npx ajv-cli test -s schemas/dbt_project-1.6.json -d tests/1.6/valid/dbt_project.yml --valid
-          npx ajv-cli test -s schemas/dbt_project-1.6.json -d tests/1.6/invalid/dbt_project.yml --invalid
-      - name: Validate resource yml sample files for dbt version 1.6
-        run: |
-          npx ajv-cli test -s schemas/dbt_yml_files-1.6.json -d tests/1.6/valid/schema.yml --valid
-          npx ajv-cli test -s schemas/dbt_yml_files-1.6.json -d tests/1.6/invalid/schema.yml --invalid
-      - name: Validate packages.yml sample files for dbt version 1.6
-        run: |
-          npx ajv-cli test -s schemas/packages-1.6.json -d tests/1.6/valid/packages.yml --valid
-          npx ajv-cli test -s schemas/packages-1.6.json -d tests/1.6/invalid/packages.yml --invalid
-      - name: Validate selectors.yml sample files for dbt version 1.6
-        run: |
-          npx ajv-cli test -s schemas/selectors-1.6.json -d tests/1.6/valid/selectors.yml --valid
-          npx ajv-cli test -s schemas/selectors-1.6.json -d tests/1.6/invalid/selectors.yml --invalid
-
-      - name: Validate dependencies.yml sample files for dbt version 1.6
-        run: |
-          npx ajv-cli test -s schemas/dependencies-1.6.json -d tests/1.6/valid/dependencies.yml --valid
-          npx ajv-cli test -s schemas/dependencies-1.6.json -d tests/1.6/invalid/dependencies.yml --invalid
+          npx ajv-cli test -s schemas/${{ matrix.dbt-version }}/${{ matrix.dbt-file-name }}-${{ matrix.dbt-version }}.json -d tests/${{ matrix.dbt-version }}/valid/${{ matrix.dbt-file-name }}.yml --valid
+          npx ajv-cli test -s schemas/${{ matrix.dbt-version }}/${{ matrix.dbt-file-name }}-${{ matrix.dbt-version }}.json -d tests/${{ matrix.dbt-version }}/invalid/${{ matrix.dbt-file-name }}.yml --invalid
+


:chefs-kiss:

:shreks-kick:

dave-connors-3 · 2023-06-28T14:14:51Z

@joellabes i didn't mean to dismiss your stale review :( i moved the default files back to the root of the folder so the IDE validation doesn't break

dave-connors-3 added 6 commits June 21, 2023 09:05

version all the files, update CI tests and vscode settings

9298dcf

smol python miss

bc98e41

Merge branch 'main' into version-files

61f6999

cleanup from merge conflict

2e3f2e4

more cleanup

127560c

add dependencies.yml

d4fdd23

dave-connors-3 requested a review from joellabes June 21, 2023 15:05

joellabes previously approved these changes Jun 26, 2023

View reviewed changes

dave-connors-3 added 2 commits June 27, 2023 11:26

reorganize folders

f15edec

use matrix in CI

58f08f6

dave-connors-3 dismissed joellabes’s stale review via 58f08f6 June 27, 2023 16:34

dave-connors-3 added 5 commits June 27, 2023 11:36

update filepaths

81b1954

matrix syntax

dcbf6da

move matrix'

b8f5701

rename files for CI matrix strategy

38afeac

we love a typo

82525f6

joellabes previously approved these changes Jun 27, 2023

View reviewed changes

move default files to root so as to not break the IDE

b5db613

dave-connors-3 dismissed joellabes’s stale review via b5db613 June 28, 2023 14:00

fix CI

0c6d629

dave-connors-3 requested a review from joellabes June 28, 2023 14:14

joellabes approved these changes Jun 28, 2023

View reviewed changes

dave-connors-3 merged commit 0f52fde into main Jun 29, 2023

jtcohen6 mentioned this pull request Jul 19, 2023

Add support for projects in dependencies.yml #81

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

version all the files, update CI tests and vscode settings #77

version all the files, update CI tests and vscode settings #77

dave-connors-3 commented Jun 21, 2023 •

edited

Loading

benmosher commented Jun 23, 2023

joellabes left a comment

joellabes Jun 26, 2023

joellabes Jun 26, 2023

joellabes Jun 26, 2023

joellabes Jun 26, 2023

dave-connors-3 commented Jun 27, 2023

joellabes left a comment

joellabes Jun 27, 2023

dave-connors-3 Jun 28, 2023

dave-connors-3 commented Jun 28, 2023

version all the files, update CI tests and vscode settings #77

version all the files, update CI tests and vscode settings #77

Conversation

dave-connors-3 commented Jun 21, 2023 • edited Loading

benmosher commented Jun 23, 2023

joellabes left a comment

Choose a reason for hiding this comment

joellabes Jun 26, 2023

Choose a reason for hiding this comment

joellabes Jun 26, 2023

Choose a reason for hiding this comment

joellabes Jun 26, 2023

Choose a reason for hiding this comment

joellabes Jun 26, 2023

Choose a reason for hiding this comment

dave-connors-3 commented Jun 27, 2023

joellabes left a comment

Choose a reason for hiding this comment

joellabes Jun 27, 2023

Choose a reason for hiding this comment

dave-connors-3 Jun 28, 2023

Choose a reason for hiding this comment

dave-connors-3 commented Jun 28, 2023

dave-connors-3 commented Jun 21, 2023 •

edited

Loading