Merge pull request #298 from zanebclark/feat/integrate-connections-toml

Feat/integrate connections toml

.github/CONTRIBUTING.md

@@ -7,7 +7,8 @@ operating environment, schemachange version and python version. Whenever possibl
 also include a brief, self-contained code example that demonstrates the problem.
 
 We have
-included [issue templates](https://github.com/Snowflake-Labs/schemachange/issues/new/choose) for reporting bugs, requesting features and seeking clarifications. Choose the appropriate issue template to contribute to the repository.
+included [issue templates](https://github.com/Snowflake-Labs/schemachange/issues/new/choose) for reporting bugs,
+requesting features and seeking clarifications. Choose the appropriate issue template to contribute to the repository.
 
 ## Contributing code
 
@@ -22,8 +23,6 @@ Thank you for your interest in contributing code to schemachange!
 
 ### Guide to contributing to schemachange
 
-> **IMPORTANT** : You will need to follow the [provisioning and schemachange setup instructions](../demo/README.MD) to ensure you can run GitHub actions against your Snowflake account before placing a PR with main schemachange repository so that your PR can be merged into schemachange master branch.
-
 1. If you are a first-time contributor
     + Go to [Snowflake-Labs/Schemachange](https://github.com/Snowflake-Labs/schemachange) and click the "fork" button to
       create your own copy of the project.
@@ -53,8 +52,8 @@ Thank you for your interest in contributing code to schemachange!
     + [Pull](https://github.com/git-guides/git-pull) the latest changes from upstream, including tags:
 
     ```shell
-    git checkout main
-    git pull upstream main --tags
+    git checkout master
+    git pull upstream master --tags
     ```
 
 2. Create and Activate a Virtual Environment
@@ -98,24 +97,32 @@ Thank you for your interest in contributing code to schemachange!
 
     + Commit locally as you progress ( [git add](https://github.com/git-guides/git-add)
       and [git commit](https://github.com/git-guides/git-commit) ). Use a properly formatted commit message. Be sure to
-      document any changed behavior in the [CHANGELOG.md](../CHANGELOG.md) file to help us collate the changes for a specific release.
+      document any changed behavior in the [CHANGELOG.md](../CHANGELOG.md) file to help us collate the changes for a
+      specific release.
 
 4. Test your contribution locally
 
    ```bash
    python -m pytest
    ```
-   PS: Please add test cases to the features you are developing so that over time, we can capture any lapse in functionality changes.
+   PS: Please add test cases to the features you are developing so that over time, we can capture any lapse in
+   functionality changes.
+
+5. Perform integration tests on your branch from your fork
+    - Follow the [provisioning and schemachange setup instructions](../demo/README.MD) to configure your Snowflake
+      account for testing.
+    - Follow [these](https://docs.github.com/en/actions/managing-workflow-runs-and-deployments/managing-workflow-runs/manually-running-a-workflow)
+      instructions to manually run the `master-pytest` workflow on your fork of the repo, targeting your feature branch.
 
-5. Push your contribution to GitHub
+6. Push your contribution to GitHub
 
-    [Push](https://github.com/git-guides/git-push) your changes back to your fork on GitHub
+   [Push](https://github.com/git-guides/git-push) your changes back to your fork on GitHub
 
     ```shell
     git push origin update-build-library-dependencies
     ```
 
-6. Raise a Pull Request to merge your contribution into the a Schemachange Release
+7. Raise a Pull Request to merge your contribution into the a Schemachange Release
     + Go to GitHub. The new branch will show up with a
       green [Pull Request](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/about-pull-requests#initiating-the-pull-request)
       button. Make sure the title and message are clear, concise and self-explanatory. Then click the button to submit

.github/workflows/master-pytest.yml

@@ -29,13 +29,13 @@ jobs:
     runs-on: ${{ matrix.os }}
     if: ${{ github.event.label.name == 'ci-run-tests' || github.event_name == 'push' || github.event_name == 'workflow_dispatch' }}
     env:
-      SNOWFLAKE_PASSWORD: ${{ secrets.SCHEMACHANGE_SNOWFLAKE_PASSWORD }}
-      SNOWFLAKE_USER: ${{ secrets.SCHEMACHANGE_SNOWFLAKE_USER }}
       SNOWFLAKE_ACCOUNT: ${{ secrets.SCHEMACHANGE_SNOWFLAKE_ACCOUNT }}
-      SNOWFLAKE_DATABASE: SCHEMACHANGE_DEMO
-      SNOWFLAKE_WAREHOUSE: SCHEMACHANGE_DEMO_WH
+      SNOWFLAKE_USER: ${{ secrets.SCHEMACHANGE_SNOWFLAKE_USER }}
       SNOWFLAKE_ROLE: SCHEMACHANGE_DEMO-DEPLOY
+      SNOWFLAKE_WAREHOUSE: SCHEMACHANGE_DEMO_WH
+      SNOWFLAKE_DATABASE: SCHEMACHANGE_DEMO
       MY_TARGET_SCHEMA: ${{ matrix.scenario-name }}_${{ github.run_number }}_${{ strategy.job-index }}
+      SNOWFLAKE_PASSWORD: ${{ secrets.SCHEMACHANGE_SNOWFLAKE_PASSWORD }}
       SCENARIO_NAME: ${{ matrix.scenario-name }}
     steps:
       - uses: actions/checkout@v4
@@ -58,6 +58,18 @@ jobs:
           flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics
           # exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide
           flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics
+      - name: Create and populate connections.toml
+        run: |
+          touch ./connections.toml
+          echo [default] >> ./connections.toml
+          echo account = \"${SNOWFLAKE_ACCOUNT}\" >> ./connections.toml
+          echo user = \"${SNOWFLAKE_USER}\" >> ./connections.toml
+          echo role = \"${SNOWFLAKE_ROLE}\" >> ./connections.toml
+          echo warehouse = \"${SNOWFLAKE_WAREHOUSE}\" >> ./connections.toml
+          echo database = \"${SNOWFLAKE_DATABASE}\" >> ./connections.toml
+          echo password = \"${SNOWFLAKE_PASSWORD}\" >> ./connections.toml
+          echo "cat connections.toml"
+          cat ./connections.toml
       - name: Test with pytest
         id: pytest
         run: |
@@ -66,16 +78,36 @@ jobs:
       - name: Test Schemachange on ${{ matrix.os }} targeting ${{ env.SNOWFLAKE_DATABASE }}.${{ env.MY_TARGET_SCHEMA }} schema
         run: |
           echo "::group::Setting up ${MY_TARGET_SCHEMA}"
-          schemachange deploy --config-folder ./demo/setup/${SCENARIO_NAME}
+          schemachange deploy \
+            --config-folder ./demo \
+            --config-file-name schemachange-config-setup.yml \
+            --root-folder ./demo/${SCENARIO_NAME}/1_setup \
+            --connection-name default \
+            --connections-file-path ./connections.toml \
+            --verbose
           echo "::endgroup::"
+
           echo "::group::Testing Rendering to ${MY_TARGET_SCHEMA}"
-          schemachange render --config-folder ./demo/${SCENARIO_NAME} ./demo/${SCENARIO_NAME}/A__render.sql
-          schemachange render --config-folder ./demo/${SCENARIO_NAME} ./demo/${SCENARIO_NAME}/R__render.sql
-          schemachange render --config-folder ./demo/${SCENARIO_NAME} ./demo/${SCENARIO_NAME}/V1.0.0__render.sql
+
+          schemachange render \
+            --config-folder ./demo/${SCENARIO_NAME} \
+            ./demo/${SCENARIO_NAME}/2_test/A__render.sql
+          schemachange render \
+            --config-folder ./demo/${SCENARIO_NAME} \
+            ./demo/${SCENARIO_NAME}/2_test/R__render.sql
+          schemachange render \
+            --config-folder ./demo/${SCENARIO_NAME} \
+            ./demo/${SCENARIO_NAME}/2_test/V1.0.0__render.sql
           echo "::endgroup::"
+
           echo "::group::Testing Deployment using ${MY_TARGET_SCHEMA}"
           set +e
-          schemachange deploy --config-folder ./demo/${SCENARIO_NAME}
+          schemachange deploy \
+            --config-folder ./demo/${SCENARIO_NAME} \
+            --connection-name default \
+            --connections-file-path ./connections.toml \
+            --root-folder ./demo/${SCENARIO_NAME}/2_test \
+            --verbose
           RESULT=$?
           if [ $RESULT -eq 0 ]; then
             echo "Deployment Completed!"
@@ -84,9 +116,17 @@ jobs:
           fi
           echo "::endgroup::"
           set -e
+
           echo "::group::Tearing down up ${MY_TARGET_SCHEMA}"
-          schemachange deploy --config-folder ./demo/teardown/${SCENARIO_NAME}
+          schemachange deploy \
+            --config-folder ./demo \
+            --config-file-name schemachange-config-teardown.yml \
+            --connection-name default \
+            --connections-file-path ./connections.toml \
+            --root-folder ./demo/${SCENARIO_NAME}/3_teardown \
+            --verbose
           echo "::endgroup::"
+
           if [ $RESULT -ne 0 ]; then
             exit 1
           fi

CHANGELOG.md

@@ -7,12 +7,14 @@ All notable changes to this project will be documented in this file.
 ### Added
 - Use of `structlog~=24.1.0` for standard log outputs
 - Verified Schemachange against Python 3.12
+- Support for connections.toml configurations
+- Support for supplying the authenticator, private key path, token path, connections file path, and connection name via the YAML and command-line configurations.
 
 ### Changed
 - Refactored the main cli.py into multiple modules - config, session.
 - Updated contributing guidelines and demo readme content to help contributors setup local snowflake account to run the github actions in their fork before pushing the PR to upstream repository.
 - Removed tests against Python 3.8 [End of Life on 2024-10-07](https://devguide.python.org/versions/#supported-versions)
-
+- Command-line vars are now merged into YAML vars instead of overwriting them entirely
 
 ## [3.7.0] - 2024-07-22
 ### Added