From 1099d0e9011b16473108fbb9f606719e66539770 Mon Sep 17 00:00:00 2001 From: Thomas Date: Thu, 4 Aug 2022 07:51:46 -0700 Subject: [PATCH 01/18] Fix `fides --version` showing dirty in releases --- .gitignore | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/.gitignore b/.gitignore index 85520d2b05..26a870ab47 100644 --- a/.gitignore +++ b/.gitignore @@ -1,5 +1,5 @@ -docs/fides/docs/api/openapi.json -docs/fides/docs/schemas/config_schema.json + +# frontend ui-build/ ## generic files to ignore @@ -20,6 +20,8 @@ tmp/* # docs docs/fides/site/ +docs/fides/docs/api/openapi.json +docs/fides/docs/schemas/config_schema.json # python specific *.pyc From 83f632143fec4569aaceaf7afeb060291ccd202b Mon Sep 17 00:00:00 2001 From: Thomas Date: Thu, 4 Aug 2022 22:55:46 +0800 Subject: [PATCH 02/18] update the gitignore --- .gitignore | 1 + 1 file changed, 1 insertion(+) diff --git a/.gitignore b/.gitignore index 26a870ab47..a833621f09 100644 --- a/.gitignore +++ b/.gitignore @@ -1,6 +1,7 @@ # frontend ui-build/ +src/fidesctl/ui-build/static/admin/ ## generic files to ignore *~ From a8c96535bc35bc1189122cccb8f4fb02a9b7b402 Mon Sep 17 00:00:00 2001 From: Allison King Date: Thu, 4 Aug 2022 08:52:01 -0700 Subject: [PATCH 03/18] Add workflow for uploading to testpypi (#951) * Add workflow for uploading to testpypi * Update token name * Change versioning style ti pep440-pre * Remove debugging trigger on PR * Update changelog * Try to properly tag version * Add a debugging test * Trigger build * Edit the right file * Remove debugging * Update .github/workflows/publish_package_testpypi.yaml Co-authored-by: Paul Sanders * Combine test pypi logic with original publish package workflow * Add debugging trigger to workflow * Fix syntax error * Install wheel * Remove debugging Co-authored-by: Paul Sanders --- .github/workflows/publish_package.yaml | 34 ++++++++++++++++++++++---- CHANGELOG.md | 1 + setup.cfg | 2 +- 3 files changed, 31 insertions(+), 6 deletions(-) diff --git a/.github/workflows/publish_package.yaml b/.github/workflows/publish_package.yaml index 1a8df03bff..c1118eb6d2 100644 --- a/.github/workflows/publish_package.yaml +++ b/.github/workflows/publish_package.yaml @@ -2,6 +2,8 @@ name: Publish fidesctl on: push: + branches: + - main tags: - "*" @@ -10,6 +12,13 @@ jobs: runs-on: ubuntu-latest steps: - uses: actions/checkout@v3 + with: + fetch-depth: 0 # This is required to properly tag packages + + - name: Setup Python 3.8 + uses: actions/setup-python@v2 + with: + python-version: 3.8 - name: Use Node.js 16 uses: actions/setup-node@v2 @@ -26,13 +35,28 @@ jobs: cd clients/admin-ui npm run prod-export - - name: Install Twine - run: pip install twine + - name: Install Twine and wheel + run: pip install twine wheel + + - name: Build the package + run: python setup.py sdist bdist_wheel - - name: Twine Upload + - name: Upload to test pypi + run: twine upload --repository testpypi dist/* + env: + TWINE_USERNAME: __token__ + TWINE_PASSWORD: ${{ secrets.TEST_PYPI_TOKEN }} + + - name: Check Prod Tag + id: check-tag run: | - python setup.py sdist bdist_wheel - twine upload dist/* + if [[ ${{ github.event.ref }} =~ ^refs/tags/[0-9]+\.[0-9]+\.[0-9]+$ ]]; then + echo ::set-output name=match::true + fi + + - name: Upload to pypi + if: steps.check-tag.outputs.match == 'true' + run: twine upload dist/* env: TWINE_USERNAME: __token__ TWINE_PASSWORD: ${{ secrets.PYPI_TOKEN }} diff --git a/CHANGELOG.md b/CHANGELOG.md index ff2e6cf420..7105fa2595 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -44,6 +44,7 @@ The types of changes are: * Update the `fidesctl` python package to automatically serve the UI [#941](https://github.com/ethyca/fides/pull/941) * Add `push` cli command alias for `apply` and deprecate `apply` [943](https://github.com/ethyca/fides/pull/943) * Add resource groups tagging api as a source of system generation [939](https://github.com/ethyca/fides/pull/939) +* Add GitHub Action to publish the `fidesctl` package to testpypi on pushes to main [#951](https://github.com/ethyca/fides/pull/951) ### Changed diff --git a/setup.cfg b/setup.cfg index 826c747787..c03de01679 100644 --- a/setup.cfg +++ b/setup.cfg @@ -1,6 +1,6 @@ [versioneer] VCS = git -style = pep440 +style = pep440-pre versionfile_source = src/fidesctl/_version.py versionfile_build = fidesctl/_version.py tag_prefix = From cd152460f9dcb6d5462d1d8e14ee43b9e5d9c65a Mon Sep 17 00:00:00 2001 From: Thomas Date: Fri, 5 Aug 2022 00:39:33 +0800 Subject: [PATCH 04/18] remove extra line --- .gitignore | 1 - 1 file changed, 1 deletion(-) diff --git a/.gitignore b/.gitignore index a833621f09..2d573bd893 100644 --- a/.gitignore +++ b/.gitignore @@ -1,4 +1,3 @@ - # frontend ui-build/ src/fidesctl/ui-build/static/admin/ From 447fbdbe85087bd295878f33819ba608e3e72d66 Mon Sep 17 00:00:00 2001 From: Thomas Date: Fri, 5 Aug 2022 00:45:06 +0800 Subject: [PATCH 05/18] update gitignore --- .gitignore | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/.gitignore b/.gitignore index 2d573bd893..5b21f48ec1 100644 --- a/.gitignore +++ b/.gitignore @@ -1,6 +1,5 @@ # frontend -ui-build/ -src/fidesctl/ui-build/static/admin/ +ui-build/* ## generic files to ignore *~ From 0e9671dab50e5d6dbd1c170011050f4979bb7257 Mon Sep 17 00:00:00 2001 From: Thomas Date: Fri, 5 Aug 2022 01:13:06 +0800 Subject: [PATCH 06/18] add a line to the Dockerfile that resets the busted cache --- Dockerfile | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/Dockerfile b/Dockerfile index e0ee598176..99021b759d 100644 --- a/Dockerfile +++ b/Dockerfile @@ -83,6 +83,10 @@ RUN pip install -r optional-requirements.txt COPY . /fides WORKDIR /fides +# Reset the busted git cache +RUN git rm --cached -r . +RUN git reset --hard + # Immediately flush to stdout, globally ENV PYTHONUNBUFFERED=TRUE From d14cfe164154ace83ac3fbdd192c7f8dfb280655 Mon Sep 17 00:00:00 2001 From: Thomas Date: Fri, 5 Aug 2022 01:15:27 +0800 Subject: [PATCH 07/18] add the no-deps flag to the fidesctl pip install --- Dockerfile | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Dockerfile b/Dockerfile index 99021b759d..77a3b9fb3c 100644 --- a/Dockerfile +++ b/Dockerfile @@ -103,7 +103,7 @@ CMD ["fidesctl", "webserver"] FROM builder as dev # Install fidesctl as a symlink -RUN pip install -e ".[all,mssql]" +RUN pip install --no-deps -e ".[all,mssql]" ################################## ## Production Application Setup ## @@ -116,4 +116,4 @@ COPY --from=frontend /fides/clients/admin-ui/out/ /fides/src/fidesctl/ui-build/s # Install without a symlink RUN python setup.py bdist_wheel -RUN pip install dist/fidesctl-*.whl +RUN pip install --no-deps dist/fidesctl-*.whl From 406fa982f6c967086753724bbf94961ff9fcb013 Mon Sep 17 00:00:00 2001 From: Thomas Date: Fri, 5 Aug 2022 15:26:00 +0800 Subject: [PATCH 08/18] update the git ignore to test the change --- .gitignore | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.gitignore b/.gitignore index 5b21f48ec1..157fe84d09 100644 --- a/.gitignore +++ b/.gitignore @@ -1,5 +1,5 @@ # frontend -ui-build/* +ui-build/ ## generic files to ignore *~ From 3fb0706645b439373bb8e55e588b6e525c7451ec Mon Sep 17 00:00:00 2001 From: Thomas Date: Mon, 8 Aug 2022 13:13:44 +0800 Subject: [PATCH 09/18] modify publish_package workflow to debug in CI --- .github/workflows/publish_package.yaml | 11 ++++------- 1 file changed, 4 insertions(+), 7 deletions(-) diff --git a/.github/workflows/publish_package.yaml b/.github/workflows/publish_package.yaml index 1a8df03bff..f74a4951a7 100644 --- a/.github/workflows/publish_package.yaml +++ b/.github/workflows/publish_package.yaml @@ -1,9 +1,9 @@ name: Publish fidesctl on: - push: - tags: - - "*" + pull_request: + branches: + - main jobs: upload_to_pypi: @@ -32,7 +32,4 @@ jobs: - name: Twine Upload run: | python setup.py sdist bdist_wheel - twine upload dist/* - env: - TWINE_USERNAME: __token__ - TWINE_PASSWORD: ${{ secrets.PYPI_TOKEN }} + git status From ae9d8209a3f3b3d96f67b17833531625ab4c8b7c Mon Sep 17 00:00:00 2001 From: Thomas Date: Mon, 8 Aug 2022 13:14:44 +0800 Subject: [PATCH 10/18] revert last commit --- .github/workflows/publish_package.yaml | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/.github/workflows/publish_package.yaml b/.github/workflows/publish_package.yaml index f74a4951a7..1a8df03bff 100644 --- a/.github/workflows/publish_package.yaml +++ b/.github/workflows/publish_package.yaml @@ -1,9 +1,9 @@ name: Publish fidesctl on: - pull_request: - branches: - - main + push: + tags: + - "*" jobs: upload_to_pypi: @@ -32,4 +32,7 @@ jobs: - name: Twine Upload run: | python setup.py sdist bdist_wheel - git status + twine upload dist/* + env: + TWINE_USERNAME: __token__ + TWINE_PASSWORD: ${{ secrets.PYPI_TOKEN }} From bf4f05aaddbe4f0611c1810ba073956e13ca7e43 Mon Sep 17 00:00:00 2001 From: Thomas Date: Mon, 8 Aug 2022 13:16:58 +0800 Subject: [PATCH 11/18] update CI to debug --- .github/workflows/publish_package.yaml | 26 +++----------------------- 1 file changed, 3 insertions(+), 23 deletions(-) diff --git a/.github/workflows/publish_package.yaml b/.github/workflows/publish_package.yaml index c1118eb6d2..0b5425b0d4 100644 --- a/.github/workflows/publish_package.yaml +++ b/.github/workflows/publish_package.yaml @@ -1,11 +1,9 @@ name: Publish fidesctl on: - push: + pull_request: branches: - main - tags: - - "*" jobs: upload_to_pypi: @@ -39,24 +37,6 @@ jobs: run: pip install twine wheel - name: Build the package - run: python setup.py sdist bdist_wheel - - - name: Upload to test pypi - run: twine upload --repository testpypi dist/* - env: - TWINE_USERNAME: __token__ - TWINE_PASSWORD: ${{ secrets.TEST_PYPI_TOKEN }} - - - name: Check Prod Tag - id: check-tag run: | - if [[ ${{ github.event.ref }} =~ ^refs/tags/[0-9]+\.[0-9]+\.[0-9]+$ ]]; then - echo ::set-output name=match::true - fi - - - name: Upload to pypi - if: steps.check-tag.outputs.match == 'true' - run: twine upload dist/* - env: - TWINE_USERNAME: __token__ - TWINE_PASSWORD: ${{ secrets.PYPI_TOKEN }} + python setup.py sdist + git status From 76bb4777c3ec54848f60a2c6a589eac0c335f57e Mon Sep 17 00:00:00 2001 From: Thomas Date: Mon, 8 Aug 2022 13:19:41 +0800 Subject: [PATCH 12/18] debug in CI --- .github/workflows/publish_package.yaml | 26 +++----------------------- 1 file changed, 3 insertions(+), 23 deletions(-) diff --git a/.github/workflows/publish_package.yaml b/.github/workflows/publish_package.yaml index c1118eb6d2..0b5425b0d4 100644 --- a/.github/workflows/publish_package.yaml +++ b/.github/workflows/publish_package.yaml @@ -1,11 +1,9 @@ name: Publish fidesctl on: - push: + pull_request: branches: - main - tags: - - "*" jobs: upload_to_pypi: @@ -39,24 +37,6 @@ jobs: run: pip install twine wheel - name: Build the package - run: python setup.py sdist bdist_wheel - - - name: Upload to test pypi - run: twine upload --repository testpypi dist/* - env: - TWINE_USERNAME: __token__ - TWINE_PASSWORD: ${{ secrets.TEST_PYPI_TOKEN }} - - - name: Check Prod Tag - id: check-tag run: | - if [[ ${{ github.event.ref }} =~ ^refs/tags/[0-9]+\.[0-9]+\.[0-9]+$ ]]; then - echo ::set-output name=match::true - fi - - - name: Upload to pypi - if: steps.check-tag.outputs.match == 'true' - run: twine upload dist/* - env: - TWINE_USERNAME: __token__ - TWINE_PASSWORD: ${{ secrets.PYPI_TOKEN }} + python setup.py sdist + git status From 8aa8284629c9443e951c37e2f33af795bfaafa00 Mon Sep 17 00:00:00 2001 From: Thomas Date: Mon, 8 Aug 2022 13:22:15 +0800 Subject: [PATCH 13/18] tweak the test paths for CI --- .github/workflows/code_checks.yml | 2 -- .github/workflows/publish_package.yaml | 2 -- 2 files changed, 4 deletions(-) diff --git a/.github/workflows/code_checks.yml b/.github/workflows/code_checks.yml index 39d1c204da..5e0a21a96b 100644 --- a/.github/workflows/code_checks.yml +++ b/.github/workflows/code_checks.yml @@ -2,8 +2,6 @@ name: Code Checks on: pull_request: - branches: - - main paths-ignore: - "**.md" diff --git a/.github/workflows/publish_package.yaml b/.github/workflows/publish_package.yaml index 0b5425b0d4..f7d8b808d4 100644 --- a/.github/workflows/publish_package.yaml +++ b/.github/workflows/publish_package.yaml @@ -2,8 +2,6 @@ name: Publish fidesctl on: pull_request: - branches: - - main jobs: upload_to_pypi: From 3b9da8cf690088ee2d7e9a26f113885d2400fa39 Mon Sep 17 00:00:00 2001 From: Thomas Date: Mon, 8 Aug 2022 13:34:58 +0800 Subject: [PATCH 14/18] fix the publish_package workflow --- .github/workflows/frontend_pr_checks.yml | 2 -- .github/workflows/publish_package.yaml | 32 +++++++++++++++++++++--- 2 files changed, 29 insertions(+), 5 deletions(-) diff --git a/.github/workflows/frontend_pr_checks.yml b/.github/workflows/frontend_pr_checks.yml index aadcf825e6..76f4c1cf3f 100644 --- a/.github/workflows/frontend_pr_checks.yml +++ b/.github/workflows/frontend_pr_checks.yml @@ -2,8 +2,6 @@ name: Frontend PR Checks on: pull_request: - branches: - - main paths: - "clients/**" - ".github/workflows/frontend_pr_checks.yml" diff --git a/.github/workflows/publish_package.yaml b/.github/workflows/publish_package.yaml index f7d8b808d4..e148270658 100644 --- a/.github/workflows/publish_package.yaml +++ b/.github/workflows/publish_package.yaml @@ -1,7 +1,9 @@ name: Publish fidesctl on: - pull_request: + push: + tags: + - "*" jobs: upload_to_pypi: @@ -34,7 +36,31 @@ jobs: - name: Install Twine and wheel run: pip install twine wheel - - name: Build the package + # The git reset is required here because the build modifies + # egg-info and the wheel becomes a dirty version + - name: Build the sdist run: | python setup.py sdist - git status + git reset --hard + + - name: Build the wheel + run: bdist_wheel + + - name: Upload to test pypi + run: twine upload --repository testpypi dist/* + env: + TWINE_USERNAME: __token__ + TWINE_PASSWORD: ${{ secrets.TEST_PYPI_TOKEN }} + + - name: Check Prod Tag + id: check-tag + run: | + if [[ ${{ github.event.ref }} =~ ^refs/tags/[0-9]+\.[0-9]+\.[0-9]+$ ]]; then + echo ::set-output name=match::true + fi + - name: Upload to pypi + if: steps.check-tag.outputs.match == 'true' + run: twine upload dist/* + env: + TWINE_USERNAME: __token__ + TWINE_PASSWORD: ${{ secrets.PYPI_TOKEN }} From 392652fed50112a9b0be9cd99f3982864340da1d Mon Sep 17 00:00:00 2001 From: Thomas Date: Mon, 8 Aug 2022 13:35:52 +0800 Subject: [PATCH 15/18] tweak when the publish_package workflow is triggered --- .github/workflows/publish_package.yaml | 2 ++ 1 file changed, 2 insertions(+) diff --git a/.github/workflows/publish_package.yaml b/.github/workflows/publish_package.yaml index e148270658..f6480900c8 100644 --- a/.github/workflows/publish_package.yaml +++ b/.github/workflows/publish_package.yaml @@ -2,6 +2,8 @@ name: Publish fidesctl on: push: + branches: + - "*" tags: - "*" From f487816223c0edf669b91f68d857b1fe5e951611 Mon Sep 17 00:00:00 2001 From: Thomas Date: Mon, 8 Aug 2022 13:55:37 +0800 Subject: [PATCH 16/18] fix the wheel build command --- .github/workflows/publish_package.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/publish_package.yaml b/.github/workflows/publish_package.yaml index f6480900c8..f0b8c9396a 100644 --- a/.github/workflows/publish_package.yaml +++ b/.github/workflows/publish_package.yaml @@ -46,7 +46,7 @@ jobs: git reset --hard - name: Build the wheel - run: bdist_wheel + run: python setup.py bdist_wheel - name: Upload to test pypi run: twine upload --repository testpypi dist/* From 33a34fd127e1fac09d2c13e32e6b888880326755 Mon Sep 17 00:00:00 2001 From: Thomas Date: Mon, 8 Aug 2022 14:23:28 +0800 Subject: [PATCH 17/18] update the changelog --- CHANGELOG.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 8c8a61b729..a6b93ba3fe 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -41,6 +41,8 @@ The types of changes are: ### Fixed * Deprecated config options will continue to be respected when set via environment variables [#965](https://github.com/ethyca/fides/pull/965) +* The git cache is rebuilt within the Docker container [#962](https://github.com/ethyca/fides/pull/962) +* The `wheel` pypi build no longer has a dirty version tag [#962](https://github.com/ethyca/fides/pull/962) ## [1.8.0](https://github.com/ethyca/fides/compare/1.7.1...1.8.0) - 2022-08-04 From 3729d6d762315aa257d2b72f7ef0592cbdcf25f8 Mon Sep 17 00:00:00 2001 From: Thomas Date: Mon, 8 Aug 2022 00:57:03 -0700 Subject: [PATCH 18/18] Update .github/workflows/publish_package.yaml --- .github/workflows/publish_package.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/publish_package.yaml b/.github/workflows/publish_package.yaml index f0b8c9396a..4303ae55c7 100644 --- a/.github/workflows/publish_package.yaml +++ b/.github/workflows/publish_package.yaml @@ -3,7 +3,7 @@ name: Publish fidesctl on: push: branches: - - "*" + - main tags: - "*"