Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Prepare to remove actions hooks #1038

Merged
merged 3 commits into from
Jul 24, 2024
Merged

Prepare to remove actions hooks #1038

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
4beba4e
Add integrationTestProvider option
danielrbradley Jul 12, 2024
ea2b0d9
Add sskPrivateKey option for test setup
danielrbradley Jul 12, 2024
a8d1fff
Apply to nightly-test workflow
danielrbradley Jul 12, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions provider-ci/internal/pkg/templates/bridged-provider.config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -129,6 +129,9 @@ extraTests:
# name: My Test
# ...

# Run e2e tests in the provider as well as in the examples directory
integrationTestProvider: false

# Run e2e tests using the examples and test suite in the pulumi/examples repo.
# This is unused: https://github.com/search?q=org%3Apulumi+path%3A.ci-mgmt.yaml+%22testPulumiExamples%3A%22&type=code
testPulumiExamples: false
Expand Down Expand Up @@ -208,6 +211,11 @@ publish:
# Used in 9 providers: https://github.com/search?q=org%3Apulumi+path%3A.ci-mgmt.yaml+%22docker%3A%22&type=code
#docker: false

# Setup SSH with specified private key before running tests in CI job.
# This should be provided from a secret
# Used by the docker provider only: https://github.com/search?q=org%3Apulumi+path%3A.ci-mgmt.yaml+%22sshPrivateKey%3A%22&type=code
#sshPrivateKey: ${{ secrets.PRIVATE_SSH_KEY_FOR_DIGITALOCEAN }}

# Authenticate with GCP before running tests in CI job
# Used in gcp and docker: https://github.com/search?q=org%3Apulumi+path%3A.ci-mgmt.yaml+%22gcp%3A%22&type=code
#gcp: false
Expand Down
18 changes: 16 additions & 2 deletions provider-ci/internal/pkg/templates/bridged-provider/.github/workflows/main.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -190,6 +190,16 @@ jobs:
- name: Login to Google Cloud Registry
run: gcloud --quiet auth configure-docker
#{{- end }}#
#{{- if .Config.sshPrivateKey }}#
- name: Setup SSH key
uses: webfactory/ssh-agent@v0.7.0
with:
ssh-private-key: #{{ .Config.sshPrivateKey }}#
#{{- end }}#
#{{- if .Config.integrationTestProvider }}#
- name: Prepare upstream code
run: make upstream
#{{- end }}#
#{{- if index .Config "setup-script" }}#
- name: Run setup script
run: #{{ index .Config "setup-script" }}#
Expand All @@ -204,9 +214,13 @@ jobs:
#{{- if .Config.actions.preTest }}#
#{{ .Config.actions.preTest | toYaml | indent 4 }}#
#{{- end }}#
#{{- if .Config.integrationTestProvider }}#
- name: Run provider tests
working-directory: provider
run: go test -v -json -count=1 -cover -timeout 2h -tags=${{ matrix.language }} -parallel 4 . 2>&1 | tee /tmp/gotest.log | gotestfmt
#{{- end }}#
Comment on lines +217 to +221
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is unit testing the provider, no?

It's natural for providers to have slightly different ways to test themselves (for example I think it's a mistake to assume the provider subdiretory). Simplifying this to something like make test_provider_ci (or whatever) could accomplish the same while still leaving the door open for some repo-specific behavior.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is running the language-specific tests within the provider. I think these are basically all tests which depend on either the SDKs or having access to deploy test infrastructure. These are specified by tags such as nodejs similar to the examples. These tests are excluded from running by default when running the normal provider unit tests.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh man I missed the language tag, that's weird! Is this maybe just an instance of someone dropping E2E tests under provider instead of examples?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct - which is what we already have in a few providers. I think all the replay tests are in the provider folder, for example. This just formalises what's already being done into a simple option rather than leaning on providers to inject their own custom steps and matrix options.

- name: Run tests
run: cd examples && go test -v -json -count=1 -cover -timeout 2h -tags=${{
matrix.language }} -parallel 4 . 2>&1 | tee /tmp/gotest.log | gotestfmt
run: cd examples && go test -v -json -count=1 -cover -timeout 2h -tags=${{ matrix.language }} -parallel 4 . 2>&1 | tee /tmp/gotest.log | gotestfmt
strategy:
fail-fast: false
matrix:
Expand Down
16 changes: 16 additions & 0 deletions provider-ci/internal/pkg/templates/bridged-provider/.github/workflows/nightly-test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -103,6 +103,16 @@ jobs:
- name: Login to Google Cloud Registry
run: gcloud --quiet auth configure-docker
#{{- end }}#
#{{- if .Config.sshPrivateKey }}#
- name: Setup SSH key
uses: webfactory/ssh-agent@v0.7.0
with:
ssh-private-key: #{{ .Config.sshPrivateKey }}#
#{{- end }}#
#{{- if .Config.integrationTestProvider }}#
- name: Prepare upstream code
run: make upstream
#{{- end }}#
#{{- if index .Config "setup-script" }}#
- name: Run setup script
run: #{{ index .Config "setup-script" }}#
Expand All @@ -117,6 +127,12 @@ jobs:
#{{- if .Config.actions.preTest }}#
#{{ .Config.actions.preTest | toYaml | indent 4 }}#
#{{- end }}#
#{{- if .Config.integrationTestProvider }}#
- name: Run provider tests
if: matrix.testTarget == 'local'
working-directory: provider
run: go test -v -json -count=1 -cover -timeout 2h -tags=${{ matrix.language }} -parallel 4 . 2>&1 | tee /tmp/gotest.log | gotestfmt
#{{- end }}#
- name: Run tests
run: cd examples && go test -v -json -count=1 -cover -timeout 2h -tags=${{
matrix.language }} -parallel 4 . 2>&1 | tee /tmp/gotest.log | gotestfmt
Expand Down
18 changes: 16 additions & 2 deletions provider-ci/internal/pkg/templates/bridged-provider/.github/workflows/prerelease.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -131,6 +131,16 @@ jobs:
- name: Login to Google Cloud Registry
run: gcloud --quiet auth configure-docker
#{{- end }}#
#{{- if .Config.sshPrivateKey }}#
- name: Setup SSH key
uses: webfactory/ssh-agent@v0.7.0
with:
ssh-private-key: #{{ .Config.sshPrivateKey }}#
#{{- end }}#
#{{- if .Config.integrationTestProvider }}#
- name: Prepare upstream code
run: make upstream
#{{- end }}#
#{{- if index .Config "setup-script" }}#
- name: Run setup script
run: #{{ index .Config "setup-script" }}#
Expand All @@ -145,9 +155,13 @@ jobs:
#{{- if .Config.actions.preTest }}#
#{{ .Config.actions.preTest | toYaml | indent 4 }}#
#{{- end }}#
#{{- if .Config.integrationTestProvider }}#
- name: Run provider tests
working-directory: provider
run: go test -v -json -count=1 -cover -timeout 2h -tags=${{ matrix.language }} -parallel 4 . 2>&1 | tee /tmp/gotest.log | gotestfmt
#{{- end }}#
- name: Run tests
run: cd examples && go test -v -json -count=1 -cover -timeout 2h -tags=${{
matrix.language }} -parallel 4 . 2>&1 | tee /tmp/gotest.log | gotestfmt
run: cd examples && go test -v -json -count=1 -cover -timeout 2h -tags=${{ matrix.language }} -parallel 4 . 2>&1 | tee /tmp/gotest.log | gotestfmt
strategy:
fail-fast: false
matrix:
Expand Down
18 changes: 16 additions & 2 deletions provider-ci/internal/pkg/templates/bridged-provider/.github/workflows/release.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -130,6 +130,16 @@ jobs:
- name: Login to Google Cloud Registry
run: gcloud --quiet auth configure-docker
#{{- end }}#
#{{- if .Config.sshPrivateKey }}#
- name: Setup SSH key
uses: webfactory/ssh-agent@v0.7.0
with:
ssh-private-key: #{{ .Config.sshPrivateKey }}#
#{{- end }}#
#{{- if .Config.integrationTestProvider }}#
- name: Prepare upstream code
run: make upstream
#{{- end }}#
#{{- if index .Config "setup-script" }}#
- name: Run setup script
run: #{{ index .Config "setup-script" }}#
Expand All @@ -144,9 +154,13 @@ jobs:
#{{- if .Config.actions.preTest }}#
#{{ .Config.actions.preTest | toYaml | indent 4 }}#
#{{- end }}#
#{{- if .Config.integrationTestProvider }}#
- name: Run provider tests
working-directory: provider
run: go test -v -json -count=1 -cover -timeout 2h -tags=${{ matrix.language }} -parallel 4 . 2>&1 | tee /tmp/gotest.log | gotestfmt
#{{- end }}#
- name: Run tests
run: cd examples && go test -v -json -count=1 -cover -timeout 2h -tags=${{
matrix.language }} -parallel 4 . 2>&1 | tee /tmp/gotest.log | gotestfmt
run: cd examples && go test -v -json -count=1 -cover -timeout 2h -tags=${{ matrix.language }} -parallel 4 . 2>&1 | tee /tmp/gotest.log | gotestfmt
strategy:
fail-fast: false
matrix:
Expand Down
19 changes: 17 additions & 2 deletions ...der-ci/internal/pkg/templates/bridged-provider/.github/workflows/run-acceptance-tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -173,6 +173,16 @@ jobs:
- name: Login to Google Cloud Registry
run: gcloud --quiet auth configure-docker
#{{- end }}#
#{{- if .Config.sshPrivateKey }}#
- name: Setup SSH key
uses: webfactory/ssh-agent@v0.7.0
with:
ssh-private-key: #{{ .Config.sshPrivateKey }}#
#{{- end }}#
#{{- if .Config.integrationTestProvider }}#
- name: Prepare upstream code
run: make upstream
#{{- end }}#
#{{- if index .Config "setup-script" }}#
- name: Run setup script
run: #{{ index .Config "setup-script" }}#
Expand All @@ -187,10 +197,15 @@ jobs:
#{{- if .Config.actions.preTest }}#
#{{ .Config.actions.preTest | toYaml | indent 4 }}#
#{{- end }}#
#{{- if .Config.integrationTestProvider }}#
- name: Run provider tests
if: matrix.testTarget == 'local'
working-directory: provider
run: go test -v -json -count=1 -cover -timeout 2h -tags=${{ matrix.language }} -parallel 4 . 2>&1 | tee /tmp/gotest.log | gotestfmt
#{{- end }}#
- name: Run tests
if: matrix.testTarget == 'local'
run: cd examples && go test -v -json -count=1 -cover -timeout 2h -tags=${{
matrix.language }} -skip TestPulumiExamples -parallel 4 . 2>&1 | tee /tmp/gotest.log | gotestfmt
run: cd examples && go test -v -json -count=1 -cover -timeout 2h -tags=${{ matrix.language }} -skip TestPulumiExamples -parallel 4 . 2>&1 | tee /tmp/gotest.log | gotestfmt
- name: Run pulumi/examples tests
if: matrix.testTarget == 'pulumiExamples'
run: cd examples && go test -v -json -count=1 -cover -timeout 2h -tags=${{
Expand Down
3 changes: 1 addition & 2 deletions provider-ci/test-providers/aws/.github/workflows/master.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -181,8 +181,7 @@ jobs:
- name: Make upstream
run: make upstream
- name: Run tests
run: cd examples && go test -v -json -count=1 -cover -timeout 2h -tags=${{
matrix.language }} -parallel 4 . 2>&1 | tee /tmp/gotest.log | gotestfmt
run: cd examples && go test -v -json -count=1 -cover -timeout 2h -tags=${{ matrix.language }} -parallel 4 . 2>&1 | tee /tmp/gotest.log | gotestfmt
strategy:
fail-fast: false
matrix:
Expand Down
3 changes: 1 addition & 2 deletions provider-ci/test-providers/aws/.github/workflows/prerelease.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -124,8 +124,7 @@ jobs:
- name: Make upstream
run: make upstream
- name: Run tests
run: cd examples && go test -v -json -count=1 -cover -timeout 2h -tags=${{
matrix.language }} -parallel 4 . 2>&1 | tee /tmp/gotest.log | gotestfmt
run: cd examples && go test -v -json -count=1 -cover -timeout 2h -tags=${{ matrix.language }} -parallel 4 . 2>&1 | tee /tmp/gotest.log | gotestfmt
strategy:
fail-fast: false
matrix:
Expand Down
3 changes: 1 addition & 2 deletions provider-ci/test-providers/aws/.github/workflows/release.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -123,8 +123,7 @@ jobs:
- name: Make upstream
run: make upstream
- name: Run tests
run: cd examples && go test -v -json -count=1 -cover -timeout 2h -tags=${{
matrix.language }} -parallel 4 . 2>&1 | tee /tmp/gotest.log | gotestfmt
run: cd examples && go test -v -json -count=1 -cover -timeout 2h -tags=${{ matrix.language }} -parallel 4 . 2>&1 | tee /tmp/gotest.log | gotestfmt
strategy:
fail-fast: false
matrix:
Expand Down
3 changes: 1 addition & 2 deletions provider-ci/test-providers/aws/.github/workflows/run-acceptance-tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -162,8 +162,7 @@ jobs:
run: make upstream
- name: Run tests
if: matrix.testTarget == 'local'
run: cd examples && go test -v -json -count=1 -cover -timeout 2h -tags=${{
matrix.language }} -skip TestPulumiExamples -parallel 4 . 2>&1 | tee /tmp/gotest.log | gotestfmt
run: cd examples && go test -v -json -count=1 -cover -timeout 2h -tags=${{ matrix.language }} -skip TestPulumiExamples -parallel 4 . 2>&1 | tee /tmp/gotest.log | gotestfmt
- name: Run pulumi/examples tests
if: matrix.testTarget == 'pulumiExamples'
run: cd examples && go test -v -json -count=1 -cover -timeout 2h -tags=${{
Expand Down
3 changes: 1 addition & 2 deletions provider-ci/test-providers/cloudflare/.github/workflows/master.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -166,8 +166,7 @@ jobs:
run: |
cd provider && go test -v -json -count=1 -cover -timeout 2h -tags=${{ matrix.language }} -parallel 4 . 2>&1 | tee /tmp/gotest.log | gotestfmt
- name: Run tests
run: cd examples && go test -v -json -count=1 -cover -timeout 2h -tags=${{
matrix.language }} -parallel 4 . 2>&1 | tee /tmp/gotest.log | gotestfmt
run: cd examples && go test -v -json -count=1 -cover -timeout 2h -tags=${{ matrix.language }} -parallel 4 . 2>&1 | tee /tmp/gotest.log | gotestfmt
strategy:
fail-fast: false
matrix:
Expand Down
3 changes: 1 addition & 2 deletions provider-ci/test-providers/cloudflare/.github/workflows/prerelease.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -111,8 +111,7 @@ jobs:
run: |
cd provider && go test -v -json -count=1 -cover -timeout 2h -tags=${{ matrix.language }} -parallel 4 . 2>&1 | tee /tmp/gotest.log | gotestfmt
- name: Run tests
run: cd examples && go test -v -json -count=1 -cover -timeout 2h -tags=${{
matrix.language }} -parallel 4 . 2>&1 | tee /tmp/gotest.log | gotestfmt
run: cd examples && go test -v -json -count=1 -cover -timeout 2h -tags=${{ matrix.language }} -parallel 4 . 2>&1 | tee /tmp/gotest.log | gotestfmt
strategy:
fail-fast: false
matrix:
Expand Down
3 changes: 1 addition & 2 deletions provider-ci/test-providers/cloudflare/.github/workflows/release.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -110,8 +110,7 @@ jobs:
run: |
cd provider && go test -v -json -count=1 -cover -timeout 2h -tags=${{ matrix.language }} -parallel 4 . 2>&1 | tee /tmp/gotest.log | gotestfmt
- name: Run tests
run: cd examples && go test -v -json -count=1 -cover -timeout 2h -tags=${{
matrix.language }} -parallel 4 . 2>&1 | tee /tmp/gotest.log | gotestfmt
run: cd examples && go test -v -json -count=1 -cover -timeout 2h -tags=${{ matrix.language }} -parallel 4 . 2>&1 | tee /tmp/gotest.log | gotestfmt
strategy:
fail-fast: false
matrix:
Expand Down
3 changes: 1 addition & 2 deletions provider-ci/test-providers/cloudflare/.github/workflows/run-acceptance-tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -153,8 +153,7 @@ jobs:
cd provider && go test -v -json -count=1 -cover -timeout 2h -tags=${{ matrix.language }} -parallel 4 . 2>&1 | tee /tmp/gotest.log | gotestfmt
- name: Run tests
if: matrix.testTarget == 'local'
run: cd examples && go test -v -json -count=1 -cover -timeout 2h -tags=${{
matrix.language }} -skip TestPulumiExamples -parallel 4 . 2>&1 | tee /tmp/gotest.log | gotestfmt
run: cd examples && go test -v -json -count=1 -cover -timeout 2h -tags=${{ matrix.language }} -skip TestPulumiExamples -parallel 4 . 2>&1 | tee /tmp/gotest.log | gotestfmt
- name: Run pulumi/examples tests
if: matrix.testTarget == 'pulumiExamples'
run: cd examples && go test -v -json -count=1 -cover -timeout 2h -tags=${{
Expand Down
3 changes: 1 addition & 2 deletions provider-ci/test-providers/docker/.github/workflows/master.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -200,8 +200,7 @@ jobs:
with:
ssh-private-key: ${{ secrets.PRIVATE_SSH_KEY_FOR_DIGITALOCEAN }}
- name: Run tests
run: cd examples && go test -v -json -count=1 -cover -timeout 2h -tags=${{
matrix.language }} -parallel 4 . 2>&1 | tee /tmp/gotest.log | gotestfmt
run: cd examples && go test -v -json -count=1 -cover -timeout 2h -tags=${{ matrix.language }} -parallel 4 . 2>&1 | tee /tmp/gotest.log | gotestfmt
strategy:
fail-fast: false
matrix:
Expand Down
3 changes: 1 addition & 2 deletions provider-ci/test-providers/docker/.github/workflows/prerelease.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -145,8 +145,7 @@ jobs:
with:
ssh-private-key: ${{ secrets.PRIVATE_SSH_KEY_FOR_DIGITALOCEAN }}
- name: Run tests
run: cd examples && go test -v -json -count=1 -cover -timeout 2h -tags=${{
matrix.language }} -parallel 4 . 2>&1 | tee /tmp/gotest.log | gotestfmt
run: cd examples && go test -v -json -count=1 -cover -timeout 2h -tags=${{ matrix.language }} -parallel 4 . 2>&1 | tee /tmp/gotest.log | gotestfmt
strategy:
fail-fast: false
matrix:
Expand Down
3 changes: 1 addition & 2 deletions provider-ci/test-providers/docker/.github/workflows/release.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -144,8 +144,7 @@ jobs:
with:
ssh-private-key: ${{ secrets.PRIVATE_SSH_KEY_FOR_DIGITALOCEAN }}
- name: Run tests
run: cd examples && go test -v -json -count=1 -cover -timeout 2h -tags=${{
matrix.language }} -parallel 4 . 2>&1 | tee /tmp/gotest.log | gotestfmt
run: cd examples && go test -v -json -count=1 -cover -timeout 2h -tags=${{ matrix.language }} -parallel 4 . 2>&1 | tee /tmp/gotest.log | gotestfmt
strategy:
fail-fast: false
matrix:
Expand Down
3 changes: 1 addition & 2 deletions provider-ci/test-providers/docker/.github/workflows/run-acceptance-tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -187,8 +187,7 @@ jobs:
ssh-private-key: ${{ secrets.PRIVATE_SSH_KEY_FOR_DIGITALOCEAN }}
- name: Run tests
if: matrix.testTarget == 'local'
run: cd examples && go test -v -json -count=1 -cover -timeout 2h -tags=${{
matrix.language }} -skip TestPulumiExamples -parallel 4 . 2>&1 | tee /tmp/gotest.log | gotestfmt
run: cd examples && go test -v -json -count=1 -cover -timeout 2h -tags=${{ matrix.language }} -skip TestPulumiExamples -parallel 4 . 2>&1 | tee /tmp/gotest.log | gotestfmt
- name: Run pulumi/examples tests
if: matrix.testTarget == 'pulumiExamples'
run: cd examples && go test -v -json -count=1 -cover -timeout 2h -tags=${{
Expand Down
Loading