Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test: Delete clusters and collateral after integration tests #238

Merged
merged 3 commits into from
Jun 13, 2024
Merged

test: Delete clusters and collateral after integration tests #238

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
a224546
Add iot ops delete cleanup step to integration tests
c-ryan-k Jun 12, 2024
a0b0b46
remove cleanup param and workflow from int tests and documentation
c-ryan-k Jun 12, 2024
5ad2848
add connected cluster deletion step
c-ryan-k Jun 13, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 7 additions & 22 deletions .github/workflows/int_test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,11 +21,6 @@ on:
type: boolean
required: false
default: false
cleanup:
description: Attempt to cleanup resources after testing
type: boolean
required: false
default: false
secrets:
# required for az login
AZURE_CLIENT_ID:
Expand Down Expand Up @@ -55,11 +50,6 @@ on:
type: boolean
required: false
default: false
cleanup:
description: Attempt to cleanup resources after testing
type: boolean
required: false
default: false

permissions:
# required for OpenID federation
Expand Down Expand Up @@ -312,7 +302,7 @@ jobs:
az iot ops mq get-password-hash -p test
az iot ops asset query -g ${{ env.RESOURCE_GROUP }} --location westus -o table
az iot ops verify-host
- name: "Test cluster deletion"
- name: "Delete Cluster for redeployment"
if: ${{matrix.feature == 'ca-certs'}}
run: |
az iot ops delete --cluster ${{ env.CLUSTER_NAME }} -g ${{ env.RESOURCE_GROUP }} -y
Expand All @@ -336,14 +326,9 @@ jobs:
template-file: ${{ inputs.template-content && 'custom-template.json' || '' }}
csi-config: ${{ matrix.csi-config || ''}}
include-dp: ${{ matrix.include-dp }}

# Optional cleanup job
cleanup:
needs: [test]
if: ${{ github.event.inputs.cleanup == 'true' }}
uses: './.github/workflows/cluster_cleanup.yml'
with:
cluster_prefix: ${{ needs.test.outputs.CLUSTER_PREFIX }}
resource_group: ${{ needs.test.outputs.RESOURCE_GROUP }}
keyvault_prefix: ${{ needs.test.outputs.KV_NAME }}
secrets: inherit
- name: "Delete AIO resources"
run: |
az iot ops delete --cluster ${{ env.CLUSTER_NAME }} -g ${{ env.RESOURCE_GROUP }} -y
- name: "Delete connected cluster"
run: |
az connectedk8s delete --name ${{ env.CLUSTER_NAME }} -g ${{ env.RESOURCE_GROUP }} -y
34 changes: 2 additions & 32 deletions docs/integration-tests.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,11 +23,9 @@ There are, however, some prerequisites and caveats that users should be made awa

- #### Dedicated Resource Group for Testing
You should provide a dedicated resource group for these testing resources.
During the tests, resources will be created that cannot be easily cleaned up and are typically hidden from default Azure Portal UI views.
During the tests, resources will be created that may not be automatically cleaned up and are typically hidden from default Azure Portal UI views.

We also have a scheduled [cleanup workflow](../.github/workflows/cluster_cleanup.yml) that does a best-effort cleanup of all AIO-related resources in a particular resource group - this will attempt to clean up all resources that start with a particular "prefix" or are related to clusters with a similar prefix.

Our tests use `az-iot-ops-test-cluster` for cluster resources and `opskv` for keyvaults. Without parameters, all resources with these prefixes (or that reference custom locations with these prefixes) will be deleted by the cleanup action.
Our tests use `az-iot-ops-test-cluster` prefixes for cluster resources and `opskv` for keyvaults.

### Inputs

Expand All @@ -50,7 +48,6 @@ There are, however, some prerequisites and caveats that users should be made awa
|---|---|
**resource-group** | *The resource group to run tests in*
**custom-locations-oid** | *Custom Locations Object ID - used to enable cluster-connect feature.*
**cleanup** | *An optional boolean switch that decides whether to attempt post-test cleanup rather than waiting for a scheduled cleanup job*

### Example workflow

Expand Down Expand Up @@ -93,33 +90,6 @@ Currently this pipeline does not output values, it simply displays test pass/fai
#### CLI Extension Builds
Currently our pipeline uses the most recent dev branch of the IoT Operations extension to build our extension. The extension repo is cloned from the `dev` branch, the wheel is built from that source, and then added to the agent's CLI extension path.

#### Cleanup
As mentioned [above](#dedicated-resource-group-for-testing), these tests create resources that cannot be easily cleaned up (besides deleting and recreating the entire resource group).

Please use the [cleanup workflow](../.github/workflows/cluster_cleanup.yml) for cleaning up leftover resources.
You can call this workflow automatically after the integration tests by including setting the `cleanup` input to `True`, or you can schedule / invoke the workflow as you wish.

#### **Important**

Running cleanup from the integration test workflow will **only delete resources created in that test run**.

Calling cleanup on a schedule will **delete all resources with our given prefixes**.

Calling the [cleanup workflow](../.github/workflows/cluster_cleanup.yml) as a reusable workflow requires the following inputs and secrets, and similar permissions as described [above](#service-principal-and-federated-pipeline-permissions)

| Input | Description |
|---|---|
**cluster_prefix** | *prefix of clusters to delete*
**resource_group** | *resource group to delete resources in*
**keyvault_prefix** | *prefix of keyvaults to delete*

| Secret | Description |
|---|---|
**AZURE_CLIENT_ID** | *Entra Application client ID*
**AZURE_TENANT_ID** | *Azure Tenant ID*
**AZURE_SUBSCRIPTION_ID** | *Azure Subscription ID*


## Using github actions to independently connect a cluster to ARC and/or deploy AIO

We have two custom actions for [connecting a kubernetes cluster to ARC](../.github/actions/connect-arc/action.yml), and for [deploying AIO resources](../.github/actions/deploy-aio/action.yml).
Expand Down