Skip to content

Commit

Permalink
[POWEROPS-2598] feat: setup prod toolkit configs (#390)
Browse files Browse the repository at this point in the history
* feat: add prod toolkit configs

* feat: add prod toolkit configs

* feat: change type in the toolkit config to use staging instead of dev

* docs: update docs for auth env variables

---------

Co-authored-by: emiliedahl <56264522+emiliedahl@users.noreply.github.com>
  • Loading branch information
nodegard and emiliedahl authored Jul 22, 2024
1 parent 05739ae commit 11eef1e
Show file tree
Hide file tree
Showing 7 changed files with 114 additions and 31 deletions.
101 changes: 74 additions & 27 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,24 +12,27 @@ You can control which module is deployed by setting the `selected_module_and_pac

```yaml
...
selected_modules_and_packages:
- power_model_v1
selected_modules_and_packages:
- power_model_v1
...
```

This will only deploy the `power_model_v1` module.

## Authentication

To authenticate with the `cognite-toolkit` you will have to use a `.env` file in the `power-ops-sdk` root directory,
which should look like this:

```dotenv
LOGIN_FLOW=client_credentials
CDF_CLUSTER=<CLUSTER>
CDF_URL=https://<CLUSTER>.cognitedata.com
CDF_PROJECT=<PROJECT>
IDP_TENANT_ID=431fcc8b-74b8-4171-b7c9-e6fab253913b
IDP_TENANT_ID=<TENANT_ID>
IDP_CLIENT_ID=***
IDP_CLIENT_SECRET=***
IDP_TOKEN_URL="https://login.microsoftonline.com/431fcc8b-74b8-4171-b7c9-e6fab253913b/oauth2/v2.0/token"
IDP_TOKEN_URL="https://login.microsoftonline.com/<TENANT_ID>/oauth2/v2.0/token"
SENTRY_ENABLED=false
```

Expand All @@ -56,14 +59,14 @@ Many changes to containers may result in changes to the related views and data m
versioned in order to avoid breaking changes in the consumption layer.

If a change requires a version change, the following version system should be used (MAJOR.MINOR.PATCH):

- Test changes to data model in `power-ops-dev` with a version bump to the PATCH
- Test changes to data model along with related resources (functions, workflows, shop) in `power-ops-staging` with a version bump to MINOR
- Changes deployed to customer environment (dev & prod) should bump the MAJOR version
- Once deployed to the next environment the previous environment will "rest"
- If testing is done in `power-ops-dev` with version 1.0.3, then those models will be deployed to `power-ops-staging`
as 1.1.0. When testing is done in staging and changes get deployed to customer environments the version would become 2


## Local development process

1. Make the relevant changes to the `.yaml` files in the `data_models` folder.
Expand All @@ -74,39 +77,80 @@ If a change requires a version change, the following version system should be us
6. Create a PR and request feedback on your suggested changes.
7. Once PR is approved inform the team that the changes will be deployed to `power-ops-staging`.
8. Deploy changes manually to **power-ops-staging** ensuring you've used the correct version if needed.
9. Regenerate the SDK for the data model changes by calling `python scripts/pygen_generate_clients.py`.
9. Regenerate the SDK for the data model changes by calling `python scripts/pygen_generate_clients.py`.
10. Bump the SDK version in `pyproject.toml` and `cognite/powerops/_version.py`.
11. Update the `CHANGELOG.md` with the changes made.
12. Get a second approval on the PR.

## Manual Deployment

1. Check which environment to deploy to, they are defined in the `cognite` folder in the `config.<ENV>.yaml` files
1. config.dev.yaml
1. ENV=dev
2. CDF_PROJECT=power-ops-dev
2. config.staging.yaml
1. ENV=staging
2. CDF_PROJECT=power-ops-staging
3. config.lyse-dev.yaml
1. ENV=lyse-dev
2. CDF_PROJECT=lyse-dev
4. config.heco-dev.yaml
1. ENV=heco-dev
2. CDF_PROJECT=heco-dev
1. **config.dev.yaml**
1. CDF_PROJECT: `power-ops-dev`
2. ENV: `dev`

```bash
export ENV="dev"
```

2. **config.staging.yaml**
1. CDF_PROJECT: `power-ops-staging`
2. ENV: `staging`

```bash
export ENV="staging"
```

3. **config.lyse-dev.yaml**
1. CDF_PROJECT: `lyse-dev`
2. ENV: `lyse-dev`

```bash
export ENV="lyse-dev"
```

4. **config.lyse-prod.yaml**
1. CDF_PROJECT: `lyse-prod`
2. ENV: `lyse-prod`

```bash
export ENV="lyse-prod"
```

5. **config.heco-dev.yaml**
1. CDF_PROJECT: `heco-dev`
2. ENV: `heco-dev`

```bash
export ENV="heco-dev"
```

6. **config.heco-prod.yaml**
1. CDF_PROJECT: `heco-prod`
2. ENV: `heco-prod`

```bash
export ENV="heco-prod"
```

2. Ensure the credentials in your `.env` file point to the same environment you want to deploy to
3. Build the configurations and remove existing files in the build directory:

```bash
cdf-tk build toolkit/ --env ENV
cdf-tk build toolkit/ --env=$ENV
```

4. Dry-run the deployment to see what changes would be made:

```bash
cdf-tk deploy --env=ENV --dry-run
```
cdf-tk deploy --env=$ENV --dry-run
```

5. Deploy the changes if the environment and changes your making won't impact other people's work:

```bash
cdf-tk deploy --env=ENV
```
cdf-tk deploy --env=$ENV
```

### WARNING: Full manual redeploy steps

Expand All @@ -120,10 +164,13 @@ DO NOT perform these steps unless it has been aligned with the full team and a s

1. Follow the same 1-3 steps from the [Manual Deployment](#manual-deployment) steps
2. Dry-run the deployment to see what changes would be made with the `drop` flags:

```bash
cdf-tk deploy --drop-data --drop --env=ENV --dry-run
```
cdf-tk deploy --drop-data --drop --env=$ENV --dry-run
```

3. Deploy the changes if the environment and changes your making won't impact other people's work:

```bash
cdf-tk deploy --drop-data --drop --env=ENV
```
cdf-tk deploy --drop-data --drop --env=$ENV
```
4 changes: 3 additions & 1 deletion toolkit/config.dev.yaml
Original file line number Diff line number Diff line change
@@ -1,7 +1,9 @@
environment:
name: dev
project: power-ops-dev
type: dev
# type is set to staging since this field is only used for raising error if project defined here
# doesn't match the environment variables (valid options are dev, staging, prod)
type: staging
selected:
- power_model_v1

Expand Down
4 changes: 3 additions & 1 deletion toolkit/config.heco-dev.yaml
Original file line number Diff line number Diff line change
@@ -1,7 +1,9 @@
environment:
name: heco-dev
project: heco-dev
type: dev
# type is set to staging since this field is only used for raising error if project defined here
# doesn't match the environment variables (valid options are dev, staging, prod)
type: staging
selected:
- power_model_v1

Expand Down
14 changes: 14 additions & 0 deletions toolkit/config.heco-prod.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
environment:
name: heco-prod
project: heco-prod
type: prod
selected:
- power_model_v1

variables:
custom_modules:
power_model_v1:
power_ops_type_space: "power_ops_types"
power_ops_instance_space: "power_ops_instances"
power_ops_models: "power_ops_core"
version: "1"
4 changes: 3 additions & 1 deletion toolkit/config.lyse-dev.yaml
Original file line number Diff line number Diff line change
@@ -1,7 +1,9 @@
environment:
name: lyse-dev
project: lyse-dev
type: dev
# type is set to staging since this field is only used for raising error if project defined here
# doesn't match the environment variables (valid options are dev, staging, prod)
type: staging
selected:
- power_model_v1

Expand Down
14 changes: 14 additions & 0 deletions toolkit/config.lyse-prod.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
environment:
name: lyse-prod
project: lyse-prod
type: prod
selected:
- power_model_v1

variables:
custom_modules:
power_model_v1:
power_ops_type_space: "power_ops_types"
power_ops_instance_space: "power_ops_instances"
power_ops_models: "power_ops_core"
version: "1"
4 changes: 3 additions & 1 deletion toolkit/config.staging.yaml
Original file line number Diff line number Diff line change
@@ -1,7 +1,9 @@
environment:
name: dev
project: power-ops-staging
type: dev
# type is set to staging since this field is only used for raising error if project defined here
# doesn't match the environment variables (valid options are dev, staging, prod)
type: staging
selected:
- power_model_v1

Expand Down

0 comments on commit 11eef1e

Please sign in to comment.