IBF Pipeline DTAP

DTAP Development, Testing, Acceptance and Production is where all software/app developments including data pipelines go through. This approach enables transparent and quality of any development/improvement in the production environment.

Resources

1. Pipeline code are open source in a public repository in GitHub

2. The repository should have at least 2 branches, main and dev correspondingly to production and to development/testing.

3. 2 CICD workflows should be included in the GitHub repository and it should be set up to automize the update to a designated container registry in Azure. One for dev one for main

4. The branch main is strictly for production and main is protected from unauthorized merge. All the tests happens in the branch dev.

5. Logic apps should have at least 2 copies, prod and dev correspondingly to production and to development/testing.

Process

Development and Testing

GitHub branch: dev

Local testing:

• Data pipeline developers work locally on the branch dev

• Testing is done locally and point to to ibf-test.510.global

• Image with commits pushed will automatically be pushed to Azure container registry

Remote testing:

• Through the logic app river-flood-pipeline-dev

• Runs the version of the code on the dev branch

• Points to ibf-test.510.global

Acceptance

• Who: data pipeline developers make PR from dev to main

• The PR reviewed by peers.

Production

GitHub branch: main

Merge to main:

• The PR is approved by peers.

• Who: PR can only be merged by Data specialist a Developer in the IBF System Team

• The PR is merged to the branch main

• Image with commits pushed will automatically be pushed to Azure container registry

Production can only be executed remotely:

• Logic app prod links to this prod image for production

• Logic app prod points to ibf.510.global