[Story] Refactor CI/CD to support e2e and deployment to AWS

[tschaffter]

What product(s) is this story for?

Sage Monorepo

As a user, I want

No response

Description

We need to:

• Refactor the CI/CD workflow so make it more modular
  • The current implementation involve a long file and lot of duplicated code that could lead to issues.
• We need to run tasks that have access to project-specific secrets
  • The secrets should only be exposed to the projects and tasks that needs them.
  • Example: schematic-api e2e test requires secrets to access Synapse
    • Should we use Synapse secrets or use another way to authenticate (OIDC?)
• The deployment of OC to AWS should be done through the CI/CD workflow
  • OC uses the CDK for Terraform (CDKTF) to deploy the stack.
  • Need to identify how to deploy the following environments staging and production
  • Additional environment development and testing should be considered (see Microsoft docs)

Other considerations:

• We are developing inside a monorepo
  • Only a project affected by a change should be processed by the CI/CD and deployed
  • Creating separate monorepo CI/CD pipelines with GitHub Actions
• We use the Forking workflow
  • GH secrets are not available to PRs opened from forks
  • Enabling maintainers to manually allow workflows would be nice, but that seems to be possible only for private forks?

References:

• AWS CodePipeline (CI / CD) Tutorial with CDK (video)
  • Deployment of the different stages is defined in the CodePipeline
• Multi-Stack Deployments with CDK for Terraform (video)

Acceptance criteria

TBA

Tasks

• Identify the deployment environment OC needs
• Review Creating separate monorepo CI/CD pipelines with GitHub Actions
• Review GitHub Actions: Reusable workflows are generally available
• Identify if schematic e2e execution can be limited to only Schematic contributors? Otherwise, can it be limited to a subset of schematic developers or a different set of users?
• TBA

Anything else?

cc @andrewelamb

Have you linked this story to a GitHub Project?

• I have linked this story to a GitHub Project and set its metadata.

[tschaffter]

About environments

• You should, however, conduct your performance testing in a physical environment with hardware and software that is identical to the production environment.

Conclusion

• The development environment is the environment used by the developers (N developers => N development environments). The development environment can be local or on AWS, for instance a stack deployed by each developers.

• The testing environment is where testing suites are run (unit, integration, e2e). Tests could also measure the performance of the stack, for instance, maximum sustainable throughput (MST) and maximum sustainable tracking throughput. At least e2e and performance-related tests should be performed in an environment almost identical to the production environment.

• The staging environment is where QA testers review the app. Once the staging version is approved, the production environment can be updated.

• The production environment is the "live" environment that hosts the app.

The environments that OC should deploy are:

• testing (optional)
• staging
• production

[tschaffter]

Branching strategies

Trunk Base Development Vs Feature Branching (Github Flow)

• Trunk Base Development Branching Model (single, deployable branch)
  • Main is always deployable, which is not the case for OC
• Feature Branching (including Github Flow)

Feature	Trunk Base Development	Feature Branching (Github Flow)
Main branch	Always deployable	Not always deployable
Feature branches	Short-lived, optional	Long-lived, required
Merge frequency	Frequent	Infrequent
CI/CD emphasis	Strong	Weaker
Complexity	Simpler	More complex
Merge conflicts	Less likely	More likely

Forking Workflow

• Forking workflow is almost the same as the feature branch but it does not create the branch for developing some features, it forks the repository.

Gitlab Branches

• The most difference between GitLab Flow and GitHub Flow are the environment branches having in GitLab Flow (e.g. staging and production) because there will be a project that isn’t able to deploy to production every time you merge a feature branch.
• An environmental branch is a branch that contains commits that apply to the source code to reconfigure the product to run in different environments.

Example workflow:

1. developer creates your own branch ( Switch to master branch then pull all changes and finally create a feature branch
2. developer staging your branch ( git add and commit changes )
3. developer integrating his feature branch into environment branches
   • How is main updated?
4. usually, CI/CD is used to deploy in developer, staging, tests, and production environment

References

• CI/CD: Github/GitLab Branching Strategies (Medium)

[tschaffter]

Review Creating separate monorepo CI/CD pipelines with GitHub Actions

This is an article about creating separate CI/CD pipelines for microservices in a monorepo using GitHub Actions. It discusses the benefits of using microservices and why deploying them independently is important. It also explains how traditional monorepo CI/CD pipelines can’t handle individual deployments. The article then introduces a solution using GitHub Actions workflows to create separate CI/CD pipelines for each microservice in a monorepo. Finally, it provides an example project with instructions on how to run it.

About Metarepo:

Metarepos combine the convenience of a monorepo with the flexibility of separate code repositories. But still, having separate code repositories adds complexity, and it would be better if we could stick to the simpler monorepo for as long as possible.

[tschaffter]

Enable approval for workflow runs from forks in GitHub Actions

Current settings:

Google Bard:

Here's a guide on how to run e2e tests with GitHub Workflows in a forking workflow:

1. Create or Update a Workflow File:

• In your repository's .github/workflows directory, create or update a YAML file defining your workflow.
• Specify the events that will trigger the workflow, such as pull requests or pushes to specific branches.

2. Configure Actions for E2E Tests:

• Within the workflow file, use GitHub Actions to set up the environment for running your tests:
  • Install dependencies for your testing framework.
  • Build your application if necessary.
  • Start any required services or emulators.
• Choose appropriate actions for your testing needs (e.g., actions/setup-node, actions/setup-java, etc.).

3. Run the E2E Tests:

• Use a dedicated action for executing your test suite.
• Consider options like:
  • actions/github-script: Directly run test commands within the workflow.
  • Framework-specific actions: Leverage actions tailored for your testing framework (e.g., cypress-io/github-action for Cypress).

4. Customize Actions for Forked Repositories:

• Approval Requirements: Enable approval for workflow runs from forks to ensure security and prevent unwanted actions.
• Conditional Logic: Use conditional statements to control workflow execution based on the repository context (e.g., github.repository_owner).

5. Example Workflow File:

name: E2E Tests

on: [pull_request]

jobs:
  test:
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-node@v3
        with:
          node-version: 16
      - run: npm install
      - run: npm test:e2e

Additional Considerations:

• Environment Variables: Set any necessary environment variables for your tests, including those for external services or test data.
• Artifacts: Save test reports and logs as artifacts for inspection and analysis.
• Caching: Optimize workflow performance by caching dependencies and build outputs.
• Security: Protect sensitive information like API keys using GitHub Secrets.
• Matrix Builds: Test across multiple configurations using matrix builds.

[tschaffter]

Enable workflows from forks to access secrets with labels

See https://stackoverflow.com/a/77007901

[tschaffter]

Added to Sprint 24.2

[tschaffter]

Added to Sprint 24.3

[tschaffter]

Conclusion

We have a general solution to run e2e tests as part of PRs that originate from forks.

The OC team and Platform are now teaming up to deploy OC to AWS in a scalable way. The Platform team will be responsible for deciding how to deploy OC to AWS.