Integration Testing Proposal

[Ian2012]

This discussion proposes a solution for the integration testing implementation on Harmony.

Requirements

• Everything should run in Github Action in a reasonable time (<20 min) without considering OpenedX initialization itself.
• Every step can be reproduced locally with documentation on how to test it.
• Run multiple scenarios such as:
  • All helmCharts enabled
  • ElasticSearch
  • OpenSearch
  • Running multiple OpenedX installations in the same cluster
• Set up different tests for every cluster/environment to verify domains and deployments are working, etc.

Proposed solution

Kind runs k8s on a cluster and works pretty well for our purposes, this tool is already explored on Aspects integration tests. The POC implemented here #80 proposes a folder structure and a GitHub action workflow to test different configurations of the helm chart in parallel, a template for adding new integration testing scenarios, and an initial scenario to setup and test ElasticSearch.

[Ian2012]

tagging you here as you are probably the ones more interested in this issue. @bradenmacdonald @gabor-boros @jfavellar90

[bradenmacdonald]

I would also add that the integration test code itself should be something that's easy to maintain. If it's complex and fragile, it may not be worth the effort.

[Ian2012]

Following this discussion, are there any specific scenarios we need to cover? The POC already has a test for ElasticSearch but I don't have any other scenarios in mind to test.

[bradenmacdonald]

For Sumac, we'll actually need Meilisearch instead of Elasticsearch :p

[Ian2012]

Once support is added I will add the integration testing scenarios. Will the current implementation cover most of our use cases?

I tested ClickHouse and ElasticSearch and found and resolved an issue with ElasticSearch disabled. We can also merge the integration testing PR and start adding scenarios as part of the new features/fixes.

[Ian2012]

Is there anything blocking this PR, FYI only a small change is needed to the workflow so that any issues on the scripts will trigger a CI failure