CI/CD: Automated testing with pytest and mocks

[evanroyrees]

Continuous Integration / Continuous Deployment (CI/CD)

Provide unit tests and integration tests using pytest and a mock library (could be built-in unittest.mock, pytest.monkeypatch, pytest-mock, etc.)

After implementation of these tests:

1. 🛠️ 🧰 the pipeline can be continuously tested (automatically) with each pull request and merge. Multiple CI services such as travis-CI, circle-CI, etc. offer their service for free if you are an open source project.
2. ✅ ❗ ❌ These tests can then be reflected in each PR, similar to how our documentation is now checked with each PR.
3. 🐛 This will save us the effort of having to manually look through any changes in the code for possible introductions of bugs to the pipeline.
4. 🎨 Writing these tests requires a slight shift in mindset where exceptions need to be considered before-hand.

Resources:

• Raphael Pierzina pytest blog post (part 1) - link
  I would highly recommend starting here if you are unfamiliar with pytest. This is a wonderful post with many practice examples.
• medium post using pytest and pytest-mock - link
• good pytest practices - link
• realpython.com - mocking
• schedule lib tests
• intro to mocking in python
• (medium post) Showing some basic mocking usage
• Python Mocking 101: Fake It Before You Make It
• (medium post) A few simple examples explaining when and how to use mock
• mocking API calls (nice example of calling external tools)
• gist showcasing some python mocking
• Detailed examples of mocking in python
• blog post discussing mocking regular functions (and a mention of a common gotcha at the beginning)
• hackersoft.io post on mocking (includes section on mocking calls to third-party tool)
• 40 examples showcasing the mock.patch method
• (video) Testing the untestable: a beginner’s guide to mock objects
• Another guide on mocking in python
• pytest: How to mock in Python (blog post)

Legend

Work-in-progress (WIP) 🛠️

Data for coverage and quick CI/CD

Contigs containing annotations:

• Eukarya, Viruses, Bacteria, Archaea, Unclassified - taxonomy-specific
• Bacterial markers - marker-specific (43 contigs contain all unique 139 markers)
• Archaeal markers - marker-specific (8 contigs contain all unique 164 markers)
• Contigs with pre-assigned target genomes - binning assessment (determinism)

Associated data to mock for CI/CD

• blast output - skip diamond blastp
• blast hits assigned taxids - skip parsing accession2taxid.gz
• Read alignments to contigs - skip read alignment
• ncbi databases - skip download/formatting of databases
• marker database - skip download/formatting of marker databases
• hmmscan table - skip search with hmmscan

Unit test implementations

Common

• test_metagenome.py 🛠️
• test_metabin.py
• test_coverage.py
• test_kmers.py
• test_markers.py

External

• test_diamond.py
• test_prodigal.py
• test_bowtie.py
• test_samtools.py
• test_hmmer.py
• test_bedtools.py

Config

• test_user.py
• test_project.py
• test_databases.py
• test_environ.py

Taxonomy

• test_lca.py
• test_majority_vote.py
• test_ncbi.py
• test_vote.py

Binning

• test_recursive_dbscan.py
• test_unclustered_recruitment.py
• test_summary.py

Integration test implementations

• test_imports.py
• test_entrypoints.py
• test docker
• test bioconda build

[evanroyrees]

Note: For adding pre-commits with CI:

💚 https://pre-commit.com/#usage-in-continuous-integration