Set up AWS megatests

[ewels]

AWS megatests is now running nicely and we’re trying to set up all (most) nf-core pipelines to run a big dataset. We need to identify a set of public data to run benchmarks for the pipeline.

The idea is that this will run automatically for every release of the nf-core/hic pipeline. The results will then be publicly accessible from s3 and viewable through the website: https://nf-co.re/hic/results - this means that people can manually compare differences in output between pipeline releases if they wish.

We need a dataset that is as “normal” as possible, mouse or human, sequenced relatively recently and with a bunch of replicates etc. It can be a fairly large project

I'm hoping that @nservant can help here, but suggestions from anyone and everyone are more than welcome! ✋🏻

In practical terms, once decided we need to:

• Upload the FastQ files to s3: s3://nf-core-awsmegatests/hic/input_data/ (I can help with this)
• Update test_full.config to work with these file paths
• Check .github/workflows/awsfulltest.yml (should be no changes required I think?)
• Merge, and try running the dev branch manually

[nservant]

Hi @ewels

I think we could go for this dataset which are a reference in the field.
https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE96107

However, the sequencing depth is huge ... we are talking about 2.5 Billions reads per sample.
See https://www.ncbi.nlm.nih.gov/sra?term=SRX2636666

So I would only select one of the sra file for test ...
Which sequencing depth sounds reasonnable for aws testing ?

N

[nservant]

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5538188/

[nservant]

https://www.ncbi.nlm.nih.gov/sra?term=SRX2636670