Use unbound queue for transfer thread pool

[pditommaso]

This PR changes the default setting ThreadPoolBuilder class so that:

• by default uses an unbound queue instead in order to prevent the triggering of the rejection policy when exceeding the queue size
• use BlockingBlockingQueue when the queue size is specified
• use the AbortPolicy as rejection policy to prevent the issue reported here
• default to -1(unbound) queue size in the ThreadPoolManager

Solves #5363

[netlify]

✅ Deploy Preview for nextflow-docs-staging ready!

Name	Link
🔨 Latest commit	a444d5f
🔍 Latest deploy log	https://app.netlify.com/sites/nextflow-docs-staging/deploys/67922197ae29d00008a1090d
😎 Deploy Preview	https://deploy-preview-5700--nextflow-docs-staging.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[jorgee]

I have reproduced the error with this small pipeline that generates a directory with some files and a configuration with a small number of threads and queue size. It was failing with the master and working with this PR. I was thinking of adding it as validation or AWS test, but it could hang the CI if it fails. @pditommaso what do you think? there is a timeout for the CI build, isn't it?

main.nf

params.outdir = 's3://jorgee-eu-west1-test1/results/'
params.num_files = 10
params.file_mb = 10

process gen_data {
        publishDir "${params.outdir}", mode: 'copy'
        input:
        val(x)
        output:
        path 'data'

        script:
        """
        mkdir data
        for i in {1..${params.num_files}}
        do
            dd if=/dev/urandom of=data/chunk_${x}_\$i count=${params.file_mb} bs=1M
        done
        """
}

workflow {
        Channel.of(1) | gen_data
}

nexflow.config:

threadPool.S3TransferManager.minThreads = 1
threadPool.S3TransferManager.maxThreads = 2
threadPool.S3TransferManager.maxQueueSize = 2

[pditommaso]

It was failing with the master and working with this PR

well done!

I was thinking of adding it as validation or AWS test, but it could hang the CI if it fails

not sure it's needed