Running test dataset on aws fails

[prmac]

Running

nextflow run nf-core/rnaseq -profile test,docker,awsbatch -c aws.config -r 2.0

with the following config file

outdir = 's3://test-core/nf-output/'
workDir = 's3://test-core/nf-work/'
tracedir = "/local/aws/nfcore_test/"

// Docker options
docker {
  enabled = true
  runOptions = '-u $(id -u):$(id -g)'
}

// AWS options
process {
  executor = "awsbatch"
  queue = "default-test-core"
}

aws {
  region = 'eu-west-1'
  batch {
    cliPath = '/home/ec2-user/miniconda/bin/aws'
  }
}

results in the following error

Specify a local tracedir or run without trace! S3 cannot be used for tracefiles.. Expression: params.tracedir.startsWith(s3:)

 -- Check script '/local/aws/.nextflow/assets/nf-core/rnaseq/main.nf' at line: 37 or see '.nextflow.log' file for more details

[drpatelh]

This is indeed quite a problematic bug. We don't normally directly test the awsbatch profile which is why it was missed here. Thanks for reporting! It should be fixed when #510 is merged into dev. Once that happens it would be great if you can try and re-run with -r dev instead to see if all is well again. This fix warrants a minor release but I am hoping to test and fix something else before that so hopefully I will do this in the next week or so 🤞

[drpatelh]

This should be fixed now in #510. Be grateful if you can test with -r dev please 🙂

[prmac]

Thanks, all looks fine now!

[drpatelh]

Thanks. So your tests are working?

[prmac]

Actually, running with -r dev the version of the pipeline is given as v1.4.3dev and I get the following

Error executing process > 'output_documentation'

Caused by:
  Process `output_documentation` terminated with an error exit status (1)

Command executed:

  markdown_to_html.py output.md -o results_description.html

Command exit status:
  1

Command output:
  (empty)

Command error:
  Traceback (most recent call last):
    File "//nextflow-bin/markdown_to_html.py", line 4, in <module>
      import markdown
  ModuleNotFoundError: No module named 'markdown'

Work dir:
  s3://test-core/nf-work/9a/6d74ca247cd85428ab82c4c65585e3

Tip: view the complete command output by changing to the process work dir and entering the command `cat .command.out`
[nf-core/rnaseq]  Pipeline completed with errors

[drpatelh]

Can you try and update your local version of the pipeline? nextflow pull nf-core/rnaseq

[prmac]

Thanks,

I've updated locally and now get.

Specify correct --awsqueue and --awsregion parameters on AWSBatch!. Expression: (params.awsqueue && params.awsregion)

 -- Check script '/home/paul/.nextflow/assets/nf-core/rnaseq/main.nf' at line: 37 or see '.nextflow.log' file for more details

I'm specifying these parameters as I would usually in the config file based on the documentation here
https://www.nextflow.io/docs/latest/awscloud.html#configuration

// AWS options
process {
  executor = "awsbatch"
  queue = "default-test-core"
}

aws {
  region = 'eu-west-1'
  batch {
    cliPath = '/home/ec2-user/miniconda/bin/aws'
  }
}

[drpatelh]

Ok. So we are making progress here.

If you want to use the awsbatch.config available on nf-core/configs then you need to set --awsregion 'eu-west-1' --awsqueue 'default-test-core'. This means you shouldn't need the additional config above at all because the necessary settings should be applied via awsbatch.config.

[prmac]

OK great. All seems to be working now

[drpatelh]

Thanks for your help in testing all of this!