Can't complete test_full, Error executing process > 'NFCORE_MAG:MAG:PROKKA

[medmaca]

Description of the bug

The pipeline runs until it gets to process NFCORE_MAG:MAG:PROKKA, it then fails with the error shown below.

Command used and terminal output

nextflow run 'https://github.com/nf-core/mag'
		 -name distraught_baekeland_2
		 -params-file 'https://api.tower.nf/ephemeral/****.json'
		 -with-tower
		 -r ****
		 -profile ebi_codon_slurm,apptainer,test_full
		 -resume ****


Workflow execution completed unsuccessfully
The exit status of the task that caused the workflow execution to fail was: 2

Error executing process > 'NFCORE_MAG:MAG:PROKKA (MEGAHIT-MetaBAT2-CAPES_S7.13)'

Caused by:
  Process `NFCORE_MAG:MAG:PROKKA (MEGAHIT-MetaBAT2-CAPES_S7.13)` terminated with an error exit status (2)

Command executed:

  prokka \
      --metagenome \
      --cpus 2 \
      --prefix MEGAHIT-MetaBAT2-CAPES_S7.13 \
       \
       \
      MEGAHIT-MetaBAT2-CAPES_S7.13.fa
  
  cat <<-END_VERSIONS > versions.yml
  "NFCORE_MAG:MAG:PROKKA":
      prokka: $(echo $(prokka --version 2>&1) | sed 's/^.*prokka //')
  END_VERSIONS

Command exit status:
  2

Command output:
  (empty)

Command error:
  [20:46:39] This is prokka 1.14.6
  [20:46:39] Written by Torsten Seemann <torsten.seemann@gmail.com>
  [20:46:39] Homepage is https://github.com/tseemann/prokka
  [20:46:39] Local time is Wed Dec 13 20:46:39 2023
  [20:46:39] You are not telling me who you are!
  [20:46:39] Operating system is linux
  [20:46:39] You have BioPerl 1.7.8
  Argument "1.7.8" isn't numeric in numeric lt (<) at /usr/local/bin/prokka line 259.
  [20:46:39] System has 96 cores.
  [20:46:39] Will use maximum of 2 cores.
  [20:46:39] Annotating as >>> Bacteria <<<
  [20:46:39] Generating locus_tag from 'MEGAHIT-MetaBAT2-CAPES_S7.13.fa' contents.
  [20:46:39] Setting --locustag OAGBGAJL from MD5 8a0b0a3511a2ca09357ba5d47b898b28
  [20:46:39] Creating new output folder: MEGAHIT-MetaBAT2-CAPES_S7.13
  [20:46:39] Running: mkdir -p MEGAHIT\-MetaBAT2\-CAPES_S7\.13
  [20:46:39] Using filename prefix: MEGAHIT-MetaBAT2-CAPES_S7.13.XXX
  [20:46:39] Setting HMMER_NCPU=1
  [20:46:39] Writing log to: MEGAHIT-MetaBAT2-CAPES_S7.13/MEGAHIT-MetaBAT2-CAPES_S7.13.log
  [20:46:39] Command: /usr/local/bin/prokka --metagenome --cpus 2 --prefix MEGAHIT-MetaBAT2-CAPES_S7.13 MEGAHIT-MetaBAT2-CAPES_S7.13.fa
  [20:46:39] Appending to PATH: /usr/local/bin
  [20:46:39] Looking for 'aragorn' - found /usr/local/bin/aragorn
  [20:46:39] Determined aragorn version is 001002 from 'ARAGORN v1.2.38 Dean Laslett'
  [20:46:39] Looking for 'barrnap' - found /usr/local/bin/barrnap
  [20:46:39] Determined barrnap version is 000009 from 'barrnap 0.9'
  [20:46:39] Looking for 'blastp' - found /usr/local/bin/blastp
  [20:46:39] Determined blastp version is 002012 from 'blastp: 2.12.0+'
  [20:46:39] Looking for 'cmpress' - found /usr/local/bin/cmpress
  [20:46:39] Determined cmpress version is 001001 from '# INFERNAL 1.1.4 (Dec 2020)'
  [20:46:39] Looking for 'cmscan' - found /usr/local/bin/cmscan
  [20:46:39] Determined cmscan version is 001001 from '# INFERNAL 1.1.4 (Dec 2020)'
  [20:46:39] Looking for 'egrep' - found /bin/egrep
  [20:46:39] Looking for 'find' - found /usr/bin/find
  [20:46:39] Looking for 'grep' - found /bin/grep
  [20:46:39] Looking for 'hmmpress' - found /usr/local/bin/hmmpress
  [20:46:39] Determined hmmpress version is 003003 from '# HMMER 3.3.2 (Nov 2020); http://hmmer.org/'
  [20:46:39] Looking for 'hmmscan' - found /usr/local/bin/hmmscan
  [20:46:39] Determined hmmscan version is 003003 from '# HMMER 3.3.2 (Nov 2020); http://hmmer.org/'
  [20:46:39] Looking for 'java' - found /usr/local/bin/java
  [20:46:39] Looking for 'makeblastdb' - found /usr/local/bin/makeblastdb
  [20:46:40] Determined makeblastdb version is 002012 from 'makeblastdb: 2.12.0+'
  [20:46:40] Looking for 'minced' - found /usr/local/bin/minced
  [20:46:40] Coult not parse version from '#'

Work dir:
  /mnt/scratch/projects/biol-tf-2018/nextflow/work/a3/8f571342697069af592e08e37e8153

Tip: view the complete command output by changing to the process work dir and entering the command `cat .command.out`

Relevant files

nf-5IBlscu8mRjHeQ.log
nf-5IBlscu8mRjHeQ.txt

System information

• nf-core/mag version e728900 (2.5.1)
• Nextflow version 23.10.0.5891
• Apptainer V1.2.5
• Tower Agent version 0.5.0 (build d758993)
• Executor: slurm
• Hardware HPC -- submitting pipeline from tower.nf
• OS Rocky Linux, VERSION="8.8 (Green Obsidian)",ID_LIKE="rhel centos fedora")

The pipeline is run with the following profiles: ebi_codon_slurm, apptainer, test_full. I'm only providing an output directory I'm leaving everything else as set in test_full.

[jfy133]

That's very strange... I've not seen that before.

Googling the error looks like it's something to do with Prokka (rather than the pipeline):

tseemann/prokka#557

Could it be your HPC is somehow adding some extra text to stdout logs etc.?

[medmaca]

Other pipelines run fine. However, I'll try to have a root around to see what I can find.

This thread: https://nfcore.slack.com/archives/CE9MS66BS/p1681840960696719 contains reports that others are having the same issue.

[jfy133]

Do you mean other pipelines run fine with the prokka module?

Thanks for pointing to the slack threead!

[medmaca]

Do you mean other pipelines run fine with the prokka module?

No, sorry, I mean that in general nf-core pipelines work (atacseq, rnaseq etc), it must be something specific to the prokka module, or the way that its container is being handled.

[jfy133]

Yes exactly :/. You could also try running manually prokka from different contexts and see if you get the same error.

If you go into the working directory that the error reports, you should see a hidden file called .command.sh that has the bash command nextflow is executing.

So you could try running with a locally compiled/downloaded version, with a version installed with conda etc.

If it's neither of those then it is something specific to the container I guess and we will need to investigate it further

[jfy133]

We will eventually replace/update all modules anyway which should hopfeully fix this, and as this seems to be possibly be an edge case as it's not being reported by others I will close this for now.