clusthash doesn't seem to use all given threads, and result2flat breaks with "segmentation fault"

[UriNeri]

Expected Behavior

clusthash uses all threads and result2flat produces a complete fasta file (not ending in %)

Current Behavior

clusthash seems to use only 1 of all given threads, and eventually result2flat breaks "segmentation fault"

Steps to Reproduce (for bugs)

Ran from the terminal in the same directory as a contigs fasta file (DNA) (named "cated_sk100.fna"):
THREADS=10
mkdir resultsDB scafDB
mmseqs createdb cated_sk100.fna scafDB/cated_sk100
mmseqs clusthash scafDB/cated_sk100 resultsDB/resultDB --min-seq-id 0.99 --threads $THREADS
mmseqs clust scafDB/cated_sk100 resultsDB/resultDB clusterDB --threads $THREADS
mmseqs result2repseq scafDB/cated_sk100 clusterDB DB_clu_rep
mmseqs result2flat scafDB/cated_sk100 scafDB/cated_sk100 DB_clu_rep scafs_reps.fasta --use-fasta-header

When "Compute 1 unique hashes." is printed, there are 10 resultsDB files and 10 resultDB.index files, however, only one (resultDB.index.7) is getting larger with time (and is > 0 in size). Meanwhile only one thread seems to be utilized (around 8% of the total 10 threads given).
When the clusthash finishes there is one resultsDB.index file, and 10 resultsDB files, 8 with zero size, and resultsDB.index7 and resultsDB.index both with the same size). After this, the process breaks in the last command:
mmseqs result2flat scafDB/cated_sk100 scafDB/cated_sk100 DB_clu_rep scafs_reps.fasta --use-fasta-header
With the message:
`result2flat scafDB/cated_sk100 scafDB/cated_sk100 DB_clu_rep scafs_reps.fasta --use-fasta-header

MMseqs Version: 48a037a
Use fasta header true
Verbosity 3

[1] 18252 segmentation fault (core dumped) mmseqs result2flat scafDB/cated_sk100 scafDB/cated_sk100 DB_clu_rep`

MMseqs Output (for bugs)

Which output should I upload?

Context

I'm trying to remove redundancy by collapsing sequences that are either highly similar (99%) or are also contained within longer sequences from other fasta entries in the file. This fasta file size <1gb but I first tried to run this process on a >80gb file on remote compute node and was concerned when I saw the job was using only a small part of the resources.
Not part of this issue but realted; also tried to do the same thing with a large protein file but I get invalid fasta entry errors (maybe because of the "*" marking STOPs left by the ORF predictor, but that wouldn't happen in the nucleic acid file example above).

Your Environment

• Git commit used:
  I tried on my personal machine and a compute node (PBS), similar behaviour in both.
  Personal machine MMseqs2 Version: 48a037a.
  Server MMseqs2 Version: 2a8c5f0.
• Which MMseqs version was used: Statically-compiled
• Server specifications:
  Server: (2a8c5f0)
  CPU: Intel(R) Xeon(R) Platinum 8168
  Memory: 366 GB
  Personal machine: (48a037a)
  CPU: Intel Core i7-8700 6-Core model: bits: 64 type: L2 cache: 12.0 MiB
  Memory: 15.33 GB
• Operating system and version:
  Personal machine: Linux Mint 19.2 Tina Kernel: 4.15.0-72-generic x86_64;
  Server: Linux 3.10.0-693.el7.x86_64

Thanks for developing and maintaining this totally amazing tool !

[milot-mirdita]

I reviewed the code and found multiple possible issues with the module for smaller datasets. I'll try to finish up the refactoring either later today or tomorrow. Thanks for the bug report.

We are still doing the free sticker thing (see https://twitter.com/thesteinegger/status/1201076220957315074). If you want a set send me your address at milot at mirdita dot de.

[UriNeri]

Thanks for the super quick reply !
Not sure I understand what you mean by "smaller datasets"
I tried to run this on the server with the 90gb file and locally with the small file (a 0.6gb subset of the sequences in the larger one). Both had the issue (but only the local run finished the clusthash step)

[martin-steinegger]

@UriNeri clusthash can not find contained sequences. The best would be to used linclust

 mmseqs easy-linclust cated_sk100.fna cated_sk100.clu tmp --min-seq-id 0.99 -c 1.0 --cov-mode 1 --kmer-per-seq-scale 0.4

[UriNeri]

@martin-steinegger @milot-mirdita
Makes much more sense indeed! Thanks ! will try soon

[milot-mirdita]

Disregarding how much biological sense it makes, would you mind rerunning the clusthash workflow above with the latest commit? I refactored some code and want to know if it improved the performance.

Also the crash in result2flat was probably because of the wrong input database (clusterDB instead of DB_clu_rep). It should be:

mmseqs result2flat scafDB/cated_sk100 scafDB/cated_sk100 clusterDB scafs_reps.fasta --use-fasta-header

[UriNeri]

@milot-mirdita No problem!
MMseqs2 version: 55bcdd3
Adjusted workflow, same input as before; clusthash finishes in seconds, the files previously of zero size are around the same size now. Now result2repseq is the "slowest" step.
The last step:
mmseqs result2flat scafDB/cated_sk100 scafDB/cated_sk100 clusterDB scafs_reps.fasta --use-fasta-header
Create the scafs_reps.fasta file but it's in this format:
>header
[number]
Any way to transform this into regular fasta? ( >header \n [sequence]...)
(Although I'll use linclust anyway as Martin's suggested)
Thanks again !

[UriNeri]

@martin-steinegger
Tried the easy-linclust as suggested on the server described before, with the latest version ( 55bcdd3 )
I receive the following error:
Time for merging to clu_rep: 0h 0m 21s 931ms
Time for processing: 0h 0m 39s 80ms
result2flat tmp/6767229871110119818/input tmp/6767229871110119818/input tmp/6767229871110119818/clu_rep tmp/6767229871110119818/rep_seq.fasta --use-fasta-header -v 3

tmp/6767229871110119818/easycluster.sh: line 38: 4147 Segmentation fault "$MMSEQS" result2flat "${TMP_PATH}/input" "${TMP_PATH}/input" "${TMP_PATH}/clu_rep" "${TMP_PATH}/rep_seq.fasta" --use-fasta-header ${VERBOSITY_PAR}
Error: result2flat died

[martin-steinegger]

Could you please provide the full log? Is it possible to provide your input?

[UriNeri]

@martin-steinegger
I didn't delete the ./tmp/ folder, is the full log in there? or do you mean the stdout ?
About the input, please email me : uri dot neri at gmail dot com
Thanks !

[martin-steinegger]

@UriNeri the log is only written to stdout. We do not store a copy of the log in the temp directory. So you probably need to rerun the whole job. If you use the same tmp folder and command then it will just perform the last step.

[UriNeri]

@martin-steinegger I re-run the easy-linclust command on a different input (in email).
This time kept the stdout and stderr

stderr.txt
stdout.txt

[martin-steinegger]

@UriNeri thank you for sending me the input. I could reproduce the issue and fix it.

[UriNeri]

@martin-steinegger thank you !
Can confirm that using the latest commit the issue is fixed.
Thanks again for this truly amazing tool - absolutely fantastic !