The output files don't update

[Betula0422]

Hi there,
I have been successfully run the script with your test dataset within two days. But when I use my data, my output files have not been updated for a week, but there are no errors in the log file. I can't figure out what happened. There are 46,200,228 contigs in my data. Is it because the data is too large? Looking forward to your reply.

Here is my script:

#!/bin/bash
source activate base
conda activate vs
virsorter run -w /data2/Betula/virus/virsorter2/test.out2 -i /data1/wxh/contigs.rnm.fa -d ~/database/virsorter2/db --min-length 5000 -j 28 --include-groups dsDNAphage,ssDNA --min-score 0.5 --use-conda-off all

Here is my log:

[2024-04-30 15:11 INFO] VirSorter 2.2.4
[2024-04-30 15:11 INFO] /data2/Betula/miniconda3/envs/vs/bin/virsorter run -w /data2/Betula/virus/virsorter2/test.out -i /data1/wxh/contigs.rnm.fa -d /data2/Betula/database/virsorter2/db --min-length 1500 -j 28 --use-conda-off all
[2024-04-30 15:11 INFO] Using /data2/Betula/miniconda3/envs/vs/VirSorter2/virsorter/template-config.yaml as config template
[2024-04-30 15:11 INFO] conig file written to /data2/Betula/virus/virsorter2/test.out/config.yaml

[2024-04-30 15:11 INFO] Executing: snakemake --snakefile /data2/Betula/miniconda3/envs/vs/VirSorter2/virsorter/Snakefile --directory /data2/Betula/virus/virsorter2/test.out --jobs 28 --configfile /data2/Betula/virus/virsorter2/test.out/config.yaml --latency-wait 600 --rerun-incomplete --nolock --quiet all
[2024-04-30 15:18 INFO] # of seqs < 1500 bp and removed: 43178270
[2024-04-30 15:18 INFO] # of circular seqs: 1017
[2024-04-30 15:18 INFO] # of linear seqs : 3020941
[2024-04-30 15:19 INFO] Finish spliting circular contig file with common rbs
[2024-04-30 15:20 INFO] Finish spliting linear contig file with common rbs
[2024-04-30 16:26 INFO] Step 1 - preprocess finished.
[2024-05-05 17:10 INFO] Step 2 - extract-feature finished.

[jiarong]

Hi, your data is quite large. I see the step 2 (the longest step) was finished a week ago. The last step should NOT take this long. Can you check which process it is stuck at? (using top command). Also note that if you this run errors out, you can just rerun the same command, VirSorter2 can pick up where it stops and continue.

[Betula0422]

Thank you for your reply! I don't know why I can't reply at github. I used "top" command. The results are as follows. top - 21:14:47 up 43 days, 7:49, 2 users, load average: 28.00, 28.04, 28.05 Tasks: 433 total, 31 running, 402 sleeping, 0 stopped, 0 zombie %Cpu(s): 33.1 us, 2.8 sy, 0.0 ni, 64.0 id, 0.1 wa, 0.0 hi, 0.0 si, 0.0 st KiB Mem : 65439004 total, 41670432 free, 20060116 used, 3708456 buff/cache KiB Swap: 67108860 total, 66984144 free, 124716 used. 44642244 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 226414 Betula 20 0 966024 751388 21308 R 50.2 1.1 4710:43 python 226392 Betula 20 0 966284 731720 21308 R 49.8 1.1 4726:53 python 226426 Betula 20 0 966028 731636 21308 R 49.2 1.1 4722:23 python 226386 Betula 20 0 900868 686108 21308 R 48.8 1.0 4718:59 python 226421 Betula 20 0 933512 718840 21308 R 34.7 1.1 4718:42 python 226400 Betula 20 0 966028 731604 21308 R 34.3 1.1 4714:19 python 226390 Betula 20 0 966028 731396 21308 R 33.7 1.1 4722:25 python 226394 Betula 20 0 966028 732992 21308 R 33.3 1.1 4715:13 python 226396 Betula 20 0 966024 751392 21308 R 33.3 1.1 4715:17 python 226398 Betula 20 0 900480 685944 21308 R 33.3 1.0 4720:18 python 226402 Betula 20 0 966412 751652 21308 R 33.3 1.1 4720:49 python 226404 Betula 20 0 966028 731552 21308 R 33.3 1.1 4714:23 python 226408 Betula 20 0 966028 731584 21308 R 33.3 1.1 4728:26 python 226412 Betula 20 0 900484 683500 21308 R 33.3 1.0 4720:43 python 226416 Betula 20 0 900484 661692 21308 R 33.3 1.0 4718:54 python 226424 Betula 20 0 900616 664976 21308 R 33.3 1.0 4721:11 python 226429 Betula 20 0 966408 751828 21308 R 33.3 1.1 4713:25 python 226430 Betula 20 0 966416 731820 21308 R 33.3 1.1 4719:45 python 226432 Betula 20 0 900484 665912 21308 R 33.3 1.0 4725:41 python 226436 Betula 20 0 900484 638932 21308 R 33.3 1.0 4714:19 python 59943 Betula 20 0 795176 583612 24792 R 33.0 0.9 4568:41 python 226388 Betula 20 0 966024 751460 21308 R 33.0 1.1 4721:37 python 226407 Betula 20 0 900740 666020 21308 R 33.0 1.0 4719:48 python 226410 Betula 20 0 900740 686160 21308 R 33.0 1.0 4716:02 python 226418 Betula 20 0 900480 685960 21308 R 33.0 1.0 4720:03 python 226422 Betula 20 0 966024 751404 21308 R 33.0 1.1 4716:34 python 226434 Betula 20 0 900484 658764 21308 R 33.0 1.0 4714:36 python 226511 Betula 20 0 729884 514924 21304 R 33.0 0.8 4658:24 python 70311 root 20 0 44924 1388 1088 R 3.0 0.0 0:00.09 ps 69994 Betula 20 0 158008 2848 1816 R 0.7 0.0 0:00.62 top I also used "ps ux" command. The results are as follows. USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND Betula 59942 0.0 0.0 113136 1456 ? S May05 0:00 /bin/bash -c set -euo pipefail; Log=/data2/Betula/virus/virsorter2/test.out/iter-0/ssDNA Betula 59943 35.2 0.8 761888 531292 ? R May05 4566:51 python /data2/Betula/miniconda3/envs/vs/VirSorter2/virsorter/./scripts/provirus.py iter-0/ssDNA/all.p Betula 68996 0.0 0.0 156376 2452 ? S 21:09 0:00 sshd: ***@***.***/1 Betula 68997 0.7 0.0 116968 3564 pts/1 Ss 21:09 0:00 -bash Betula 69129 0.0 0.0 151068 2116 pts/1 R+ 21:09 0:00 ps ux Betula 103797 0.0 0.0 115384 1736 ? Ss Apr30 0:00 -bash Betula 103994 0.0 0.0 113136 1552 ? S Apr30 0:00 /bin/bash /opt/gridview//pbs/dispatcher/mom_priv/jobs/3804.node200.SC Betula 104000 0.0 0.0 154096 24108 ? S Apr30 0:00 /data2/Betula/miniconda3/envs/vs/bin/python3.8 /data2/Betula/miniconda3/envs/vs/bin/virsorter run -w / Betula 104018 0.0 0.1 3482232 112800 ? Sl Apr30 1:05 /data2/Betula/miniconda3/envs/vs/bin/python3.8 /data2/Betula/miniconda3/envs/vs/bin/snakemake --snakef Betula 226385 0.0 0.0 113136 1456 ? S May05 0:00 /bin/bash -c set -euo pipefail; Log=/data2/Betula/virus/virsorter2/test.out/iter-0/dsDNA Betula 226386 35.7 1.0 900740 658348 ? R May05 4717:15 python /data2/Betula/miniconda3/envs/vs/VirSorter2/virsorter/./scripts/provirus.py iter-0/dsDNAphage/ Betula 226387 0.0 0.0 113136 1456 ? S May05 0:00 /bin/bash -c set -euo pipefail; Log=/data2/Betula/virus/virsorter2/test.out/iter-0/dsDNA Betula 226388 35.7 1.0 900480 685980 ? R May05 4719:52 python /data2/Betula/miniconda3/envs/vs/VirSorter2/virsorter/./scripts/provirus.py iter-0/dsDNAphage/ Betula 226389 0.0 0.0 113136 1456 ? S May05 0:00 /bin/bash -c set -euo pipefail; Log=/data2/Betula/virus/virsorter2/test.out/iter-0/dsDNA Betula 226390 35.7 1.1 966024 751156 ? R May05 4720:41 python /data2/Betula/miniconda3/envs/vs/VirSorter2/virsorter/./scripts/provirus.py iter-0/dsDNAphage/ Betula 226391 0.0 0.0 113136 1456 ? S May05 0:00 /bin/bash -c set -euo pipefail; Log=/data2/Betula/virus/virsorter2/test.out/iter-0/dsDNA Betula 226392 35.8 1.0 900484 686032 ? R May05 4725:04 python /data2/Betula/miniconda3/envs/vs/VirSorter2/virsorter/./scripts/provirus.py iter-0/dsDNAphage/ Betula 226393 0.0 0.0 113136 1456 ? S May05 0:00 /bin/bash -c set -euo pipefail; Log=/data2/Betula/virus/virsorter2/test.out/iter-0/dsDNA Betula 226394 35.7 1.1 966024 751580 ? R May05 4713:30 python /data2/Betula/miniconda3/envs/vs/VirSorter2/virsorter/./scripts/provirus.py iter-0/dsDNAphage/ Betula 226395 0.0 0.0 113136 1456 ? S May05 0:00 /bin/bash -c set -euo pipefail; Log=/data2/Betula/virus/virsorter2/test.out/iter-0/dsDNA Betula 226396 35.7 1.0 900480 685916 ? R May05 4713:28 python /data2/Betula/miniconda3/envs/vs/VirSorter2/virsorter/./scripts/provirus.py iter-0/dsDNAphage/ Betula 226397 0.0 0.0 113136 1456 ? S May05 0:00 /bin/bash -c set -euo pipefail; Log=/data2/Betula/virus/virsorter2/test.out/iter-0/dsDNA Betula 226398 35.7 1.0 900480 685996 ? R May05 4718:37 python /data2/Betula/miniconda3/envs/vs/VirSorter2/virsorter/./scripts/provirus.py iter-0/dsDNAphage/ Betula 226399 0.0 0.0 113136 1456 ? S May05 0:00 /bin/bash -c set -euo pipefail; Log=/data2/Betula/virus/virsorter2/test.out/iter-0/dsDNA Betula 226400 35.7 1.0 933256 702432 ? R May05 4712:32 python /data2/Betula/miniconda3/envs/vs/VirSorter2/virsorter/./scripts/provirus.py iter-0/dsDNAphage/ Betula 226401 0.0 0.0 113136 1456 ? S May05 0:00 /bin/bash -c set -euo pipefail; Log=/data2/Betula/virus/virsorter2/test.out/iter-0/dsDNA Betula 226402 35.7 1.1 966412 751652 ? R May05 4719:03 python /data2/Betula/miniconda3/envs/vs/VirSorter2/virsorter/./scripts/provirus.py iter-0/dsDNAphage/ Betula 226403 0.0 0.0 113136 1456 ? S May05 0:00 /bin/bash -c set -euo pipefail; Log=/data2/Betula/virus/virsorter2/test.out/iter-0/dsDNA Betula 226404 35.7 1.1 966024 751368 ? R May05 4712:38 python /data2/Betula/miniconda3/envs/vs/VirSorter2/virsorter/./scripts/provirus.py iter-0/dsDNAphage/ Betula 226405 0.0 0.0 113136 1456 ? S May05 0:00 /bin/bash -c set -euo pipefail; Log=/data2/Betula/virus/virsorter2/test.out/iter-0/dsDNA Betula 226406 0.0 0.0 113136 1452 ? S May05 0:00 /bin/bash -c set -euo pipefail; Log=/data2/Betula/virus/virsorter2/test.out/iter-0/dsDNA Betula 226407 35.7 1.1 966280 751524 ? R May05 4718:06 python /data2/Betula/miniconda3/envs/vs/VirSorter2/virsorter/./scripts/provirus.py iter-0/dsDNAphage/ Betula 226408 35.8 1.1 966024 751496 ? R May05 4726:45 python /data2/Betula/miniconda3/envs/vs/VirSorter2/virsorter/./scripts/provirus.py iter-0/dsDNAphage/ Betula 226409 0.0 0.0 113136 1452 ? S May05 0:00 /bin/bash -c set -euo pipefail; Log=/data2/Betula/virus/virsorter2/test.out/iter-0/dsDNA Betula 226410 35.7 1.1 966284 731804 ? R May05 4714:20 python /data2/Betula/miniconda3/envs/vs/VirSorter2/virsorter/./scripts/provirus.py iter-0/dsDNAphage/ Betula 226411 0.0 0.0 113136 1456 ? S May05 0:00 /bin/bash -c set -euo pipefail; Log=/data2/Betula/virus/virsorter2/test.out/iter-0/dsDNA Betula 226412 35.7 1.1 966028 731604 ? R May05 4718:58 python /data2/Betula/miniconda3/envs/vs/VirSorter2/virsorter/./scripts/provirus.py iter-0/dsDNAphage/ Betula 226413 0.0 0.0 113136 1456 ? S May05 0:00 /bin/bash -c set -euo pipefail; Log=/data2/Betula/virus/virsorter2/test.out/iter-0/dsDNA Betula 226414 35.6 1.0 900484 665044 ? R May05 4708:58 python /data2/Betula/miniconda3/envs/vs/VirSorter2/virsorter/./scripts/provirus.py iter-0/dsDNAphage/ Betula 226415 0.0 0.0 113136 1456 ? S May05 0:00 /bin/bash -c set -euo pipefail; Log=/data2/Betula/virus/virsorter2/test.out/iter-0/dsDNA Betula 226416 35.7 1.0 900480 685976 ? R May05 4717:13 python /data2/Betula/miniconda3/envs/vs/VirSorter2/virsorter/./scripts/provirus.py iter-0/dsDNAphage/ Betula 226417 0.0 0.0 113136 1456 ? S May05 0:00 /bin/bash -c set -euo pipefail; Log=/data2/Betula/virus/virsorter2/test.out/iter-0/dsDNA Betula 226418 35.7 1.1 966024 751432 ? R May05 4718:23 python /data2/Betula/miniconda3/envs/vs/VirSorter2/virsorter/./scripts/provirus.py iter-0/dsDNAphage/ Betula 226419 0.0 0.0 113136 1456 ? S May05 0:00 /bin/bash -c set -euo pipefail; Log=/data2/Betula/virus/virsorter2/test.out/iter-0/dsDNA Betula 226420 0.0 0.0 113136 1456 ? S May05 0:00 /bin/bash -c set -euo pipefail; Log=/data2/Betula/virus/virsorter2/test.out/iter-0/dsDNA Betula 226421 35.7 1.1 966280 751660 ? R May05 4716:59 python /data2/Betula/miniconda3/envs/vs/VirSorter2/virsorter/./scripts/provirus.py iter-0/dsDNAphage/ Betula 226422 35.7 1.0 900484 666132 ? R May05 4714:47 python /data2/Betula/miniconda3/envs/vs/VirSorter2/virsorter/./scripts/provirus.py iter-0/dsDNAphage/ Betula 226423 0.0 0.0 113136 1456 ? S May05 0:00 /bin/bash -c set -euo pipefail; Log=/data2/Betula/virus/virsorter2/test.out/iter-0/dsDNA Betula 226424 35.7 1.0 933388 715440 ? R May05 4719:27 python /data2/Betula/miniconda3/envs/vs/VirSorter2/virsorter/./scripts/provirus.py iter-0/dsDNAphage/ Betula 226425 0.0 0.0 113136 1456 ? S May05 0:00 /bin/bash -c set -euo pipefail; Log=/data2/Betula/virus/virsorter2/test.out/iter-0/dsDNA Betula 226426 35.7 1.1 966024 751548 ? R May05 4720:36 python /data2/Betula/miniconda3/envs/vs/VirSorter2/virsorter/./scripts/provirus.py iter-0/dsDNAphage/ Betula 226427 0.0 0.0 113136 1456 ? S May05 0:00 /bin/bash -c set -euo pipefail; Log=/data2/Betula/virus/virsorter2/test.out/iter-0/dsDNA Betula 226428 0.0 0.0 113136 1456 ? S May05 0:00 /bin/bash -c set -euo pipefail; Log=/data2/Betula/virus/virsorter2/test.out/iter-0/dsDNA Betula 226429 35.7 1.1 966408 751672 ? R May05 4711:42 python /data2/Betula/miniconda3/envs/vs/VirSorter2/virsorter/./scripts/provirus.py iter-0/dsDNAphage/ Betula 226430 35.7 1.1 966156 751564 ? R May05 4718:00 python /data2/Betula/miniconda3/envs/vs/VirSorter2/virsorter/./scripts/provirus.py iter-0/dsDNAphage/ Betula 226431 0.0 0.0 113136 1456 ? S May05 0:00 /bin/bash -c set -euo pipefail; Log=/data2/Betula/virus/virsorter2/test.out/iter-0/dsDNA Betula 226432 35.7 1.0 900484 663240 ? R May05 4723:58 python /data2/Betula/miniconda3/envs/vs/VirSorter2/virsorter/./scripts/provirus.py iter-0/dsDNAphage/ Betula 226433 0.0 0.0 113136 1456 ? S May05 0:00 /bin/bash -c set -euo pipefail; Log=/data2/Betula/virus/virsorter2/test.out/iter-0/dsDNA Betula 226434 35.7 1.1 966028 731584 ? R May05 4712:46 python /data2/Betula/miniconda3/envs/vs/VirSorter2/virsorter/./scripts/provirus.py iter-0/dsDNAphage/ Betula 226435 0.0 0.0 113136 1452 ? S May05 0:00 /bin/bash -c set -euo pipefail; Log=/data2/Betula/virus/virsorter2/test.out/iter-0/dsDNA Betula 226436 35.7 1.0 900480 685924 ? R May05 4712:33 python /data2/Betula/miniconda3/envs/vs/VirSorter2/virsorter/./scripts/provirus.py iter-0/dsDNAphage/ Betula 226510 0.0 0.0 113136 1456 ? S May05 0:00 /bin/bash -c set -euo pipefail; Log=/data2/Betula/virus/virsorter2/test.out/iter-0/ssDNA Betula 226511 35.2 0.8 794912 579948 ? R May05 4656:40 python /data2/Betula/miniconda3/envs/vs/VirSorter2/virsorter/./scripts/provirus.py iter-0/ssDNA/all.p At 2024-05-14 20:15:10, "jiarong" ***@***.***> wrote: Hi, your data is quite large. I see the step 2 (the longest step) was finished a week ago. The last step should NOT take this long. Can you check which process it is stuck at? (using top command). Also note that if you this run errors out, you can just rerun the same command, VirSorter2 can pick up where it stops and continue. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: ***@***.***>

[jiarong]

It shows that VirSorter2 is still processing the prophage extraction step, NOT stuck due to errors. For future runs, you can split the large input file and process them in parallel.

[Betula0422]

Thank you so much for your reply. I got it! Good luck with your job!

…

---- Replied Message ---- | From | ***@***.***> | | Date | 05/15/2024 10:25 | | To | jiarong/VirSorter2 ***@***.***> | | Cc | Betula0422 ***@***.***>, Author ***@***.***> | | Subject | Re: [jiarong/VirSorter2] The output files don't update (Issue #198) | It shows that VirSorter2 is still processing the prophage extraction step, NOT stuck due to errors. For future runs, you can split the large input file and process them in parallel. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: ***@***.***>