Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Nfcore and structuring modules for collaboration #157

Merged
merged 251 commits into from
Sep 28, 2021
Merged

Nfcore and structuring modules for collaboration #157

merged 251 commits into from
Sep 28, 2021

Conversation

chasemc
Copy link
Member

@chasemc chasemc commented Apr 28, 2021

Test execution:
pipeline_info.zip

Execution Details
task_id hash native_id name status exit submit duration realtime %cpu peak_rss peak_vmem rchar wchar
1 4c/06205c 2237350 AUTOMETA:LENGTH_FILTER (filtering metagenome metagenome) COMPLETED 0 2021-04-28 14:39:08.950 3.6s 3s 153.8% 218.6 MB 3.3 GB 184.4 MB 76.1 MB
2 0d/5d13c8 2237734 AUTOMETA:KMER_COVERAGE (Calculating k-mer coverage for metagenome) COMPLETED 0 2021-04-28 14:39:12.594 1.2s 625ms 305.4% 11.3 MB 15.9 MB 85.4 MB 281.7 KB
8 b8/4fa560 2238087 AUTOMETA:ORFS (Calling orfs for metagenome) COMPLETED 0 2021-04-28 14:39:13.843 5.1s 3.5s 97.7% 57.3 MB 113.8 MB 1.3 MB 2.3 MB
10 e0/c30153 2238159 AUTOMETA:ORFS (Calling orfs for metagenome) COMPLETED 0 2021-04-28 14:39:13.856 6.8s 5.2s 96.2% 57.9 MB 114.1 MB 1.1 MB 1.8 MB
5 77/b83513 2238019 AUTOMETA:ORFS (Calling orfs for metagenome) COMPLETED 0 2021-04-28 14:39:13.831 7.7s 6.7s 97.8% 66.5 MB 120 MB 1.1 MB 1.7 MB
9 2c/62c9bc 2238120 AUTOMETA:ORFS (Calling orfs for metagenome) COMPLETED 0 2021-04-28 14:39:13.849 8.1s 6.4s 97.5% 60.6 MB 114.1 MB 1.2 MB 1.9 MB
3 af/66baae 2237997 AUTOMETA:ORFS (Calling orfs for metagenome) COMPLETED 0 2021-04-28 14:39:13.822 9.3s 8.5s 97.3% 70.8 MB 127.2 MB 1.4 MB 2.3 MB
14 18/cadcec 2238260 AUTOMETA:ORFS (Calling orfs for metagenome) COMPLETED 0 2021-04-28 14:39:13.869 9.8s 7.8s 97.6% 58.1 MB 113.9 MB 1.2 MB 1.9 MB
20 44/abd8ae 2238508 AUTOMETA:ORFS (Calling orfs for metagenome) COMPLETED 0 2021-04-28 14:39:13.921 11s 7.9s 96.2% 55.7 MB 113.7 MB 1.1 MB 1.8 MB
18 49/089aac 2238449 AUTOMETA:ORFS (Calling orfs for metagenome) COMPLETED 0 2021-04-28 14:39:13.911 11.1s 7.8s 95.6% 56.6 MB 113.9 MB 1.1 MB 1.7 MB
25 35/618b16 2238627 AUTOMETA:ORFS (Calling orfs for metagenome) COMPLETED 0 2021-04-28 14:39:13.941 12s 10.2s 97.7% 55.1 MB 113.6 MB 1.2 MB 1.9 MB
24 dd/194b84 2238590 AUTOMETA:ORFS (Calling orfs for metagenome) COMPLETED 0 2021-04-28 14:39:13.936 12.6s 10.6s 96.0% 55.1 MB 113.7 MB 1.1 MB 1.8 MB
7 0c/930231 2238059 AUTOMETA:ORFS (Calling orfs for metagenome) COMPLETED 0 2021-04-28 14:39:13.838 12.9s 11.3s 96.4% 63.1 MB 114.3 MB 1.5 MB 2.5 MB
26 e6/131a95 2238640 AUTOMETA:ORFS (Calling orfs for metagenome) COMPLETED 0 2021-04-28 14:39:13.945 13s 9.3s 94.5% 54.9 MB 113.6 MB 1.1 MB 1.8 MB
19 94/adb728 2238477 AUTOMETA:ORFS (Calling orfs for metagenome) COMPLETED 0 2021-04-28 14:39:13.915 13.2s 10.6s 96.6% 56.4 MB 113.7 MB 1.2 MB 2 MB
6 62/29d31c 2238040 AUTOMETA:ORFS (Calling orfs for metagenome) COMPLETED 0 2021-04-28 14:39:13.835 13.9s 12.3s 96.7% 66.1 MB 119.4 MB 2 MB 3.2 MB
23 74/de3f57 2238572 AUTOMETA:ORFS (Calling orfs for metagenome) COMPLETED 0 2021-04-28 14:39:13.932 14s 10.4s 96.5% 55.6 MB 113.6 MB 1.1 MB 1.8 MB
17 82/0c5366 2238345 AUTOMETA:ORFS (Calling orfs for metagenome) COMPLETED 0 2021-04-28 14:39:13.883 14.5s 12.2s 97.3% 57.3 MB 113.8 MB 1.2 MB 1.9 MB
22 bb/b00cae 2238552 AUTOMETA:ORFS (Calling orfs for metagenome) COMPLETED 0 2021-04-28 14:39:13.928 15.7s 12s 96.6% 55.7 MB 113.6 MB 1.2 MB 1.9 MB
28 68/656ba9 2247386 AUTOMETA:ORFS (Calling orfs for metagenome) COMPLETED 0 2021-04-28 14:39:20.639 9.1s 7.8s 94.7% 54.5 MB 113.6 MB 1.1 MB 1.8 MB
16 0e/77bad2 2238388 AUTOMETA:ORFS (Calling orfs for metagenome) COMPLETED 0 2021-04-28 14:39:13.889 15.9s 12.9s 96.8% 57 MB 113.8 MB 1.3 MB 2.1 MB
11 68/bcd2bb 2238185 AUTOMETA:ORFS (Calling orfs for metagenome) COMPLETED 0 2021-04-28 14:39:13.859 16.5s 13.9s 96.1% 59.6 MB 114.3 MB 1.4 MB 2.3 MB
30 08/941745 2248986 AUTOMETA:ORFS (Calling orfs for metagenome) COMPLETED 0 2021-04-28 14:39:21.943 8.5s 7.5s 93.4% 54.4 MB 113.6 MB 1.1 MB 1.8 MB
34 90/610e41 2252749 AUTOMETA:ORFS (Calling orfs for metagenome) COMPLETED 0 2021-04-28 14:39:24.966 6.4s 5.5s 96.6% 53.8 MB 113.6 MB 1.1 MB 1.8 MB
27 af/3f1e8d 2245277 AUTOMETA:ORFS (Calling orfs for metagenome) COMPLETED 0 2021-04-28 14:39:18.931 13.3s 11.8s 96.6% 54.7 MB 113.6 MB 1.1 MB 1.7 MB
21 d4/473943 2238529 AUTOMETA:ORFS (Calling orfs for metagenome) COMPLETED 0 2021-04-28 14:39:13.925 18.4s 15.2s 96.0% 55.9 MB 113.7 MB 1.2 MB 2 MB
32 c1/4c0b26 2251057 AUTOMETA:ORFS (Calling orfs for metagenome) COMPLETED 0 2021-04-28 14:39:23.698 9.6s 8s 97.0% 54.1 MB 113.6 MB 1.1 MB 1.8 MB
33 35/45607f 2252682 AUTOMETA:ORFS (Calling orfs for metagenome) COMPLETED 0 2021-04-28 14:39:24.928 8.5s 6.8s 97.3% 54.2 MB 113.6 MB 1.1 MB 1.7 MB
31 44/4ed54b 2250515 AUTOMETA:ORFS (Calling orfs for metagenome) COMPLETED 0 2021-04-28 14:39:23.138 10.6s 8.7s 96.5% 54.3 MB 113.6 MB 1.2 MB 1.8 MB
36 fb/b1d701 2255064 AUTOMETA:ORFS (Calling orfs for metagenome) COMPLETED 0 2021-04-28 14:39:26.564 7.3s 5.6s 96.4% 53 MB 113.4 MB 1.1 MB 1.7 MB
13 4b/dc5bd4 2238242 AUTOMETA:ORFS (Calling orfs for metagenome) COMPLETED 0 2021-04-28 14:39:13.866 20.1s 16.8s 97.3% 58.9 MB 113.9 MB 1.3 MB 2.1 MB
15 d2/68e084 2238326 AUTOMETA:ORFS (Calling orfs for metagenome) COMPLETED 0 2021-04-28 14:39:13.874 20.3s 17.3s 98.0% 57.9 MB 113.8 MB 1.4 MB 2.2 MB
4 4b/fdbf01 2238004 AUTOMETA:ORFS (Calling orfs for metagenome) COMPLETED 0 2021-04-28 14:39:13.826 20.5s 19.4s 97.7% 71.6 MB 122.7 MB 1.2 MB 1.9 MB
29 b2/b2f6f3 2248518 AUTOMETA:ORFS (Calling orfs for metagenome) COMPLETED 0 2021-04-28 14:39:21.539 12.9s 11.6s 97.3% 54.8 MB 113.6 MB 1.1 MB 1.8 MB
38 9d/0ee178 2255454 AUTOMETA:ORFS (Calling orfs for metagenome) COMPLETED 0 2021-04-28 14:39:26.947 8.2s 6.9s 96.9% 53.7 MB 113.4 MB 1.1 MB 1.8 MB
40 ac/7f21e3 2256649 AUTOMETA:ORFS (Calling orfs for metagenome) COMPLETED 0 2021-04-28 14:39:27.767 8.1s 6.3s 98.7% 53.4 MB 113.4 MB 1.1 MB 1.8 MB
39 88/22f7ad 2255609 AUTOMETA:ORFS (Calling orfs for metagenome) COMPLETED 0 2021-04-28 14:39:27.100 9.1s 7s 97.7% 53.5 MB 113.4 MB 1.1 MB 1.8 MB
12 eb/196c48 2238206 AUTOMETA:ORFS (Calling orfs for metagenome) COMPLETED 0 2021-04-28 14:39:13.863 22.7s 20s 97.7% 60.1 MB 114 MB 1.4 MB 2.2 MB
41 d8/86f4d7 2256916 AUTOMETA:ORFS (Calling orfs for metagenome) COMPLETED 0 2021-04-28 14:39:27.947 8.7s 7s 98.1% 53.5 MB 113.4 MB 1.1 MB 1.8 MB
35 ed/0eb013 2254166 AUTOMETA:ORFS (Calling orfs for metagenome) COMPLETED 0 2021-04-28 14:39:25.972 10.9s 8.8s 98.3% 53.6 MB 113.4 MB 1.1 MB 1.9 MB
37 be/e827d3 2255264 AUTOMETA:ORFS (Calling orfs for metagenome) COMPLETED 0 2021-04-28 14:39:26.717 10.3s 8.1s 98.9% 53.3 MB 113.4 MB 1.1 MB 1.8 MB
47 43/a0aa5e 2259898 AUTOMETA:ORFS (Calling orfs for metagenome) COMPLETED 0 2021-04-28 14:39:30.475 6.6s 5.1s 98.7% 52.7 MB 113.4 MB 1.1 MB 1.7 MB
43 79/2835ca 2258890 AUTOMETA:ORFS (Calling orfs for metagenome) COMPLETED 0 2021-04-28 14:39:29.633 7.7s 6s 97.9% 53.1 MB 113.4 MB 1.1 MB 1.7 MB
46 7b/fa805d 2259612 AUTOMETA:ORFS (Calling orfs for metagenome) COMPLETED 0 2021-04-28 14:39:30.317 7.2s 5.9s 99.3% 53.2 MB 113.4 MB 1.1 MB 1.7 MB
42 a3/277152 2257645 AUTOMETA:ORFS (Calling orfs for metagenome) COMPLETED 0 2021-04-28 14:39:28.414 9.6s 8.5s 98.7% 53.4 MB 113.4 MB 1.1 MB 1.7 MB
48 b6/9f5857 2261138 AUTOMETA:ORFS (Calling orfs for metagenome) COMPLETED 0 2021-04-28 14:39:31.400 7.1s 5.2s 99.5% 52.8 MB 113.4 MB 1.1 MB 1.7 MB
45 6b/9a61f3 2259064 AUTOMETA:ORFS (Calling orfs for metagenome) COMPLETED 0 2021-04-28 14:39:29.781 8.9s 7.1s 99.8% 52.7 MB 113.4 MB 1.1 MB 1.8 MB
44 2c/09d655 2258967 AUTOMETA:ORFS (Calling orfs for metagenome) COMPLETED 0 2021-04-28 14:39:29.699 9.3s 7.4s 99.9% 53.4 MB 113.4 MB 1.1 MB 1.8 MB
50 57/4b41cd 2262101 AUTOMETA:ORFS (Calling orfs for metagenome) COMPLETED 0 2021-04-28 14:39:32.328 7.1s 5.6s 100.6% 52.5 MB 113.4 MB 1.1 MB 1.8 MB
49 b1/9f4290 2261914 AUTOMETA:ORFS (Calling orfs for metagenome) COMPLETED 0 2021-04-28 14:39:32.188 7.3s 5.4s 100.0% 52.5 MB 113.4 MB 1.1 MB 1.7 MB
53 43/5a4ed6 2263892 AUTOMETA:ORFS (Calling orfs for metagenome) COMPLETED 0 2021-04-28 14:39:33.748 6.9s 4.6s 100.0% 52.8 MB 113.4 MB 1.1 MB 1.7 MB
51 f4/076c1a 2263259 AUTOMETA:ORFS (Calling orfs for metagenome) COMPLETED 0 2021-04-28 14:39:33.269 8.1s 6.5s 99.9% 52.9 MB 113.4 MB 1.1 MB 1.8 MB
62 72/36cc8f 2267547 AUTOMETA:ORFS (Calling orfs for metagenome) COMPLETED 0 2021-04-28 14:39:36.596 5.6s 3.6s 100.3% 52.8 MB 113.4 MB 1.1 MB 1.7 MB
56 49/b59d3a 2264404 AUTOMETA:ORFS (Calling orfs for metagenome) COMPLETED 0 2021-04-28 14:39:34.175 8.2s 6.1s 99.9% 52.3 MB 113.4 MB 1.1 MB 1.7 MB
54 90/f44b80 2263974 AUTOMETA:ORFS (Calling orfs for metagenome) COMPLETED 0 2021-04-28 14:39:33.858 8.7s 6.2s 100.6% 52.9 MB 113.4 MB 1.1 MB 1.7 MB
52 c9/54ddc0 2263408 AUTOMETA:ORFS (Calling orfs for metagenome) COMPLETED 0 2021-04-28 14:39:33.445 9.5s 6.7s 100.6% 52.9 MB 113.4 MB 1.1 MB 1.7 MB
71 7a/cebb94 2270206 AUTOMETA:ORFS (Calling orfs for metagenome) COMPLETED 0 2021-04-28 14:39:38.678 4.3s 1.9s 99.6% 52 MB 113.4 MB 735.9 KB 1.2 MB
57 97/b28bce 2264569 AUTOMETA:ORFS (Calling orfs for metagenome) COMPLETED 0 2021-04-28 14:39:34.344 8.8s 6.1s 100.0% 52.3 MB 113.4 MB 1.1 MB 1.7 MB
55 eb/313464 2264150 AUTOMETA:ORFS (Calling orfs for metagenome) COMPLETED 0 2021-04-28 14:39:33.960 9.4s 6.4s 100.2% 52.3 MB 113.4 MB 1.1 MB 1.7 MB
59 d7/d02065 2265770 AUTOMETA:ORFS (Calling orfs for metagenome) COMPLETED 0 2021-04-28 14:39:35.149 8.5s 4.5s 100.4% 52.3 MB 113.4 MB 1.1 MB 1.7 MB
60 d4/d5af07 2266728 AUTOMETA:ORFS (Calling orfs for metagenome) COMPLETED 0 2021-04-28 14:39:35.827 8.2s 3.9s 99.7% 52.5 MB 113.4 MB 1.1 MB 1.7 MB
63 a5/03380c 2267700 AUTOMETA:ORFS (Calling orfs for metagenome) COMPLETED 0 2021-04-28 14:39:36.693 7.4s 4s 99.7% 52.6 MB 113.4 MB 1.1 MB 1.7 MB
58 aa/0f0e00 2264669 AUTOMETA:ORFS (Calling orfs for metagenome) COMPLETED 0 2021-04-28 14:39:34.446 9.7s 5.6s 100.2% 52.3 MB 113.4 MB 1.1 MB 1.7 MB
66 e2/cd8c58 2268159 AUTOMETA:ORFS (Calling orfs for metagenome) COMPLETED 0 2021-04-28 14:39:37.082 7.1s 3.6s 100.1% 51.8 MB 113.4 MB 1.1 MB 1.7 MB
68 c1/3159a8 2268723 AUTOMETA:ORFS (Calling orfs for metagenome) COMPLETED 0 2021-04-28 14:39:37.542 6.7s 3s 99.1% 52.1 MB 113.4 MB 1.1 MB 1.7 MB
67 1d/9b4439 2268375 AUTOMETA:ORFS (Calling orfs for metagenome) COMPLETED 0 2021-04-28 14:39:37.293 7s 3.2s 99.6% 52 MB 113.4 MB 1.1 MB 1.7 MB
69 fd/8a0127 2269335 AUTOMETA:ORFS (Calling orfs for metagenome) COMPLETED 0 2021-04-28 14:39:38.032 6.4s 3.1s 99.6% 52 MB 113.4 MB 1.1 MB 1.8 MB
70 1c/19bd66 2269817 AUTOMETA:ORFS (Calling orfs for metagenome) COMPLETED 0 2021-04-28 14:39:38.455 6s 2.8s 100.0% 52 MB 113.4 MB 1.1 MB 1.8 MB
65 c6/5e3849 2268036 AUTOMETA:ORFS (Calling orfs for metagenome) COMPLETED 0 2021-04-28 14:39:36.982 7.5s 3.9s 99.1% 52 MB 113.4 MB 1.1 MB 1.7 MB
61 4f/9a736c 2267091 AUTOMETA:ORFS (Calling orfs for metagenome) COMPLETED 0 2021-04-28 14:39:36.174 8.4s 4.7s 100.1% 52.7 MB 113.4 MB 1.1 MB 1.7 MB
64 50/94b2fd 2267989 AUTOMETA:ORFS (Calling orfs for metagenome) COMPLETED 0 2021-04-28 14:39:36.890 7.8s 4.1s 99.4% 52 MB 113.4 MB 1.1 MB 1.7 MB
72 00/2bf5aa 2275661 AUTOMETA:TAXON_ASSIGNMENT:DIAMOND (diamond blastp on metagenome) COMPLETED 0 2021-04-28 14:39:44.737 5s 4.4s 1982.4% 726.3 MB 3.1 GB 115.6 MB 82 MB
73 a8/d34ed8 2277515 AUTOMETA:MARKERS (Finding markers for metagenome) COMPLETED 0 2021-04-28 14:39:49.743 4m 6s 4m 5s 239.1% 93 MB 3.5 GB 159.6 GB 105.2 MB
74 c0/8e1fb1 4038042 AUTOMETA:TAXON_ASSIGNMENT:LCA (Assigning LCA to metagenome) COMPLETED 0 2021-04-28 14:43:55.264 7m 37s 7m 37s 100.4% 3.8 GB 7 GB 34.2 GB 5.1 MB
75 a7/86babd 4043735 AUTOMETA:TAXON_ASSIGNMENT:MAJORITY_VOTE (Performing taxon majority vote on metagenome) COMPLETED 0 2021-04-28 14:51:32.394 7.5s 7s 123.3% 1.3 GB 4.5 GB 369.6 MB 80.8 KB
76 23/10de17 4044339 AUTOMETA:TAXON_ASSIGNMENT:SPLIT_KINGDOMS (Splitting votes into kingdoms for metagenome) COMPLETED 0 2021-04-28 14:51:39.935 7.5s 7s 123.1% 1.3 GB 4.5 GB 435.5 MB 76 MB
77 0c/7d0367 4044951 AUTOMETA:KMERS (counting kmers for metagenome) COMPLETED 0 2021-04-28 14:51:47.505 1m 1s 1m 1s 704.0% 4.1 GB 81.1 GB 180.8 MB 99.7 MB
78 9f/86e4f8 4051768 AUTOMETA:BINNING (Performing Autometa binning) COMPLETED 0 2021-04-28 14:52:48.842 17m 17s 17m 16s 108.6% 357.7 MB 5.1 GB 29.5 MB 452.4 KB
79 e4/3aa9de 4136136 AUTOMETA:UNCLUSTERED_RECRUITMENT (Performing Autometa unclustered recruitment) COMPLETED 0 2021-04-28 15:10:05.602 2.3s 1.7s 345.0% 144.1 MB 3.4 GB 38 MB 987.6 KB
80 5b/002dd2 4136121 AUTOMETA:BINNING_SUMMARY (Binning summary for metagenome) COMPLETED 0 2021-04-28 15:10:05.592 8.3s 7.7s 118.5% 1.3 GB 4.5 GB 435.9 MB 75.8 MB

Will come back around to this right before PR in case things change
This still needs work. Plan:

- true = default to "${params.outdir}/interim_outputs"
- false = switch off publishDir for intermediates
- path = store intermediates at specified path
Move main.nf to nextflow/autometa.nf
Change config so that main script is found and the different config files are included
A decent amount of text in here can be removed/is specific to nf-core.
Originally was main.nf and may revert to autometa.nf later. Bringing stuff from another repo where I tested the layout first
nf-core requires having an `params.input` so may as well change the input argument to that
Lots of changes here. Still a work in progress (WIP)
Copy link
Collaborator

@evanroyrees evanroyrees left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will be merging by EOTD

modules/local/diamond_blastp.nf Outdated Show resolved Hide resolved
nextflow.config Outdated Show resolved Hide resolved
nextflow_schema.json Outdated Show resolved Hide resolved
nextflow_schema.json Outdated Show resolved Hide resolved
nextflow_schema.json Outdated Show resolved Hide resolved
conf/base.config Show resolved Hide resolved
conf/base.config Show resolved Hide resolved
@chasemc
Copy link
Member Author

chasemc commented Sep 28, 2021

Will be merging by EOTD
d65350badaede26fa1f3ecc1a589046b

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment