Refactor POC #188

maxulysse · 2023-08-24T15:46:12Z

PR checklist

remove public_aws_ecr

Co-authored-by: Adam Talbot <12817534+adamrtalbot@users.noreply.github.com>

Fix tests

Adds emit statement for FASTQs and metadata to SRA workflow

split up config files to be more modular

github-actions · 2023-08-24T15:47:56Z

`nf-core lint` overall result: Passed ✅ ⚠️

Posted for pipeline commit 4d13a44

+| ✅ 133 tests passed       |+
#| ❔  18 tests were ignored |#
!| ❗   6 tests had warnings |!

❗ Test warnings:

nextflow_config - Config manifest.version should end in dev: 1.11.0
pipeline_todos - TODO string in methods_description_template.yml: #Update the HTML below to your preferred methods description, e.g. add publication citation for this pipeline
pipeline_todos - TODO string in meta.yml: #Add a description of the subworkflow and list keywords
pipeline_todos - TODO string in meta.yml: #Add a list of the modules and/or subworkflows used in the subworkflow
pipeline_todos - TODO string in meta.yml: #List all of the channels used as input with a description and their structure
pipeline_todos - TODO string in meta.yml: #List all of the channels used as output with a descriptions and their structure

❔ Tests ignored:

files_exist - File is ignored: .github/workflows/awsfulltest.yml
files_exist - File is ignored: .github/workflows/awstest.yml
files_exist - File is ignored: assets/multiqc_config.yml
files_exist - File is ignored: conf/igenomes.config
files_exist - File is ignored: conf/modules.config
files_exist - File is ignored: lib/NfcoreTemplate.groovy
files_exist - File is ignored: lib/Utils.groovy
files_exist - File is ignored: lib/WorkflowFetchngs.groovy
files_exist - File is ignored: lib/WorkflowMain.groovy
files_exist - File is ignored: lib/nfcore_external_java_deps.jar
files_unchanged - File ignored due to lint config: .gitattributes
files_unchanged - File ignored due to lint config: assets/sendmail_template.txt
files_unchanged - File does not exist: lib/nfcore_external_java_deps.jar
files_unchanged - File ignored due to lint config: lib/NfcoreTemplate.groovy
files_unchanged - File ignored due to lint config: .gitignore or .prettierignore or pyproject.toml
actions_ci - actions_ci
actions_awstest - 'awstest.yml' workflow not found: /home/runner/work/fetchngs/fetchngs/.github/workflows/awstest.yml
multiqc_config - 'assets/multiqc_config.yml' not found

✅ Tests passed:

files_exist - File found: .gitattributes
files_exist - File found: .gitignore
files_exist - File found: .nf-core.yml
files_exist - File found: .editorconfig
files_exist - File found: .prettierignore
files_exist - File found: .prettierrc.yml
files_exist - File found: CHANGELOG.md
files_exist - File found: CITATIONS.md
files_exist - File found: CODE_OF_CONDUCT.md
files_exist - File found: CODE_OF_CONDUCT.md
files_exist - File found: LICENSE or LICENSE.md or LICENCE or LICENCE.md
files_exist - File found: nextflow_schema.json
files_exist - File found: nextflow.config
files_exist - File found: README.md
files_exist - File found: .github/.dockstore.yml
files_exist - File found: .github/CONTRIBUTING.md
files_exist - File found: .github/ISSUE_TEMPLATE/bug_report.yml
files_exist - File found: .github/ISSUE_TEMPLATE/config.yml
files_exist - File found: .github/ISSUE_TEMPLATE/feature_request.yml
files_exist - File found: .github/PULL_REQUEST_TEMPLATE.md
files_exist - File found: .github/workflows/branch.yml
files_exist - File found: .github/workflows/ci.yml
files_exist - File found: .github/workflows/linting_comment.yml
files_exist - File found: .github/workflows/linting.yml
files_exist - File found: assets/email_template.html
files_exist - File found: assets/email_template.txt
files_exist - File found: assets/sendmail_template.txt
files_exist - File found: assets/nf-core-fetchngs_logo_light.png
files_exist - File found: conf/test.config
files_exist - File found: conf/test_full.config
files_exist - File found: docs/images/nf-core-fetchngs_logo_light.png
files_exist - File found: docs/images/nf-core-fetchngs_logo_dark.png
files_exist - File found: docs/output.md
files_exist - File found: docs/README.md
files_exist - File found: docs/README.md
files_exist - File found: docs/usage.md
files_exist - File found: main.nf
files_exist - File found: conf/base.config
files_exist - File found: modules.json
files_exist - File found: pyproject.toml
files_exist - File not found check: Singularity
files_exist - File not found check: parameters.settings.json
files_exist - File not found check: pipeline_template.yml
files_exist - File not found check: .nf-core.yaml
files_exist - File not found check: bin/markdown_to_html.r
files_exist - File not found check: conf/aws.config
files_exist - File not found check: .github/workflows/push_dockerhub.yml
files_exist - File not found check: .github/ISSUE_TEMPLATE/bug_report.md
files_exist - File not found check: .github/ISSUE_TEMPLATE/feature_request.md
files_exist - File not found check: docs/images/nf-core-fetchngs_logo.png
files_exist - File not found check: .markdownlint.yml
files_exist - File not found check: .yamllint.yml
files_exist - File not found check: lib/Checks.groovy
files_exist - File not found check: lib/Completion.groovy
files_exist - File not found check: lib/Workflow.groovy
files_exist - File not found check: .travis.yml
nextflow_config - Config variable found: manifest.name
nextflow_config - Config variable found: manifest.nextflowVersion
nextflow_config - Config variable found: manifest.description
nextflow_config - Config variable found: manifest.version
nextflow_config - Config variable found: manifest.homePage
nextflow_config - Config variable found: timeline.enabled
nextflow_config - Config variable found: trace.enabled
nextflow_config - Config variable found: report.enabled
nextflow_config - Config variable found: dag.enabled
nextflow_config - Config variable found: process.cpus
nextflow_config - Config variable found: process.memory
nextflow_config - Config variable found: process.time
nextflow_config - Config variable found: params.outdir
nextflow_config - Config variable found: params.input
nextflow_config - Config variable found: params.validationShowHiddenParams
nextflow_config - Config variable found: params.validationSchemaIgnoreParams
nextflow_config - Config variable found: manifest.mainScript
nextflow_config - Config variable found: timeline.file
nextflow_config - Config variable found: trace.file
nextflow_config - Config variable found: report.file
nextflow_config - Config variable found: dag.file
nextflow_config - Config variable (correctly) not found: params.nf_required_version
nextflow_config - Config variable (correctly) not found: params.container
nextflow_config - Config variable (correctly) not found: params.singleEnd
nextflow_config - Config variable (correctly) not found: params.igenomesIgnore
nextflow_config - Config variable (correctly) not found: params.name
nextflow_config - Config variable (correctly) not found: params.enable_conda
nextflow_config - Config timeline.enabled had correct value: true
nextflow_config - Config report.enabled had correct value: true
nextflow_config - Config trace.enabled had correct value: true
nextflow_config - Config dag.enabled had correct value: true
nextflow_config - Config manifest.name began with nf-core/
nextflow_config - Config variable manifest.homePage began with https://github.com/nf-core/
nextflow_config - Config dag.file ended with .html
nextflow_config - Config variable manifest.nextflowVersion started with >= or !>=
nextflow_config - Config params.custom_config_version is set to master
nextflow_config - Config params.custom_config_base is set to https://raw.githubusercontent.com/nf-core/configs/master
nextflow_config - Lines for loading custom profiles found
files_unchanged - .prettierrc.yml matches the template
files_unchanged - CODE_OF_CONDUCT.md matches the template
files_unchanged - LICENSE matches the template
files_unchanged - .github/.dockstore.yml matches the template
files_unchanged - .github/CONTRIBUTING.md matches the template
files_unchanged - .github/ISSUE_TEMPLATE/bug_report.yml matches the template
files_unchanged - .github/ISSUE_TEMPLATE/config.yml matches the template
files_unchanged - .github/ISSUE_TEMPLATE/feature_request.yml matches the template
files_unchanged - .github/PULL_REQUEST_TEMPLATE.md matches the template
files_unchanged - .github/workflows/branch.yml matches the template
files_unchanged - .github/workflows/linting_comment.yml matches the template
files_unchanged - .github/workflows/linting.yml matches the template
files_unchanged - assets/email_template.html matches the template
files_unchanged - assets/email_template.txt matches the template
files_unchanged - assets/nf-core-fetchngs_logo_light.png matches the template
files_unchanged - docs/images/nf-core-fetchngs_logo_light.png matches the template
files_unchanged - docs/images/nf-core-fetchngs_logo_dark.png matches the template
files_unchanged - docs/README.md matches the template
readme - README Nextflow minimum version badge matched config. Badge: 23.04.0, Config: 23.04.0
readme - README Zenodo placeholder was replaced with DOI.
pipeline_name_conventions - Name adheres to nf-core convention
template_strings - Did not find any Jinja template strings (170 files)
schema_lint - Schema lint passed
schema_lint - Schema title + description lint passed
schema_lint - Input mimetype lint passed: 'text/csv'
schema_params - Schema matched params returned from nextflow config
system_exit - No System.exit calls found
actions_schema_validation - Workflow validation passed: ci.yml
actions_schema_validation - Workflow validation passed: fix-linting.yml
actions_schema_validation - Workflow validation passed: branch.yml
actions_schema_validation - Workflow validation passed: linting.yml
actions_schema_validation - Workflow validation passed: release-announcments.yml
actions_schema_validation - Workflow validation passed: clean-up.yml
actions_schema_validation - Workflow validation passed: cloud_tests_small.yml
actions_schema_validation - Workflow validation passed: cloud_tests_full.yml
actions_schema_validation - Workflow validation passed: linting_comment.yml
merge_markers - No merge markers found in pipeline files
modules_json - Only installed modules found in modules.json
modules_structure - modules directory structure is correct 'modules/nf-core/TOOL/SUBTOOL'

Run details

nf-core/tools version 2.10
Run at 2023-10-18 10:04:17

Co-authored-by: Adam Talbot <12817534+adamrtalbot@users.noreply.github.com>

Move out multiQC and versions

maxulysse · 2023-08-24T17:46:19Z

.github/workflows/ci.yml

@@ -4,10 +4,12 @@ on:
  push:
    branches:
      - dev
+      - refactor


Reminder to remove that when we want to merge it

maxulysse · 2023-08-24T17:59:37Z

from @adamrtalbot

We should tidy

fetchngs/main.nf

Lines 49 to 67 in 5dab8d0

// Auto-detect input id type

def input_type = ''

if (WorkflowMain.isSraId(ch_input)) {

input_type = 'sra'

} else if (WorkflowMain.isSynapseId(ch_input)) {

input_type = 'synapse'

} else {

exit 1, 'Ids provided via --input not recognised please make sure they are either SRA / ENA / GEO / DDBJ or Synapse ids!'

}

if (params.input_type == input_type) {

if (params.input_type == 'sra') {

include { SRA } from './workflows/sra'

} else if (params.input_type == 'synapse') {

include { SYNAPSE } from './workflows/synapse'

}

} else {

exit 1, "Ids auto-detected as ${input_type}. Please provide '--input_type ${input_type}' as a parameter to the pipeline!"

}

[...]

tiny refactor

…kflow Use nf-core subworkflow: NFCORE_PIPELINE_UTILS

Fix all

fix sratools

…oftware-versions-via-collectfile and remove CUSTOM_DUMPSOFTWAREVERSIONS

…llectfile Replace CUSTOM_DUMPSOFTWAREVERSIONS with collectFile operator

fix sratools and fewer ids

.github/workflows/ci.yml

CHANGELOG.md

Co-authored-by: Maxime U Garcia <maxime.garcia@seqera.io>

maxulysse and others added 14 commits August 23, 2023 14:00

remove public aws ecr profile

dd94d03

split up config files to be more modular

1c5a396

Merge pull request #185 from maxulysse/remove_public_aws_ecr

d779e91

remove public_aws_ecr

no cascade inclusion of config files

9c5fadb

use schema_ignore_params to remove warnings

5073409

fix lint

bc7fd25

Adds emit statement for FASTQs and metadata to SRA workflow

81324d2

Update nextflow.config

82fa500

Co-authored-by: Adam Talbot <12817534+adamrtalbot@users.noreply.github.com>

remove if statement

011addc

Merge pull request #187 from maxulysse/fix_tests

604949b

Fix tests

Add emit statement to synpase workflow

464397c

Update CHANGELOG.md

97af5cd

Merge pull request #184 from adamrtalbot/emit_statement_to_SRA_workflow

e7c2e18

Adds emit statement for FASTQs and metadata to SRA workflow

Merge pull request #186 from maxulysse/split_configs

addfa36

split up config files to be more modular

maxulysse and others added 6 commits August 24, 2023 18:07

move out multiqc and versions

c7ab8b8

enable tests in refactor

372f9e4

Apply suggestions from code review

5dab8d0

Apply suggestions from code review

aff72b7

Co-authored-by: Adam Talbot <12817534+adamrtalbot@users.noreply.github.com>

fix syntax

7f6439c

Merge pull request #189 from maxulysse/move_things

54e7d7d

Move out multiQC and versions

maxulysse commented Aug 24, 2023

View reviewed changes

maxulysse and others added 7 commits August 25, 2023 10:36

small refactor

6355a33

staging

ef03d06

validate input in main.nf

1be5ebc

use error

698fab1

Merge pull request #190 from maxulysse/better_syntax

493a544

tiny refactor

Merge branch 'refactor' into better_tests

a3b2e8a

use workflow testing

e83e909

adamrtalbot and others added 20 commits October 16, 2023 15:46

Merge branch 'refactor' into use-nfcore-pipeline-utils-subworkflow

ce2237e

Merge pull request #223 from nf-core/use-nfcore-pipeline-utils-subwor…

8acf3ab

…kflow Use nf-core subworkflow: NFCORE_PIPELINE_UTILS

update modules

16ce425

patch custom-dumpsoftwareversions snap

5875c57

update snaps

4312b74

Replace CUSTOM_DUMPSOFTWAREVERSIONS with collectFile operator

2f47bef

fixup

f941c4d

gitignore

ba19243

Use software versions file that matches MultiQC input

364e6ba

tidy

e6c79c5

Merge pull request #225 from maxulysse/fix_all

4bd7d3a

Fix all

forgot patch

a9d8ae8

add data

cedd600

Merge pull request #227 from maxulysse/fix_all_again

fc23783

fix sratools

Merge branch 'refactor' of github.com:nf-core/fetchngs into collect-s…

643b71d

…oftware-versions-via-collectfile and remove CUSTOM_DUMPSOFTWAREVERSIONS

Merge pull request #226 from nf-core/collect-software-versions-via-co…

2a02ba7

…llectfile Replace CUSTOM_DUMPSOFTWAREVERSIONS with collectFile operator

tinier tests + patch fasterdump

5884bba

diff file

b9a8609

one last test

0d5f1d0

Merge pull request #228 from maxulysse/refactor

b452bd3

fix sratools and fewer ids

maxulysse commented Oct 18, 2023

View reviewed changes

.github/workflows/ci.yml Outdated Show resolved Hide resolved

maxulysse commented Oct 18, 2023

View reviewed changes

CHANGELOG.md Outdated Show resolved Hide resolved

Update CHANGELOG.md

b885d52

maxulysse mentioned this pull request Oct 18, 2023

Prepare 1.11.0 RC #230

Merged

adamrtalbot and others added 3 commits October 18, 2023 09:51

qwqUpdate snaps

833ad7e

Merge remote-tracking branch 'origin/dev' into refactor

e573640

Update .github/workflows/ci.yml

4d13a44

Co-authored-by: Maxime U Garcia <maxime.garcia@seqera.io>

adamrtalbot approved these changes Oct 18, 2023

View reviewed changes

adamrtalbot merged commit 86246fb into dev Oct 18, 2023
6 checks passed

drpatelh deleted the refactor branch October 31, 2023 11:12

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Refactor POC #188

Refactor POC #188

maxulysse commented Aug 24, 2023

github-actions bot commented Aug 24, 2023 •

edited

Loading

❗ Test warnings:

❔ Tests ignored:

✅ Tests passed:

Run details

maxulysse Aug 24, 2023

maxulysse commented Aug 24, 2023

Refactor POC #188

Refactor POC #188

Conversation

maxulysse commented Aug 24, 2023

PR checklist

github-actions bot commented Aug 24, 2023 • edited Loading

nf-core lint overall result: Passed ✅ ⚠️

❗ Test warnings:

❔ Tests ignored:

✅ Tests passed:

Run details

maxulysse Aug 24, 2023

Choose a reason for hiding this comment

maxulysse commented Aug 24, 2023

github-actions bot commented Aug 24, 2023 •

edited

Loading

`nf-core lint` overall result: Passed ✅ ⚠️