Fix merge conflicts with v1.13.2 template #74

ErikDanielsson · 2021-04-13T08:43:01Z

PR checklist

This comment contains a description of changes (with reason)
If you've fixed a bug or added code that should be tested, add tests!
If necessary, also make a PR on the nf-core/smrnaseq branch on the nf-core/test-datasets repo
Ensure the test suite passes (nextflow run . -profile test,docker).
Make sure your code lints (nf-core lint .).
Documentation in docs is updated
CHANGELOG.md is updated
README.md is updated

Learn more about contributing: https://github.com/nf-core/smrnaseq/tree/master/.github/CONTRIBUTING.md

ewels · 2021-04-13T10:05:35Z

Not quite sure what's going on with the linting logs - tonnes of sqlite log messages being spat out 👀 Do you see this locally as well @ErikDanielsson ?

ewels · 2021-04-13T10:05:57Z

`nf-core lint` overall result: Passed ✅ ⚠️

Posted for pipeline commit 1ab5a5c

+| ✅ 159 tests passed       |+
!| ❗  24 tests had warnings |!

❗ Test warnings:

actions_awsfulltest - .github/workflows/awsfulltest.yml should test full datasets, not -profile test
conda_env_yaml - Conda dep outdated: conda-forge::r-base=3.6.3, 4.0.3 available
conda_env_yaml - Conda dep outdated: conda-forge::r-statmod=1.4.32, 1.4.35 available
conda_env_yaml - Conda dep outdated: conda-forge::r-data.table=1.12.4, 1.14.0 available
conda_env_yaml - Conda dep outdated: conda-forge::r-gplots=3.0.1.1, 3.1.1 available
conda_env_yaml - Conda dep outdated: conda-forge::r-r.methodss3=1.7.1, 1.8.1 available
conda_env_yaml - Conda dep outdated: conda-forge::python=3.7.3, 3.9.2 available
conda_env_yaml - Conda dep outdated: conda-forge::markdown=3.1.1, 3.3.4 available
conda_env_yaml - Conda dep outdated: conda-forge::pymdown-extensions=6.0, 8.1.1 available
conda_env_yaml - Conda dep outdated: conda-forge::pygments=2.6.1, 2.8.1 available
conda_env_yaml - Conda dep outdated: conda-forge::pigz=2.3.4, 2.6 available
conda_env_yaml - Conda dep outdated: bioconda::trim-galore=0.6.5, 0.6.6 available
conda_env_yaml - Conda dep outdated: bioconda::samtools=1.9, 1.12 available
conda_env_yaml - Conda dep outdated: bioconda::bowtie=1.2.3, 1.3.0 available
conda_env_yaml - Conda dep outdated: bioconda::multiqc=1.9, 1.10.1 available
conda_env_yaml - Conda dep outdated: bioconda::htseq=0.11.3, 0.13.5 available
conda_env_yaml - Conda dep outdated: bioconda::seqkit=0.12.0, 0.15.0 available
conda_env_yaml - Conda dep outdated: bioconda::bioconductor-edger=3.26.5, 3.32.1 available
conda_env_yaml - Conda dep outdated: bioconda::bioconductor-limma=3.40.2, 3.46.0 available
pipeline_todos - TODO string in README.md: Add bibliography of tools and data used in your pipeline
pipeline_todos - TODO string in nextflow.config: Specify your pipeline's command line flags
pipeline_todos - TODO string in awstest.yml: You can customise CI pipeline run tests as required
pipeline_todos - TODO string in awsfulltest.yml: You can customise AWS full pipeline tests as required
schema_params - Schema param config_profile_name not found from nextflow config

✅ Tests passed:

files_exist - File found: .gitattributes
files_exist - File found: CHANGELOG.md
files_exist - File found: CODE_OF_CONDUCT.md
files_exist - File found: CODE_OF_CONDUCT.md
files_exist - File found: LICENSE or LICENSE.md or LICENCE or LICENCE.md
files_exist - File found: nextflow_schema.json
files_exist - File found: nextflow.config
files_exist - File found: README.md
files_exist - File found: .github/.dockstore.yml
files_exist - File found: .github/CONTRIBUTING.md
files_exist - File found: .github/ISSUE_TEMPLATE/bug_report.md
files_exist - File found: .github/ISSUE_TEMPLATE/config.yml
files_exist - File found: .github/ISSUE_TEMPLATE/feature_request.md
files_exist - File found: .github/markdownlint.yml
files_exist - File found: .github/PULL_REQUEST_TEMPLATE.md
files_exist - File found: .github/workflows/branch.yml
files_exist - File found: .github/workflows/ci.yml
files_exist - File found: .github/workflows/linting_comment.yml
files_exist - File found: .github/workflows/linting.yml
files_exist - File found: assets/email_template.html
files_exist - File found: assets/email_template.txt
files_exist - File found: assets/sendmail_template.txt
files_exist - File found: assets/nf-core-smrnaseq_logo.png
files_exist - File found: bin/markdown_to_html.py
files_exist - File found: docs/images/nf-core-smrnaseq_logo.png
files_exist - File found: docs/output.md
files_exist - File found: docs/README.md
files_exist - File found: docs/README.md
files_exist - File found: docs/usage.md
files_exist - File found: lib/nfcore_external_java_deps.jar
files_exist - File found: lib/NfcoreSchema.groovy
files_exist - File found: main.nf
files_exist - File found: environment.yml
files_exist - File found: Dockerfile
files_exist - File found: conf/base.config
files_exist - File found: .github/workflows/awstest.yml
files_exist - File found: .github/workflows/awsfulltest.yml
files_exist - File not found check: Singularity
files_exist - File not found check: parameters.settings.json
files_exist - File not found check: bin/markdown_to_html.r
files_exist - File not found check: conf/aws.config
files_exist - File not found check: .github/workflows/push_dockerhub.yml
files_exist - File not found check: .travis.yml
nextflow_config - Config variable found: manifest.name
nextflow_config - Config variable found: manifest.nextflowVersion
nextflow_config - Config variable found: manifest.description
nextflow_config - Config variable found: manifest.version
nextflow_config - Config variable found: manifest.homePage
nextflow_config - Config variable found: timeline.enabled
nextflow_config - Config variable found: trace.enabled
nextflow_config - Config variable found: report.enabled
nextflow_config - Config variable found: dag.enabled
nextflow_config - Config variable found: process.cpus
nextflow_config - Config variable found: process.memory
nextflow_config - Config variable found: process.time
nextflow_config - Config variable found: params.outdir
nextflow_config - Config variable found: params.input
nextflow_config - Config variable found: manifest.mainScript
nextflow_config - Config variable found: timeline.file
nextflow_config - Config variable found: trace.file
nextflow_config - Config variable found: report.file
nextflow_config - Config variable found: dag.file
nextflow_config - Config variable found: process.container
nextflow_config - Config variable (correctly) not found: params.version
nextflow_config - Config variable (correctly) not found: params.nf_required_version
nextflow_config - Config variable (correctly) not found: params.container
nextflow_config - Config variable (correctly) not found: params.singleEnd
nextflow_config - Config variable (correctly) not found: params.igenomesIgnore
nextflow_config - Config variable (correctly) not found: params.name
nextflow_config - Config timeline.enabled had correct value: true
nextflow_config - Config report.enabled had correct value: true
nextflow_config - Config trace.enabled had correct value: true
nextflow_config - Config dag.enabled had correct value: true
nextflow_config - Config manifest.name began with nf-core/
nextflow_config - Config variable manifest.homePage began with https://github.com/nf-core/
nextflow_config - Config dag.file ended with .svg
nextflow_config - Config variable manifest.nextflowVersion started with >= or !>=
nextflow_config - Config process.container looks correct: nfcore/smrnaseq:dev
nextflow_config - Config manifest.version ends in dev: '1.1dev'
files_unchanged - .gitattributes matches the template
files_unchanged - CODE_OF_CONDUCT.md matches the template
files_unchanged - LICENSE matches the template
files_unchanged - .github/.dockstore.yml matches the template
files_unchanged - .github/CONTRIBUTING.md matches the template
files_unchanged - .github/ISSUE_TEMPLATE/bug_report.md matches the template
files_unchanged - .github/ISSUE_TEMPLATE/config.yml matches the template
files_unchanged - .github/ISSUE_TEMPLATE/feature_request.md matches the template
files_unchanged - .github/markdownlint.yml matches the template
files_unchanged - .github/PULL_REQUEST_TEMPLATE.md matches the template
files_unchanged - .github/workflows/branch.yml matches the template
files_unchanged - .github/workflows/linting_comment.yml matches the template
files_unchanged - .github/workflows/linting.yml matches the template
files_unchanged - assets/email_template.html matches the template
files_unchanged - assets/email_template.txt matches the template
files_unchanged - assets/sendmail_template.txt matches the template
files_unchanged - assets/nf-core-smrnaseq_logo.png matches the template
files_unchanged - bin/markdown_to_html.py matches the template
files_unchanged - docs/images/nf-core-smrnaseq_logo.png matches the template
files_unchanged - docs/README.md matches the template
files_unchanged - lib/nfcore_external_java_deps.jar matches the template
files_unchanged - lib/NfcoreSchema.groovy matches the template
files_unchanged - .gitignore matches the template
files_unchanged - .github/workflows/push_dockerhub_dev.yml matches the template
files_unchanged - .github/workflows/push_dockerhub_release.yml matches the template
files_unchanged - assets/multiqc_config.yaml matches the template
actions_ci - '.github/workflows/ci.yml' is triggered on expected events
actions_ci - CI is building the correct docker image: docker build --no-cache . -t nfcore/smrnaseq:dev
actions_ci - CI is pulling the correct docker image: docker pull nfcore/smrnaseq:dev
actions_ci - CI is tagging docker image correctly: docker tag nfcore/smrnaseq:dev nfcore/smrnaseq:dev
actions_ci - '.github/workflows/ci.yml' checks minimum NF version
actions_awstest - '.github/workflows/awstest.yml' is triggered correctly
actions_awsfulltest - .github/workflows/awsfulltest.yml is triggered correctly
readme - README Nextflow minimum version badge matched config. Badge: 20.04.0, Config: 20.04.0
readme - README had a bioconda badge
conda_env_yaml - Conda environment name was correct (nf-core-smrnaseq-1.1dev)
conda_env_yaml - Conda dep had pinned version number: conda-forge::r-base=3.6.3
conda_env_yaml - Conda dep had pinned version number: conda-forge::r-statmod=1.4.32
conda_env_yaml - Conda dep had pinned version number: conda-forge::r-data.table=1.12.4
conda_env_yaml - Conda dep had pinned version number: conda-forge::r-gplots=3.0.1.1
conda_env_yaml - Conda dep had pinned version number: conda-forge::r-r.methodss3=1.7.1
conda_env_yaml - Conda dep had pinned version number: conda-forge::python=3.7.3
conda_env_yaml - Conda dep had pinned version number: conda-forge::markdown=3.1.1
conda_env_yaml - Conda dep had pinned version number: conda-forge::pymdown-extensions=6.0
conda_env_yaml - Conda dep had pinned version number: conda-forge::pygments=2.6.1
conda_env_yaml - Conda dep had pinned version number: conda-forge::pigz=2.3.4
conda_env_yaml - Conda dep had pinned version number: bioconda::fastqc=0.11.9
conda_env_yaml - Conda package is the latest available: bioconda::fastqc=0.11.9
conda_env_yaml - Conda dep had pinned version number: bioconda::trim-galore=0.6.5
conda_env_yaml - Conda dep had pinned version number: bioconda::samtools=1.9
conda_env_yaml - Conda dep had pinned version number: bioconda::bowtie=1.2.3
conda_env_yaml - Conda dep had pinned version number: bioconda::multiqc=1.9
conda_env_yaml - Conda dep had pinned version number: bioconda::mirtop=0.4.23
conda_env_yaml - Conda package is the latest available: bioconda::mirtop=0.4.23
conda_env_yaml - Conda dep had pinned version number: bioconda::seqcluster=1.2.7
conda_env_yaml - Conda package is the latest available: bioconda::seqcluster=1.2.7
conda_env_yaml - Conda dep had pinned version number: bioconda::htseq=0.11.3
conda_env_yaml - Conda dep had pinned version number: bioconda::fastx_toolkit=0.0.14
conda_env_yaml - Conda package is the latest available: bioconda::fastx_toolkit=0.0.14
conda_env_yaml - Conda dep had pinned version number: bioconda::seqkit=0.12.0
conda_env_yaml - Conda dep had pinned version number: bioconda::mirtrace=1.0.1
conda_env_yaml - Conda package is the latest available: bioconda::mirtrace=1.0.1
conda_env_yaml - Conda dep had pinned version number: bioconda::bioconductor-edger=3.26.5
conda_env_yaml - Conda dep had pinned version number: bioconda::bioconductor-limma=3.40.2
conda_env_yaml - Conda dep had pinned version number: bioconda::mirdeep2=2.0.1.2
conda_env_yaml - Conda package is the latest available: bioconda::mirdeep2=2.0.1.2
conda_dockerfile - Found all expected strings in Dockerfile file
pipeline_name_conventions - Name adheres to nf-core convention
template_strings - Did not find any Jinja template strings (56 files)
schema_lint - Schema lint passed
schema_lint - Schema title + description lint passed
actions_schema_validation - Workflow validation passed: linting.yml
actions_schema_validation - Workflow validation passed: branch.yml
actions_schema_validation - Workflow validation passed: awstest.yml
actions_schema_validation - Workflow validation passed: awsfulltest.yml
actions_schema_validation - Workflow validation passed: push_dockerhub_dev.yml
actions_schema_validation - Workflow validation passed: ci.yml
actions_schema_validation - Workflow validation passed: linting_comment.yml
actions_schema_validation - Workflow validation passed: push_dockerhub_release.yml
merge_markers - No merge markers found in pipeline files

Run details

nf-core/tools version 1.13.3
Run at 2021-04-13 08:50:21

ewels · 2021-04-13T10:07:03Z

Looks like you have a few linting warnings that can be cleaned up still ☝🏻 The pipeline_todos and schema_params ones might warrant some attention (some stuff that hasn't been done can be left, eg. the aws full test stuff).

ErikDanielsson · 2021-04-13T10:30:12Z

I get no log messages when running the linting locally, oddly enough, @ewels.

ewels · 2021-04-13T13:11:57Z

I get no log messages when running the linting locally, oddly enough, @ewels.

I suspect that this is due to a change in a dependency then - this stuff sometimes appears on GitHub Actions first as it does a totally fresh install every time so uses the latest versions of everything.

If we download the GHA artefact with the verbose log it might say which python module these messages are coming from..

ewels · 2021-04-13T13:15:38Z

Ah yes, our friend requests_cache doing loads of log.info() calls. Fantastic.

If you fancy a slightly more tricky / python task, a PR to nf-core/tools to downgrade these messages to debug would be fantastic.

Should be something like this (see elsewhere):

# Only show error messages from pipeline creation
logging.getLogger("requests_cache").setLevel(logging.WARNING)

ErikDanielsson · 2021-04-13T13:16:54Z

Sounds good, I'll give that a go!

ewels · 2021-04-13T13:17:00Z

Ah yes, our friend requests_cache doing loads of log.info() calls. Fantastic.

Next time we're on a call, remind me to show you how I figured this out...

ErikDanielsson · 2021-04-13T13:17:03Z

I noticed that the schema param config_profile_name which is causing a lint warning is missing from the template nextflow.config as well, perhaps this test should warn with the current template?

ewels · 2021-04-13T13:18:56Z

config_profile_name

Hmm I'm lost by this. x-ref https://github.com/nf-core/tools/pull/892/files . @KevinMenden do you know what should be going on here?

KevinMenden · 2021-04-13T13:51:49Z

Nope, no idea .... config_profile_name is nowhere to be found in the nextflow.config or main.nf .... so I'm wondering if we can just leave it out? 🤔

KevinMenden · 2021-04-13T13:56:18Z

Ah, you added that param about a month ago 🤔 okay I'm really lost here 🤷

JWCook · 2021-04-13T15:03:32Z

Ah yes, our friend requests_cache doing loads of log.info() calls. Fantastic.

Those have been bumped down to debug level in requests-cache 0.6.1. You should have less spammy logs now.

ewels · 2021-04-13T18:31:40Z

Ah yes, our friend requests_cache doing loads of log.info() calls. Fantastic.

Those have been bumped down to debug level in requests-cache 0.6.1. You should have less spammy logs now.

Eagle eyes @JWCook! 👀 That's great to hear, thanks for letting us know 👍🏻 Ok then I don't think any action is needed.. Even better 🎉

ewels · 2021-04-13T18:34:21Z

I noticed that the schema param config_profile_name which is causing a lint warning is missing from the template nextflow.config as well, perhaps this test should warn with the current template?

@ErikDanielsson anyway yes, I think that config_profile_name should be in both the schema and nextflow.config 👍🏻

I think that it also needs adding to the nextflow.config file in the template itself: https://github.com/nf-core/tools/blob/ee90d06f4e09c395c913a50b4266dd12fe7f179b/nf_core/pipeline-template/nextflow.config#L34 (as you spotted).

No change to the lint tests is needed.

KevinMenden · 2021-05-06T08:05:28Z

@ErikDanielsson are you still working on this?
Need to fix some bugs with the smRNA-seq pipeline, would be nice to merge this before 😁

I'll go through the PR either today or tomorrow to see if I can help.

KevinMenden

Alright this looks good to me, not sure why the tests are not running though 🤔

.github/workflows/ci.yml

ewels

LGTM 🚀

Thanks @ErikDanielsson for this, and thanks @KevinMenden for pushing it over the finishing line..

mjsteinbaugh and others added 10 commits February 24, 2021 13:03

Fix "seqcluster" typo

2b253ff

Template update for nf-core/tools version 1.13

842225d

Template update for nf-core/tools version 1.13.1

b5b51dd

Template update for nf-core/tools version 1.13.2

9bef1e1

Template update for nf-core/tools version 1.13.3

f74d4ca

Merge remote-tracking branch 'origin/dev' into mjsteinbaugh/master

a85fa99

Merged changes from nf-core template

4deb201

Fix lint test: 'files_unchanged'

e0ac01d

Fix lint test failing due to 'single_end' missing in schema

3eb929a

Fix failing lint tests 'files_unchanged'

1ab5a5c

Remove redundant TODOs

32e8dc1

ErikDanielsson added 2 commits April 14, 2021 09:40

Remove 'single_end' and add 'custom_config_profile'

353cd38

Empty string -> null

6f01165

This was referenced Apr 19, 2021

"clearing all items from cache" is logged every 2 secs when using the a command with a wait function nf-core/website#737

Closed

cli wait logs "clearing all items from cache" every 2 seconds nf-core/tools#1029

Closed

ewels mentioned this pull request May 5, 2021

Important! Template update for nf-core/tools v1.13.2 #73

Closed

KevinMenden reviewed May 6, 2021

View reviewed changes

.github/workflows/ci.yml Outdated Show resolved Hide resolved

KevinMenden and others added 2 commits May 6, 2021 11:48

install libtbb2 in container

be31fec

Update .github/workflows/ci.yml

4f579e4

ewels approved these changes May 6, 2021

View reviewed changes

ewels merged commit 3a2044e into nf-core:dev May 6, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix merge conflicts with v1.13.2 template #74

Fix merge conflicts with v1.13.2 template #74

ErikDanielsson commented Apr 13, 2021

ewels commented Apr 13, 2021

ewels commented Apr 13, 2021

❗ Test warnings:

✅ Tests passed:

Run details

ewels commented Apr 13, 2021

ErikDanielsson commented Apr 13, 2021

ewels commented Apr 13, 2021

ewels commented Apr 13, 2021

ErikDanielsson commented Apr 13, 2021

ewels commented Apr 13, 2021

ErikDanielsson commented Apr 13, 2021

ewels commented Apr 13, 2021

KevinMenden commented Apr 13, 2021

KevinMenden commented Apr 13, 2021

JWCook commented Apr 13, 2021

ewels commented Apr 13, 2021

ewels commented Apr 13, 2021

KevinMenden commented May 6, 2021

KevinMenden left a comment

ewels left a comment

Fix merge conflicts with v1.13.2 template #74

Fix merge conflicts with v1.13.2 template #74

Conversation

ErikDanielsson commented Apr 13, 2021

PR checklist

ewels commented Apr 13, 2021

ewels commented Apr 13, 2021

nf-core lint overall result: Passed ✅ ⚠️

❗ Test warnings:

✅ Tests passed:

Run details

ewels commented Apr 13, 2021

ErikDanielsson commented Apr 13, 2021

ewels commented Apr 13, 2021

ewels commented Apr 13, 2021

ErikDanielsson commented Apr 13, 2021

ewels commented Apr 13, 2021

ErikDanielsson commented Apr 13, 2021

ewels commented Apr 13, 2021

KevinMenden commented Apr 13, 2021

KevinMenden commented Apr 13, 2021

JWCook commented Apr 13, 2021

ewels commented Apr 13, 2021

ewels commented Apr 13, 2021

KevinMenden commented May 6, 2021

KevinMenden left a comment

Choose a reason for hiding this comment

ewels left a comment

Choose a reason for hiding this comment

`nf-core lint` overall result: Passed ✅ ⚠️