Use new nf-test features by edmundmiller · Pull Request #6286 · nf-core/modules

edmundmiller · 2024-08-23T16:30:16Z

ci: Attempt to split everything out
ci: Add changed since, sharding, and ci
ci: Add filter to try to get jobs split up
ci: Switch to only-changed
ci: See if follow-dependencies works without "related-tests"

edmundmiller · 2024-08-26T16:34:12Z

We do have adamrtalbot/detect-nf-test-changes as a fallback option?

That's what we're currently using, so I don't really see it as a fallback 😆 It works fine, I think the sharding can work better and waste less resources spinning up a runner for each test instead of spinning up a consistent amount per PR.

That would close #391 as well, and might encourage people to make smaller PRs if we're only giving them 9 CI runners per PR anyway.

edmundmiller · 2024-09-18T17:13:09Z

Current things blocking this is nf-core/tools#3140 or we can give up and keep the paths-filter step for that

edmundmiller · 2024-09-18T21:05:38Z

Removed the need to use nf-core lint with pre-commit, as that wasn't truely blocking this from getting merged.

We can make a follow up issue to run every linting step through pre-commit.

maxulysse · 2024-09-19T06:34:59Z

Looking good, there's still some leftover comments in the nf-test yml, but I like it

edmundmiller · 2024-09-24T13:48:30Z

@maxulysse I think I've address the comments that can be addressed.

@adamrtalbot I think this goes slightly against #4545 (comment) but I think it's worth splitting up since we should be running pytest-workflow less and less, and the sharding is a completely different flow. Though it will probably Start up the pytest-workflow workflow every PR...

adamrtalbot

My main concern is getting stuck in a loop where there is too much data per runner and having no way of resolving it.

We over shard now, but at least it's a 1:1 relationship which makes it easy to diagnose an issue. We need to be confident we don't end up in a death loop in the middle of a hackathon.

In addition, we're over-provisioning 3 machines when 1 would suffice.

Still, this has fewer moving parts and is easier to replicate on a local machine, which makes it better under certain situations.

adamrtalbot · 2024-09-30T16:36:42Z

.github/workflows/lint.yml

+      - uses: mirpedrol/paths-filter@main
+        id: filter
+        with:
+          filters: "tests/config/pytest_modules.yml"


This relies on the pytest_modules.yml tags, is that what you want?

Nah, probably just want it to run on any module that's changed. We could include both the nf-test and pytest_modules ymls though I think.

adamrtalbot · 2024-09-30T16:41:34Z

.github/workflows/nf-test.yml

+      matrix:
+        filter: [process, workflow]
+        profile: [conda, docker_self_hosted, singularity]
+        shard: [1, 2, 3]


3 is a very small number for sharding.

If we edit UNTAR, we will run hundreds of tests packed onto the same machines. Remember, if too many tests are packed onto 1 machine there is no way to fix it other than editing this file which essentially blocks the person who opened the PR.

Conversely, if we run 1 nf-test we will add 2 machines that do nothing but install nf-test and close down.

This might be the most practical way of doing this, so I'm not 100% against it but what's the rationale for the number 3 and can we document it somewhere.

Is it possible to set it dynamically based on number of tests with an estimate of how many we can pack onto a machine? e.g. n_tests %% 5?

Of course, by making it dyanmic we're back to the current problem of having a dynamic workflow ID which causes issues with GH branch protection.

So let's go back to my initial point. If you modify a major process like UNTAR, will this still succeed or will it run out of storage? Remember every nf-test invocation downloads all the data it needs.

Is it possible to set it dynamically based on number of tests with an estimate of how many we can pack onto a machine? e.g. n_tests %% 5?

That would be the dream. But yeah I also wanted to avoid the dynamic number of jobs.

3 was just a random number that I picked. Trying to walk the line between runners sitting there, and not enough jobs.

Hot take: What if we just did a --dryRun at the beginning of the shard and if there's no jobs, exit before we set anything else up.

adamrtalbot · 2024-09-30T16:41:51Z

.github/workflows/nf-test.yml

+    strategy:
+      fail-fast: false
+      matrix:
+        filter: [process, workflow]


Suggested change

filter: [process, workflow]

filter: [process, workflow]

Function?

Do we have any in this repo?

Yep: https://github.com/nf-core/modules/blob/master/subworkflows/nf-core/utils_nfcore_pipeline/main.nf

Also, I'd like to encourage people to put this stuff in modules or a plugin so we don't fix it to the template.

adamrtalbot · 2024-09-30T16:51:17Z

.github/workflows/pytest-workflow.yml

+  confirm-pass:
+    runs-on: ubuntu-latest
+    needs: [pytest-changes, pytest]
+    if: always()
+    steps:
+      - name: All tests ok
+        if: ${{ success() || !contains(needs.*.result, 'failure') }}
+        run: exit 0
+      - name: One or more tests failed
+        if: ${{ contains(needs.*.result, 'failure') }}
+        run: exit 1
+
+      - name: debug-print
+        if: always()
+        run: |
+          echo "toJSON(needs) = ${{ toJSON(needs) }}"
+          echo "toJSON(needs.*.result) = ${{ toJSON(needs.*.result) }}"


You need to add one of these to every dynamic matrix, so it must be added to all the linting workflow as well.

Yeah, I left it out orginally because I'm hoping to get everything to run in pre-commit and we can do away with the filters and make the branch protections happy.

There's no way of forcing someone to use a pre-commit and I'm wary of forcing people to install stuff on their laptops. In general, it's a good idea to verify code at the source of truth (github), not in isolation. It's much easier to prevent bad code being added then remove it later.

Perhaps we could run pre-commit before merging? Never really looked into it.

https://github.com/nf-core/modules/blob/3c464e75051db485c1b37ab9f1ea2182fb3d3533/.github/workflows/test.yml#L29C2-L39C36
We already are ✨

I was talking about pre-commit the tool, not a pre-commit git hook.

Agreed on not expecting people to jump through a ton of hoops to get their local dev environments set up. (But I think this might be over since gitpod switched to self-hosted, but they also adopt devcontainers, which if we can get those setup well, anyone with vscode and docker should be able to start hacking quickly.)

adamrtalbot · 2024-09-30T18:30:30Z

Also - the GHA says it uses actions/setup-java v4 but so does master and they're in conflict 😬

You should add v4.4.0 for that action as a comment since you are referencing an explicit commit.

adamrtalbot · 2024-09-30T18:35:27Z

Stress test: #6716

adamrtalbot

Oh just spotted. Currently the tests aren't doing anything (this is every test!):

🚀 nf-test 0.9.0
https://www.nf-test.com/
(c) 2021 - 2024 Lukas Forer and Sebastian Schoenherr

Load .nf-test/plugins/nft-bam/0.3.0/nft-bam-0.3.0.jar
Load .nf-test/plugins/nft-vcf/1.0.7/nft-vcf-1.0.7.jar
Nothing to do.

.github/actions/nf-test-action/action.yml

.github/workflows/lint.yml

Co-authored-by: mashehu <mashehu@users.noreply.github.com>

mashehu · 2024-11-18T16:25:21Z

.github/actions/nf-test-action/action.yml

        nextflow secrets set SENTIEON_AUTH_DATA $(python3 tests/modules/nf-core/sentieon/license_message.py encrypt --key "$SENTIEON_ENCRYPTION_KEY" --message "$SENTIEON_LICENSE_MESSAGE")

+      # TODO If conda-fail.yml exists and matrix = conda skip
+      # Really we need in-verse tag selection https://github.com/askimed/nf-test/issues/260


while we wait for that, maybe we can add it here: adamrtalbot/detect-nf-test-changes#12

.github/workflows/nf-test.yml

.github/workflows/pytest-workflow.yml

.pre-commit-config.yaml

mashehu · 2024-11-18T16:32:23Z

tests/config/nf-test.config

-        singularity.runOptions  = '--no-mount tmp --writable-tmpfs --nv'
-        use_gpu                 = true
-    }
    docker_self_hosted {


the other gpu profile is outdated, needs platform argument

So cut the other one?

Well doesn't matter which one to keep, but make sure it has all the settings from the one in master

There's two on master though 😆

modules/tests/config/nf-test.config

Lines 35 to 51 in af84433

gpu {

docker.runOptions = '-u $(id -u):$(id -g) --gpus device=0'

apptainer.runOptions = '--no-mount tmp --writable-tmpfs --nv'

singularity.runOptions = '--no-mount tmp --writable-tmpfs --nv'

use_gpu = true

}

docker_self_hosted {

docker.enabled = true

docker.fixOwnership = true

docker.runOptions = '--platform=linux/amd64'

}

gpu {

docker.runOptions = '-u $(id -u):$(id -g) --gpus all'

apptainer.runOptions = '--nv'

singularity.runOptions = '--nv'

use_gpu = true

}

Co-authored-by: mashehu <mashehu@users.noreply.github.com>

https://github.com/nf-core/modules/pull/6286/files#r1846856369 Co-authored-by: mashehu <mashehu@users.noreply.github.com>

modules/nf-core/parabricks/applybqsr/main.nf

askimed/nf-test#272

For science

This reverts commit d134e78.

edmundmiller · 2024-11-18T23:47:18Z

Going to merge since the broken tests aren't in the scope, and hopefully I catch any buygs before Europe wakes up 🤞🏻

edmundmiller self-assigned this Aug 23, 2024

edmundmiller force-pushed the fresh-ci branch 3 times, most recently from 489e5fb to 26653c8 Compare August 23, 2024 17:36

edmundmiller mentioned this pull request Aug 26, 2024

Pass a list of modules/subworkflows(files) to nf-core modules lint nf-core/tools#3140

Open

edmundmiller mentioned this pull request Sep 18, 2024

Caused by: java.nio.channels.OverlappingFileLockException #6664

Closed

edmundmiller force-pushed the fresh-ci branch from 6030ae8 to 5594071 Compare September 18, 2024 16:52

edmundmiller force-pushed the fresh-ci branch 3 times, most recently from 9cd5419 to 8748f04 Compare September 18, 2024 21:01

edmundmiller marked this pull request as ready for review September 18, 2024 21:03

edmundmiller requested review from KevinMenden and drpatelh as code owners September 18, 2024 21:03

edmundmiller requested review from a team, ewels and maxulysse September 18, 2024 21:03

edmundmiller linked an issue Sep 18, 2024 that may be closed by this pull request

Caused by: java.nio.channels.OverlappingFileLockException #6664

Closed

edmundmiller mentioned this pull request Sep 24, 2024

In-verse tag selection askimed/nf-test#260

Open

edmundmiller force-pushed the fresh-ci branch from da25f80 to 2516a34 Compare September 24, 2024 13:54

edmundmiller mentioned this pull request Sep 27, 2024

ci: One changes job to rule them all #4284

Closed

edmundmiller requested a review from adamrtalbot September 30, 2024 13:18

adamrtalbot reviewed Sep 30, 2024

View reviewed changes

adamrtalbot previously requested changes Sep 30, 2024

View reviewed changes

edmundmiller added 2 commits November 17, 2024 16:39

ci: Try removing path

b477ac3

ci: Add fetch-depth and remove duplicate checkout

c47e407

edmundmiller force-pushed the fresh-ci branch from d924406 to c47e407 Compare November 18, 2024 02:01

edmundmiller added 3 commits November 17, 2024 20:41

test: Remove the second gpu profile

e04a743

ci: Add path back in

c7984e5

ci(gpu): Remove changed-since

fc7247a

mashehu reviewed Nov 18, 2024

View reviewed changes

.github/actions/nf-test-action/action.yml Outdated Show resolved Hide resolved

.github/workflows/lint.yml Outdated Show resolved Hide resolved

ci: Fix missed dorny/paths-filter update

ec92b74

Co-authored-by: mashehu <mashehu@users.noreply.github.com>

mashehu reviewed Nov 18, 2024

View reviewed changes

mashehu requested changes Nov 18, 2024

View reviewed changes

edmundmiller mentioned this pull request Nov 18, 2024

Handle GitHub runners possibly running out of space #7016

Open

Add changes from codereview

3f5fe11

Co-authored-by: mashehu <mashehu@users.noreply.github.com>

edmundmiller mentioned this pull request Nov 18, 2024

Create conda-fail.yml on Conda CI failures #7017

Open

style: Remove example TODO comments

9d05040

https://github.com/nf-core/modules/pull/6286/files#r1846856369 Co-authored-by: mashehu <mashehu@users.noreply.github.com>

edmundmiller force-pushed the fresh-ci branch from 0764ed0 to f97c351 Compare November 18, 2024 18:25

mashehu reviewed Nov 18, 2024

View reviewed changes

modules/nf-core/parabricks/applybqsr/main.nf Outdated Show resolved Hide resolved

edmundmiller added 4 commits November 18, 2024 12:57

refactor: Try tags for GPU CI

e04d248

askimed/nf-test#272

style: Set NFT_WORKDIR as an ENV variable

9863eba

style: Run lsp format on a parabricks module

d134e78

For science

test: Add gpu tags to parabricks

a5557af

edmundmiller force-pushed the fresh-ci branch from 3ab97fa to a5557af Compare November 18, 2024 18:57

ci: Split tags out of matrix

08a16d4

This was referenced Nov 18, 2024

Harshil alignment of inputs and outputs nextflow-io/language-server#71

Open

Address limitations of using nf-test sharding #7018

Open

mashehu approved these changes Nov 18, 2024

View reviewed changes

edmundmiller added 2 commits November 18, 2024 15:28

Revert "style: Run lsp format on a parabricks module"

ee7ffc0

This reverts commit d134e78.

ci: Fix name

c0771d6

edmundmiller merged commit 3a89745 into master Nov 18, 2024

edmundmiller deleted the fresh-ci branch November 18, 2024 23:47

	gpu {
	docker.runOptions = '-u $(id -u):$(id -g) --gpus device=0'
	apptainer.runOptions = '--no-mount tmp --writable-tmpfs --nv'
	singularity.runOptions = '--no-mount tmp --writable-tmpfs --nv'
	use_gpu = true
	}
	docker_self_hosted {
	docker.enabled = true
	docker.fixOwnership = true
	docker.runOptions = '--platform=linux/amd64'
	}
	gpu {
	docker.runOptions = '-u $(id -u):$(id -g) --gpus all'
	apptainer.runOptions = '--nv'
	singularity.runOptions = '--nv'
	use_gpu = true
	}

Conversation

edmundmiller commented Aug 23, 2024

Uh oh!

edmundmiller commented Aug 26, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

edmundmiller commented Sep 18, 2024

Uh oh!

edmundmiller commented Sep 18, 2024

Uh oh!

maxulysse commented Sep 19, 2024

Uh oh!

edmundmiller commented Sep 24, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

adamrtalbot left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

adamrtalbot Sep 30, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

edmundmiller Oct 17, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

adamrtalbot commented Sep 30, 2024

Uh oh!

adamrtalbot commented Sep 30, 2024

Uh oh!

adamrtalbot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

edmundmiller commented Nov 18, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

edmundmiller commented Aug 26, 2024 •

edited

Loading

edmundmiller commented Sep 24, 2024 •

edited

Loading

adamrtalbot Sep 30, 2024 •

edited

Loading

edmundmiller Oct 17, 2024 •

edited

Loading