Plugin design - dev meetings.

[abhi18av]

• Implement the NomadService to generate the expected JSON

• Explore the use of template to send the command.* files to Nomad client

• Implement the file transfer logic in .command.run file

• Accommodate the various specifications from TaskRun within NomadJobOperations

• Improve the polling logic for Nomad Job latest allocation #21

Implement Nomad Job/Task level settings in configuration

• clientToken Add clientoken #24
• clientCertificate https://developer.hashicorp.com/nomad/docs/commands#mtls-environment-variables
• jobAffinity add affinity spec configuration #37
• jobConstraint add constraint spec #40
• taskEnv
• jobVolume Mount volume spec #22
• jobMeta
• jobReschedule
• jobRestart
• taskLogs
• taskResources Accommodate Nextflow CPU/Memory in Nomad Job taskResources  #27

[abhi18av]

@jagedn , I think that moving forward this will be our core focus.

Feel free to add any comments or sub-tasks here :)

[abhi18av]

@jagedn , I was wondering whether you could please add a new gradle task, which publishes the latest version of the plugin to ~/.nextflow/plugins/ ?

Something similar to copyPluginZip but only for the locally built class files?

Also, please let me know if a better solution would be to just dump all of this into a shell script and run that instead.

[jagedn]

yes, I can try to apply same approach I did in nf-plugin-template (https://github.com/edn-es/nf-plugin-template/tree/main/buildSrc/src/main/groovy/nextflow/plugins)

[abhi18av]

Dev Meeting notes 14-Jun-2024

Participants: @jagedn @abhi18av

1. We tested the nf-core/demo pipeline and it finished successfully with the local-dev cluster (from validation folder)

2. Next we test other popular nf-core AND community pipelines

3. Improve error reporting and debugging experience (TODO)

[abhi18av]

Dev Meeting notes 21-Jun-2024

Participants: @jagedn @abhi18av

1. Publish v0.0.3

2. Add this to the main Nextflow plugin index

3. Documentation updates for setting

4. Setup a VM for next Friday (nomad-demo) recording session (30-40 minutes)

• nf-core/demo
• bactopia

==========

For next milestones (v0.0.4 onwards) we'll develop them in accordance with specific pipelines and the experience we have.

Some tasks we'd like to cover

• Configuration for nomadlab(s)

• Create a Github based release process (0.0.4) Testing infrastructure with Github Actions using custom-runners on nf-core infra #3

[abhi18av]

Dev meeting notes

Participants: @jagedn @abhi18av

1. Next week's focus would be the iteration on the release process

2. Update the doc for our custom gradle setup

3. Implement the Github Action to (@jhaezebr @matthdsm , you're help would be appreciated here to accommodate and implement @jagedn template 🙏 )

   a. Create commit, add the version and a specific command to trigger the action

   b. Action will create, publish the artifacts (+ the plugin json)

4. Also, we created a video to demo the usage of the plugin

[abhi18av]

Dev meeting notes 05-JULY-2024

Participants: @jagedn @abhi18av

1. Cover the security aspects -> Policy and SSL certs.
2. (For the week after) -> run-nf-nomad.sh script maybe via profiles
3. Specialise the three environments (local, CSI , Fusion)
4. Improve the documentation for all three environments, for agents, directives.
5. Test with Tower setup
6. Apply for Academic usage license from Hashicorp

[abhi18av]

From #72 (comment)

I'd rather prioritise the following

1. Improvement of Nextflow->Nomad layer, test with more Nextflow directives
2. Improvement of the policy and TLS setup
3. An interesting exploration would be the Nextflow secrets -> Nomad variables integration so that the secrets are not revealed in Nomad portal 🤓

[abhi18av]

Dev meeting notes 26-JULY-2024

Participants: @jagedn @abhi18av

For August, Jorge would be on a break and would be back in September. 🏖️ 🇺🇸

1. Finalize the current PR (secrets) and make a release of v0.1.2 Iteration on Nomad/Variables <-> Nextflow/Secrets #75
2. Abhinav to explore the use-case of shared resources between different research groups https://developer.hashicorp.com/nomad/docs/concepts/acl
3. Focus on actual documenting use-cases with pipelines and data-analysis

[jhaezebr]

Have a nice vacation Jorge! :)
I'm currently taking over Matthias' daily workload as he is on leave now, so I haven't done much yet for my parts.
But once he's back I'll hop back into this project.

[abhi18av]

Dev meeting notes 20-SEP-2024

Participants: @jagedn @abhi18av

1. @jagedn Finalize the job query failure and retries issues Implement an exception handler for api client #72

2. @abhi18av Improve the Tower connection Improve job-status query logic and nf-tower support  #77

3. Once these are done, we release version 0.4.0 and focus on paper + docs + presentation for the conference! 🇪🇸