Galaxy tool wrapper for Visinity #86

CameronFRWatson · 2024-08-15T20:50:57Z

Galaxy tool wrapper for visinity

There are two tools. One pre-processes inputs and creates an archive. The second is an interactive tool for viewing the Visinity dashboard

This PR is related to

Adding a new tool
Updating an existing tool to a newer version
Fixing a bug or updating just the Galaxy wrapper of an existing tool
Making a change to the tools-mti repo, CI, or other misc. change

If updating an existing tool to a newer major version

I have updated the TOOL_VERSION token in the tool's macros file
I have reset the VERSION_SUFFIX to 0 in the tool's macros file

Provide details here

Adding as a tool suite since there are two tools, although unsure if this is the right move

…so replaced with has_size assertion

bgruening · 2024-08-16T06:14:22Z

tools/visinity/macros.xml

+<macros>
+    <xml name="requirements">
+        <requirements>
+            <container type="docker">quay.io/sargentl/vis_test:0.0.3</container>


Should this still be named "test"?

nope! this PR isn't quite ready for merger, i still have cleanup to do. this is a fork of mine as visinity had a pretty serious show-stopping bug when i started developing this IT; they've since updated it, so my plan is to ensure the new update works then use that container

bgruening · 2024-08-16T06:15:09Z

tools/visinity/macros.xml

+        <stdio>
+            <exit_code range="1:" level="fatal" description="Error occurred. Please check Tool Standard Error" />
+        </stdio>


This can be removed by detect_errors on the command tag.

bgruening · 2024-08-16T06:15:56Z

tools/visinity/vis-preprocess.xml

+    cp inputs.json /visinity/ &&
+    cd /visinity/ &&
+    cp "$__tool_directory__/archive_generate.py" "./archive_generate.py" &&
+    CONDAFY="micromamba run /bin/bash -c" &&


What is mamba here doing?
Please note that mamba is deprecated.

i used a micromamba base image for easy installation while developing, as conda was visinity's recommended installation method. re: deprecation, do you have links for that? i would love to read more and wasn't able to find much with a cursory googling. the main mamba repo seems active; the closest i could find is this conversation:

Shouldn't we rather deprecate mamba? mamba-org/mamba#3054

...where they debate the validity/utility of mamba, but seem to support continuing with micromamba development

what is your preference here? i don't super care what we use as long as it works.

bgruening · 2024-08-16T06:17:29Z

tools/visinity/vis-preprocess.xml

+	    <inputs name='inputs' filename='inputs.json' data_style='paths'/>
+    </configfiles>
+    <inputs>
+        <param label='Dataset Name CSV' name='dataset_names' type='data' format='csv' help="Single column CSV file with IDs for each image in the cohort under column titled names"/>


"name" could be misleading, maybe Dataset with image IDs?

bgruening · 2024-08-16T06:18:47Z

tools/visinity/vis-preprocess.xml

+            </output>
+            <!-- FIXME: replace with chunk below once CI is upgraded to Galaxy v24 -->
+            <!-- <output name="out_archive">
+                <has_archive_member path="./config.json"/>


has_archive_member should work in older Galaxy releases I think

bgruening · 2024-08-16T06:19:30Z

tools/visinity/archive_generate.py

+requests = app.test_client()
+
+args = sys.argv[1:]
+print(f"args: {args}")


more debug cruft ;(

will remove

bgruening · 2024-08-16T06:21:56Z

tools/visinity/vis-postprocess.xml

@@ -0,0 +1,69 @@
+<tool id="visinity_postprocess" name="Visualize Visinity Archive" tool_type="interactive" version="@TOOL_VERSION@+galaxy@VERSION_SUFFIX@" profile="22.01">


Interactive tools should not go into the TS (yet). Please submit them to Galaxy directly.

Maybe it would be also nice to create a visinity.zip datatype?

sounds good; i selected zip on the assumption that it should be a datatype already in galaxy so admins wouldn't have to add a custom datatype, but if the IT can be added to galaxy directly, so too can the datatype definition.

two concerns:

anyone who wants to use the tool before PR acceptance would have to pull that branch or modify galaxy manually

would it be backported to previous versions of galaxy?

also, does:

Please submit them to Galaxy directly
... mean this should NOT go in this repo? or simply that it should be omitted from TS inclusion, and still go in this repo?

Adding a new datatype is not complicated. In your case it only changing the datatypes_conf.xml file and adding one line that subclasses from zip.

anyone who wants to use the tool before PR acceptance would have to pull that branch or modify galaxy manually

You would need to copy this file into your Galaxy, yes. Most deployments run from a fork on the main repo eitherway, I think.

would it be backported to previous versions of galaxy?

You could try. I don't think we have a rule for this yet.
ITs should go to the TS at some point we just never got time to finish that off.

... mean this should NOT go in this repo? or simply that it should be omitted from TS inclusion, and still go in this repo?

You can host it here, if you like. But it should not hit the TS. I would maintain it upstream in the Galaxy main repo.

bgruening · 2024-08-16T06:23:13Z

tools/visinity/vis-postprocess.xml

+
+Running Visinity in Galaxy requires execution of two separate tools in succession:
+
+1. Create Visinity Archive 


Just out of interest how long does the first tool take? Is it computational intensive?

there are a couple reasons for the two step process:

as you suspected, time/compute is a factor; as an example, a medium sized invocation on an 8vCPU/64GB RAM took about 15 minutes. with the two step process, you only have to create the archive once, then the IT can start visualizations immediately, and it can be re-visualized at a later time without repeating this processing.

the pre processing is much more resource intensive than visualization, and a visualization session can last hours as a user pokes around the UI; the cost savings on our AWS infrastructure is pretty big if we use a more modest instance

happy to hear thoughts/alternatives! we also wanted the visualizer to (eventually) be able to output things the user specified in the UI; we could unify the tools i suppose; something like:

if user is preprocessing, output an archive in addition to other output

user waits a while for the IT to start

if the user provides a preprocessed archive, skip preprocessing

the user does not wait a while for the IT to start

imo it comes down to opinion/convention as to which route to take. i am happy to do whatever everyone feels is best, this is merely my minimum viable product pass :D

Makes total sense! We should not waste a lot of memory and CPUs for the Visualisation. Thanks for the details!

bgruening · 2024-08-16T06:23:30Z

tools/visinity/vis-preprocess.xml

+        <import>macros.xml</import>
+    </macros>
+    <expand macro="requirements"/>
+    <expand macro="stdio"/>


Suggested change

<expand macro="stdio"/>

luke-c-sargent and others added 7 commits August 13, 2024 11:11

first pass at visinity preprocessor and IT

9fd2004

Clean up visinity tool XML and add macros file

fe57e70

Add visinity test data and help page images

039bb8c

add visnity shed file

248e106

Replace test-data and fix test for preprocessor

ebacfef

fix shed file

5a828f9

gitignore causes sim_size against zip file in test-data to not work, …

ded8cfc

…so replaced with has_size assertion

bgruening reviewed Aug 16, 2024

View reviewed changes

CameronFRWatson marked this pull request as draft August 16, 2024 16:29

luke-c-sargent added 3 commits September 4, 2024 14:03

various changes after feedback

00049d3

requesting a non-shed tool be skipped via .tt_skip

4f454a9

making the environment creation mechanic more robust

4f5b75f

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Galaxy tool wrapper for Visinity #86

Galaxy tool wrapper for Visinity #86

CameronFRWatson commented Aug 15, 2024

bgruening Aug 16, 2024

luke-c-sargent Aug 16, 2024

bgruening Aug 16, 2024

bgruening Aug 16, 2024

luke-c-sargent Aug 16, 2024

bgruening Aug 16, 2024

bgruening Aug 16, 2024

bgruening Aug 16, 2024

luke-c-sargent Aug 16, 2024

bgruening Aug 16, 2024

luke-c-sargent Aug 16, 2024

luke-c-sargent Aug 16, 2024

bgruening Aug 18, 2024

bgruening Aug 16, 2024

luke-c-sargent Aug 16, 2024

bgruening Aug 18, 2024

bgruening Aug 16, 2024

		@@ -0,0 +1,69 @@
		<tool id="visinity_postprocess" name="Visualize Visinity Archive" tool_type="interactive" version="@TOOL_VERSION@+galaxy@VERSION_SUFFIX@" profile="22.01">


		Running Visinity in Galaxy requires execution of two separate tools in succession:

		1. Create Visinity Archive

Galaxy tool wrapper for Visinity #86

Are you sure you want to change the base?

Galaxy tool wrapper for Visinity #86

Conversation

CameronFRWatson commented Aug 15, 2024

Provide details here

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment