Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Ship a package-to-conda mapping file with Galaxy #1927

Closed
jmchilton opened this issue Mar 15, 2016 · 6 comments
Closed

Ship a package-to-conda mapping file with Galaxy #1927

jmchilton opened this issue Mar 15, 2016 · 6 comments
Labels
area/tool-framework area/tools
Milestone
17.05

Comments

@jmchilton
Copy link
Member

When abstract packages don't map directly to conda packages (e.g. https://github.com/galaxyproject/tools-iuc/pull/642/files) a file inside of Galaxy should provide a concrete mapping.

For local testing, planemo should read some file or something in .shed.yml to override this for local testing (conda_dependencies.xml 😉) - but Galaxy will have no concept of this file.
@jmchilton jmchilton added this to the 16.04 milestone Mar 15, 2016
@martenson martenson mentioned this issue Mar 15, 2016
Closed
@martenson martenson modified the milestones: 16.07, 16.04 Apr 5, 2016
@nsoranzo nsoranzo mentioned this issue Apr 6, 2016
Closed
@nsoranzo
Copy link
Member

In https://github.com/galaxyproject/tools-iuc/pull/710/files/26e320ba917e0d8e3fb472fe5777ec3f5dcc7bdd#r58512818 I proposed an alternative solution: a new optional resolver attribute for <requirement type="package"> whose value would be a comma-separated list of strings among tool_shed_packages, conda and the other resolvers specified in config/dependency_resolvers_conf.xml.sample .

When found, this attribute will instruct Galaxy to resolve this requirement only with one of the specified resolvers (in the order specified in config/dependency_resolvers_conf.xml). If the attribute is not specified, Galaxy will handle the requirement as it is doing now.

I think this would be a cleaner and more transparent solution than the package-to-conda mapping file.

@martenson martenson modified the milestones: 16.10, 16.07 Jul 27, 2016
@lparsons
Copy link
Contributor

I can see the merits of both solutions. I like having a mapping file that is controlled by the Galaxy admin, as an admin I like to have control over how things are installed. However, as a tool author, it would be very nice to have some mechanism to specify attributes for specific resolvers when necessary. There will inevitably be cases where homebrew package names differ from conda package names. A simple and sane default is great, and having the flexibility to specify exceptions is also very useful.

@nsoranzo nsoranzo mentioned this issue Oct 1, 2016
Closed
@nsoranzo nsoranzo mentioned this issue Nov 2, 2016
Closed
@peterjc
Copy link
Contributor

peterjc commented Nov 2, 2016

+1 on explicit resolvers for cases where different packaging sources have adopted different names for the same package (e.g. for NCBI BLAST vs BLAST+).

jmchilton added a commit to jmchilton/galaxy that referenced this issue Nov 2, 2016
@jmchilton
Allow finer grain specification of requirements.
90a4f87 
This adapts CWL-style ``SoftwareRequirement`` ``specs`` to solve galaxyproject#1927. Here I'm trying to implement the CWL specification in a way that helps enable the feasibility of Conda packaging in Galaxy. It is a delicate balancing act aimed to upset as many interested parties as I can.

To understand the problem - consider the ``blast+`` requirement found in the Galaxy wrappers. It looks something like this:

```
<requirement type="package" version="2.2.31" name="blast+">
```

Great, that works for Galaxy and Tool Shed packages. It doesn't work for bioconda at all. I think this problem is fairly uncommon - most packages have simple names shared across Debian, Brew, Conda, etc... - but it does happen in some cases that there are inconsistencies. Some people have taken to duplicating the requirement - this is bad and should not be done since they are mutually exclusive and Galaxy will attempt to resolve both.

This introduces the following syntax for tools with profile >= 16.10:

```
<requirement type="package" version="2.2.31" name="blast+">
    <specification uri="https://anaconda.org/bioconda/blast" />
    <specification uri="https://packages.debian.org/sid/ncbi-blast+" version="2.2.31-3" />
</requirement>
```

This allows finer grain resolution of the requirement without sacrificing the abstract name at the top. It allows the name and the version to be adapted by resolvers as needed (hopefully rarely so).

This syntax is the future facing one, but obviously this tool would not work on older Galaxy versions. To remedy this - an alternative syntax can be used for tools targetting Galaxy verions pre-16.10:

```
<requirement type="package" version="2.2" specification_uris="https://anaconda.org/bioconda/blast@2.2.31,https://packages.debian.org/jessie/ncbi-blast+@2.2.29-3">blast+</requirement>
```

This syntax sucks - but it does give newer Galaxies the ability to resolve the specifications without breaking the more simple functionality for older Galaxies.

For more information on the CWL side of this - checkout the discussion on common-workflow-language/cwltool#214. The CWL specification information is defined at http://www.commonwl.org/v1.0/CommandLineTool.html#SoftwarePackage.
@jmchilton jmchilton mentioned this issue Nov 2, 2016
Closed
@peterjc peterjc mentioned this issue Nov 7, 2016
Merged
jmchilton added a commit to jmchilton/galaxy that referenced this issue Nov 10, 2016
@jmchilton
Allow finer grain specification of requirements.
f71dcbb 
This adapts CWL-style ``SoftwareRequirement`` ``specs`` to solve galaxyproject#1927. Here I'm trying to implement the CWL specification in a way that helps enable the feasibility of Conda packaging in Galaxy. It is a delicate balancing act aimed to upset as many interested parties as I can.

To understand the problem - consider the ``blast+`` requirement found in the Galaxy wrappers. It looks something like this:

```
<requirement type="package" version="2.2.31" name="blast+">
```

Great, that works for Galaxy and Tool Shed packages. It doesn't work for bioconda at all. I think this problem is fairly uncommon - most packages have simple names shared across Debian, Brew, Conda, etc... - but it does happen in some cases that there are inconsistencies. Some people have taken to duplicating the requirement - this is bad and should not be done since they are mutually exclusive and Galaxy will attempt to resolve both.

This introduces the following syntax for tools with profile >= 16.10:

```
<requirement type="package" version="2.2.31" name="blast+">
    <specification uri="https://anaconda.org/bioconda/blast" />
    <specification uri="https://packages.debian.org/sid/ncbi-blast+" version="2.2.31-3" />
</requirement>
```

This allows finer grain resolution of the requirement without sacrificing the abstract name at the top. It allows the name and the version to be adapted by resolvers as needed (hopefully rarely so).

This syntax is the future facing one, but obviously this tool would not work on older Galaxy versions. To remedy this - an alternative syntax can be used for tools targetting Galaxy verions pre-16.10:

```
<requirement type="package" version="2.2" specification_uris="https://anaconda.org/bioconda/blast@2.2.31,https://packages.debian.org/jessie/ncbi-blast+@2.2.29-3">blast+</requirement>
```

This syntax sucks - but it does give newer Galaxies the ability to resolve the specifications without breaking the more simple functionality for older Galaxies.

For more information on the CWL side of this - checkout the discussion on common-workflow-language/cwltool#214. The CWL specification information is defined at http://www.commonwl.org/v1.0/CommandLineTool.html#SoftwarePackage.
@martenson martenson modified the milestones: 17.01, 16.10 Nov 16, 2016
@martenson martenson modified the milestones: 17.05, 17.01 Jan 12, 2017
jmchilton added a commit to jmchilton/galaxy that referenced this issue Jan 18, 2017
@jmchilton
Allow finer grain specification of requirements.
9e5feb8 
This adapts CWL-style ``SoftwareRequirement`` ``specs`` to solve galaxyproject#1927. Here I'm trying to implement the CWL specification in a way that helps enable the feasibility of Conda packaging in Galaxy. It is a delicate balancing act aimed to upset as many interested parties as I can.

To understand the problem - consider the ``blast+`` requirement found in the Galaxy wrappers. It looks something like this:

```
<requirement type="package" version="2.2.31" name="blast+">
```

Great, that works for Galaxy and Tool Shed packages. It doesn't work for bioconda at all. I think this problem is fairly uncommon - most packages have simple names shared across Debian, Brew, Conda, etc... - but it does happen in some cases that there are inconsistencies. Some people have taken to duplicating the requirement - this is bad and should not be done since they are mutually exclusive and Galaxy will attempt to resolve both.

This introduces the following syntax for tools with profile >= 16.10:

```
<requirement type="package" version="2.2.31" name="blast+">
    <specification uri="https://anaconda.org/bioconda/blast" />
    <specification uri="https://packages.debian.org/sid/ncbi-blast+" version="2.2.31-3" />
</requirement>
```

This allows finer grain resolution of the requirement without sacrificing the abstract name at the top. It allows the name and the version to be adapted by resolvers as needed (hopefully rarely so).

This syntax is the future facing one, but obviously this tool would not work on older Galaxy versions. To remedy this - an alternative syntax can be used for tools targetting Galaxy verions pre-16.10:

```
<requirement type="package" version="2.2" specification_uris="https://anaconda.org/bioconda/blast@2.2.31,https://packages.debian.org/jessie/ncbi-blast+@2.2.29-3">blast+</requirement>
```

This syntax sucks - but it does give newer Galaxies the ability to resolve the specifications without breaking the more simple functionality for older Galaxies.

For more information on the CWL side of this - checkout the discussion on common-workflow-language/cwltool#214. The CWL specification information is defined at http://www.commonwl.org/v1.0/CommandLineTool.html#SoftwarePackage.

Conflicts:
	lib/galaxy/tools/deps/__init__.py
	test/functional/tools/samples_tool_conf.xml
jmchilton added a commit to jmchilton/galaxy that referenced this issue Jan 18, 2017
@jmchilton
Allow finer grain specification of requirements.
81d71d2 
This adapts CWL-style ``SoftwareRequirement`` ``specs`` to solve galaxyproject#1927. Here I'm trying to implement the CWL specification in a way that helps enable the feasibility of Conda packaging in Galaxy. It is a delicate balancing act aimed to upset as many interested parties as I can.

To understand the problem - consider the ``blast+`` requirement found in the Galaxy wrappers. It looks something like this:

```
<requirement type="package" version="2.2.31" name="blast+">
```

Great, that works for Galaxy and Tool Shed packages. It doesn't work for bioconda at all. I think this problem is fairly uncommon - most packages have simple names shared across Debian, Brew, Conda, etc... - but it does happen in some cases that there are inconsistencies. Some people have taken to duplicating the requirement - this is bad and should not be done since they are mutually exclusive and Galaxy will attempt to resolve both.

This introduces the following syntax for tools with profile >= 16.10:

```
<requirement type="package" version="2.2.31" name="blast+">
    <specification uri="https://anaconda.org/bioconda/blast" />
    <specification uri="https://packages.debian.org/sid/ncbi-blast+" version="2.2.31-3" />
</requirement>
```

This allows finer grain resolution of the requirement without sacrificing the abstract name at the top. It allows the name and the version to be adapted by resolvers as needed (hopefully rarely so).

This syntax is the future facing one, but obviously this tool would not work on older Galaxy versions. To remedy this - an alternative syntax can be used for tools targetting Galaxy verions pre-16.10:

```
<requirement type="package" version="2.2" specification_uris="https://anaconda.org/bioconda/blast@2.2.31,https://packages.debian.org/jessie/ncbi-blast+@2.2.29-3">blast+</requirement>
```

This syntax sucks - but it does give newer Galaxies the ability to resolve the specifications without breaking the more simple functionality for older Galaxies.

For more information on the CWL side of this - checkout the discussion on common-workflow-language/cwltool#214. The CWL specification information is defined at http://www.commonwl.org/v1.0/CommandLineTool.html#SoftwarePackage.

Conflicts:
	lib/galaxy/tools/deps/__init__.py
	test/functional/tools/samples_tool_conf.xml
@jmchilton jmchilton mentioned this issue Jan 18, 2017
Merged
jmchilton added a commit to jmchilton/galaxy that referenced this issue Jan 23, 2017
@jmchilton
Allow finer grain specification of requirements.
38f0dfb 
This adapts CWL-style ``SoftwareRequirement`` ``specs`` to solve galaxyproject#1927. Here I'm trying to implement the CWL specification in a way that helps enable the feasibility of Conda packaging in Galaxy. It is a delicate balancing act aimed to upset as many interested parties as I can.

To understand the problem - consider the ``blast+`` requirement found in the Galaxy wrappers. It looks something like this:

```
<requirement type="package" version="2.2.31" name="blast+">
```

Great, that works for Galaxy and Tool Shed packages. It doesn't work for bioconda at all. I think this problem is fairly uncommon - most packages have simple names shared across Debian, Brew, Conda, etc... - but it does happen in some cases that there are inconsistencies. Some people have taken to duplicating the requirement - this is bad and should not be done since they are mutually exclusive and Galaxy will attempt to resolve both.

This introduces the following syntax for tools with profile >= 16.10:

```
<requirement type="package" version="2.2.31" name="blast+">
    <specification uri="https://anaconda.org/bioconda/blast" />
    <specification uri="https://packages.debian.org/sid/ncbi-blast+" version="2.2.31-3" />
</requirement>
```

This allows finer grain resolution of the requirement without sacrificing the abstract name at the top. It allows the name and the version to be adapted by resolvers as needed (hopefully rarely so).

This syntax is the future facing one, but obviously this tool would not work on older Galaxy versions. To remedy this - an alternative syntax can be used for tools targetting Galaxy verions pre-16.10:

```
<requirement type="package" version="2.2" specification_uris="https://anaconda.org/bioconda/blast@2.2.31,https://packages.debian.org/jessie/ncbi-blast+@2.2.29-3">blast+</requirement>
```

This syntax sucks - but it does give newer Galaxies the ability to resolve the specifications without breaking the more simple functionality for older Galaxies.

For more information on the CWL side of this - checkout the discussion on common-workflow-language/cwltool#214. The CWL specification information is defined at http://www.commonwl.org/v1.0/CommandLineTool.html#SoftwarePackage.

Conflicts:
	lib/galaxy/tools/deps/__init__.py
	test/functional/tools/samples_tool_conf.xml
jmchilton added a commit to jmchilton/galaxy that referenced this issue Jan 23, 2017
@jmchilton
Allow finer grain specification of requirements.
cd2cf49 
This adapts CWL-style ``SoftwareRequirement`` ``specs`` to solve galaxyproject#1927. Here I'm trying to implement the CWL specification in a way that helps enable the feasibility of Conda packaging in Galaxy. It is a delicate balancing act aimed to upset as many interested parties as I can.

To understand the problem - consider the ``blast+`` requirement found in the Galaxy wrappers. It looks something like this:

```
<requirement type="package" version="2.2.31" name="blast+">
```

Great, that works for Galaxy and Tool Shed packages. It doesn't work for bioconda at all. I think this problem is fairly uncommon - most packages have simple names shared across Debian, Brew, Conda, etc... - but it does happen in some cases that there are inconsistencies. Some people have taken to duplicating the requirement - this is bad and should not be done since they are mutually exclusive and Galaxy will attempt to resolve both.

This introduces the following syntax for tools with profile >= 16.10:

```
<requirement type="package" version="2.2.31" name="blast+">
    <specification uri="https://anaconda.org/bioconda/blast" />
    <specification uri="https://packages.debian.org/sid/ncbi-blast+" version="2.2.31-3" />
</requirement>
```

This allows finer grain resolution of the requirement without sacrificing the abstract name at the top. It allows the name and the version to be adapted by resolvers as needed (hopefully rarely so).

This syntax is the future facing one, but obviously this tool would not work on older Galaxy versions. To remedy this - an alternative syntax can be used for tools targetting Galaxy verions pre-16.10:

```
<requirement type="package" version="2.2" specification_uris="https://anaconda.org/bioconda/blast@2.2.31,https://packages.debian.org/jessie/ncbi-blast+@2.2.29-3">blast+</requirement>
```

This syntax sucks - but it does give newer Galaxies the ability to resolve the specifications without breaking the more simple functionality for older Galaxies.

For more information on the CWL side of this - checkout the discussion on common-workflow-language/cwltool#214. The CWL specification information is defined at http://www.commonwl.org/v1.0/CommandLineTool.html#SoftwarePackage.

Conflicts:
	lib/galaxy/tools/deps/__init__.py
	test/functional/tools/samples_tool_conf.xml
@jmchilton
Copy link
Member Author

Default mapping file now available at https://github.com/jmchilton/galaxy/blob/dev/lib/galaxy/tools/deps/resolvers/default_conda_mapping.yml in the code. Feel free to contribute new ones or simply request changes based on your experience.

@martenson
Copy link
Member

the galaxyproject's galaxy repo mapping is here: https://github.com/galaxyproject/galaxy/blob/dev/lib/galaxy/tools/deps/resolvers/default_conda_mapping.yml

@jmchilton
Copy link
Member Author

Opps thanks @martenson

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/tool-framework area/tools
Projects
None yet
Milestone
17.05
Development

No branches or pull requests
5 participants
@peterjc @jmchilton @lparsons @martenson @nsoranzo