Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

change to epytope 3.3.1 version and new multi container for peptide prediction #200

Closed
wants to merge 13 commits into from
Closed

change to epytope 3.3.1 version and new multi container for peptide prediction #200

wants to merge 13 commits into from

Conversation

jonasscheid
Copy link
Contributor

Changes:

PR checklist

  • This comment contains a description of changes (with reason).
  • If you've fixed a bug or added code that should be tested, add tests!
  • If you've added a new tool - have you followed the pipeline conventions in the contribution docs
  • If necessary, also make a PR on the nf-core/epitopeprediction branch on the nf-core/test-datasets repository.
  • Make sure your code lints (nf-core lint).
  • Ensure the test suite passes (nextflow run . -profile test,docker --outdir <OUTDIR>).
  • Usage Documentation in docs/usage.md is updated.
  • Output Documentation in docs/output.md is updated.
  • CHANGELOG.md is updated.
  • README.md is updated (including new tool citations and authors/contributors).

@github-actions
Copy link

github-actions bot commented May 3, 2023 •
edited
Loading

nf-core lint overall result: Passed ✅

Posted for pipeline commit fa3780c

+| ✅ 159 tests passed       |+

✅ Tests passed:

Run details

  • nf-core/tools version 2.9
  • Run at 2023-08-16 08:53:02

@jonasscheid
Copy link
Contributor Author 

File "/opt/conda/lib/python3.7/site-packages/epytope/IO/MartsAdapter.py", line 158, in __perform_request
[166](https://github.com/nf-core/epitopeprediction/actions/runs/4873272243/jobs/8692613955?pr=200#step:4:167)
      backoff_factor=3
[167](https://github.com/nf-core/epitopeprediction/actions/runs/4873272243/jobs/8692613955?pr=200#step:4:168)
  TypeError: __init__() got an unexpected keyword argument 'allowed_methods'

Hmm the marts adapter seems to have problems. Any ideas @christopher-mohr

@christopher-mohr
Copy link
Collaborator 

File "/opt/conda/lib/python3.7/site-packages/epytope/IO/MartsAdapter.py", line 158, in __perform_request
[166](https://github.com/nf-core/epitopeprediction/actions/runs/4873272243/jobs/8692613955?pr=200#step:4:167)
      backoff_factor=3
[167](https://github.com/nf-core/epitopeprediction/actions/runs/4873272243/jobs/8692613955?pr=200#step:4:168)
  TypeError: __init__() got an unexpected keyword argument 'allowed_methods'

Hmm the marts adapter seems to have problems. Any ideas @christopher-mohr

I guess the Requests and/or urllib3 package versions are too old.

@jonasscheid
Copy link
Contributor Author

I still don't know how to fix this.. It seems to me that nextflow does not switch into the epytope environment within the docker container, even though that is happening when executing the container locally..

@christopher-mohr
Copy link
Collaborator

I still don't know how to fix this.. It seems to me that nextflow does not switch into the epytope environment within the docker container, even though that is happening when executing the container locally..

I will have look again today.

@christopher-mohr
Copy link
Collaborator

christopher-mohr commented Jul 24, 2023 •
edited
Loading

I still don't know how to fix this.. It seems to me that nextflow does not switch into the epytope environment within the docker container, even though that is happening when executing the container locally..

It did not happen (that it switches to a conda env) now when I tested it in a Gitpod environment. So this is definietly the issue. It is using the system Python and therefore also does not have the vcf package etc.

@jonasscheid
Copy link
Contributor Author

Good to know. The problem is, that all base containers that come with an installed conda (including continuum) have higher python versions. I could not install epytope and the other dependencies in the base environment, therefore I created a new environment.

Maybe there is a way to install epytope in the base environment of these containers

@christopher-mohr
Copy link
Collaborator

Good to know. The problem is, that all base containers that come with an installed conda (including continuum) have higher python versions. I could not install epytope and the other dependencies in the base environment, therefore I created a new environment.

Maybe there is a way to install epytope in the base environment of these containers

Do we need conda at all?

@jonasscheid
Copy link
Contributor Author

Good point 🤔 we actually could try a python 3.7 container, pip install epytope and apt/yum-install gawk, tcsh and perl

@christopher-mohr
Copy link
Collaborator

Good point 🤔 we actually could try a python 3.7 container, pip install epytope and apt/yum-install gawk, tcsh and perl

Yes. that would have been my suggestion.

@jonasscheid
Copy link
Contributor Author

jonasscheid commented Jul 26, 2023 •
edited
Loading

After testing some containers I think I found one that works. Local tests for peptides and proteins runs through, however not test_full (with variants)

Command error:
WARNING: Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap.
Using TensorFlow backend.
INFO:main:Running Epitope Prediction And Annotation version: 1.1
INFO:main:Starting predictions at 2023-07-26 13:55:55
2023-07-26 13:55:55,534 - main - INFO - Running Epitope Prediction And Annotation version: 1.1
2023-07-26 13:55:55,534 - main - INFO - Starting predictions at 2023-07-26 13:55:55
ERROR:root:Attribute coding not found for dataset hsapiens_gene_ensembl on http://feb2014.archive.ensembl.org/biomart/martservice

Do we need to update biomart version in epytope again? I installed epytope 3.3.1

@christopher-mohr
Copy link
Collaborator

After testing some containers I think I found one that works. Local tests for peptides and proteins runs through, however not test_full (with variants)

Command error:
WARNING: Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap.
Using TensorFlow backend.
INFO:main:Running Epitope Prediction And Annotation version: 1.1
INFO:main:Starting predictions at 2023-07-26 13:55:55
2023-07-26 13:55:55,534 - main - INFO - Running Epitope Prediction And Annotation version: 1.1
2023-07-26 13:55:55,534 - main - INFO - Starting predictions at 2023-07-26 13:55:55
ERROR:root:Attribute coding not found for dataset hsapiens_gene_ensembl on http://feb2014.archive.ensembl.org/biomart/martservice

Do we need to update biomart version in epytope again? I installed epytope 3.3.1

If it still works with the current pipeline version then I would suspect a change in the DB handling in combination with the database version that is used in the pipeline currently. I can have a look. Is the container also already online?

@jonasscheid
Copy link
Contributor Author

Pushed new container.
docker pull ghcr.io/jonasscheid/epitopeprediction-2:0.3.0

@jonasscheid jonasscheid changed the title change to epytope 3.3.0 version and new multi container for peptide prediction change to epytope 3.3.1 version and new multi container for peptide prediction Jul 26, 2023
@christopher-mohr
Copy link
Collaborator

This seems to be an issue that even exists on their page. When I try to query for the coding sequence using this archive version on their page I get the following:

ERROR: caught BioMart::Exception::Configuration: Attribute 'coding' not found in dataset default.hsapiens_gene_ensembl

If you repeatedly get directed to this error page, there may be a problem with your current session parameters. To clear your session and start with a clean slate, please click the New button below.

Stacktrace:
Exception::Class::Base::throw /ensemblweb/archive/www_75/biomart-perl/lib/BioMart/Registry.pm:490
BioMart::Registry::getAttribute /ensemblweb/archive/www_75/biomart-perl/lib/BioMart/Query.pm:1243
BioMart::Query::addAttribute /ensemblweb/archive/www_75/biomart-perl/lib/BioMart/Web.pm:808
BioMart::Web::prepare_martquery /ensemblweb/archive/www_75/biomart-perl/lib/BioMart/Web.pm:2129
BioMart::Web::handle_request /ensemblweb/archive/www_75/ensembl-webcode/biomart-perl/cgi-bin/martview:100
(eval) /ensemblweb/archive/www_75/ensembl-webcode/biomart-perl/cgi-bin/martview:99
ModPerl::ROOT::ModPerl::Registry::ensemblweb_archive_www_75_ensembl_2dwebcode_biomart_2dperl_cgi_2dbin_martview::handler /localsw/lib/perl5/site_perl/5.14.2/x86_64-linux-thread-multi/ModPerl/RegistryCooker.pm:204
(eval) /localsw/lib/perl5/site_perl/5.14.2/x86_64-linux-thread-multi/ModPerl/RegistryCooker.pm:204
ModPerl::RegistryCooker::run /localsw/lib/perl5/site_perl/5.14.2/x86_64-linux-thread-multi/ModPerl/RegistryCooker.pm:170
ModPerl::RegistryCooker::default_handler /localsw/lib/perl5/site_perl/5.14.2/x86_64-linux-thread-multi/ModPerl/Registry.pm:31
ModPerl::Registry::handler -e:0
(eval) -e:0

I will ask them what's wrong.

christopher-mohr
christopher-mohr reviewed Jul 27, 2023
View reviewed changes
modules/local/external_tools_import.nf
@@ -52,7 +52,7 @@ process EXTERNAL_TOOLS_IMPORT {
# Substitution 3: NMHOME should be the folder in which the tcsh script itself resides
#
sed -i.bak \
-e 's_bin/tcsh.*\$_usr/bin/env tcsh_' \
-e 's_bin/tcsh.*\$_opt/conda/bin/tcsh_' \
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this changed because of the container? Shouldn't this be container-unrelated?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, that needs to be adjusted to the paths in the new container. That one was from the Conda-based container.

@christopher-mohr
Copy link
Collaborator

This seems to be an issue that even exists on their page. When I try to query for the coding sequence using this archive version on their page I get the following:

ERROR: caught BioMart::Exception::Configuration: Attribute 'coding' not found in dataset default.hsapiens_gene_ensembl

If you repeatedly get directed to this error page, there may be a problem with your current session parameters. To clear your session and start with a clean slate, please click the New button below.

Stacktrace:
Exception::Class::Base::throw /ensemblweb/archive/www_75/biomart-perl/lib/BioMart/Registry.pm:490
BioMart::Registry::getAttribute /ensemblweb/archive/www_75/biomart-perl/lib/BioMart/Query.pm:1243
BioMart::Query::addAttribute /ensemblweb/archive/www_75/biomart-perl/lib/BioMart/Web.pm:808
BioMart::Web::prepare_martquery /ensemblweb/archive/www_75/biomart-perl/lib/BioMart/Web.pm:2129
BioMart::Web::handle_request /ensemblweb/archive/www_75/ensembl-webcode/biomart-perl/cgi-bin/martview:100
(eval) /ensemblweb/archive/www_75/ensembl-webcode/biomart-perl/cgi-bin/martview:99
ModPerl::ROOT::ModPerl::Registry::ensemblweb_archive_www_75_ensembl_2dwebcode_biomart_2dperl_cgi_2dbin_martview::handler /localsw/lib/perl5/site_perl/5.14.2/x86_64-linux-thread-multi/ModPerl/RegistryCooker.pm:204
(eval) /localsw/lib/perl5/site_perl/5.14.2/x86_64-linux-thread-multi/ModPerl/RegistryCooker.pm:204
ModPerl::RegistryCooker::run /localsw/lib/perl5/site_perl/5.14.2/x86_64-linux-thread-multi/ModPerl/RegistryCooker.pm:170
ModPerl::RegistryCooker::default_handler /localsw/lib/perl5/site_perl/5.14.2/x86_64-linux-thread-multi/ModPerl/Registry.pm:31
ModPerl::Registry::handler -e:0
(eval) -e:0

I will ask them what's wrong.

They forwarded my request to the technical support team. Still waiting for feedback.

@jonasscheid
Copy link
Contributor Author

I think I found the reason for the error

This is the selection list of variant types in the "older" archives.
Screenshot 2023-08-16 at 11 14 04

since the 2018 archives however, they use Variant Consequence as key for the selection list. And in these newer versions this error is not appearing anymore.

Screenshot 2023-08-16 at 11 24 59

So I guess they did not change this key for the older versions.

Off note: coding attribute can also not be found anymore on gene types (as far as I understood)

@christopher-mohr
Copy link
Collaborator

I think I found the reason for the error

This is the selection list of variant types in the "older" archives. Screenshot 2023-08-16 at 11 14 04

since the 2018 archives however, they use Variant Consequence as key for the selection list. And in these newer versions this error is not appearing anymore.

Screenshot 2023-08-16 at 11 24 59

So I guess they did not change this key for the older versions.

Off note: coding attribute can also not be found anymore on gene types (as far as I understood)

That should not be the cause for the error we also get on their website.

They also basically provide kind of a mapping of attribute names and respective database fields when using the API. So "Variant type" and "Variant Consequence" can still point to the same field. "coding" or "coding sequence" should be available in the "Sequence" category.

Btw I asked them for an update but was told that they are still looking into that.

@jonasscheid
Copy link
Contributor Author

I see! Thanks for the explanation 🙏🏼
Fingers crossed that they'll fix it soon

@christopher-mohr
Copy link
Collaborator

I see! Thanks for the explanation 🙏🏼 Fingers crossed that they'll fix it soon

With the new BiomartAdapter in epytope we also make use of this mapping table that can be retrieved via API. So the database fields (that should be stable) can be mapped to the "names" that are needed for the queries. Therefore, we are flexible with respect to the version and should be more robust for future updates.

@alina-bauer alina-bauer mentioned this pull request Aug 22, 2023
Merged
10 tasks
@christopher-mohr
Copy link
Collaborator

christopher-mohr commented Oct 3, 2023 •
edited
Loading

Changed the Ensembl reference version handling and fixed it for the most recent and most of the archive versions in #213. I would suggest to close this PR in favor of #213.

After weeks/months? I also got an answer from the Ensembl Support. Basically, they told me that we should just switch to the most recent version. I think for this specific archive version something is broken on their end.

@jonasscheid
Copy link
Contributor Author

Thanks for the update!

Yes, makes sense to me to close this PR 👍🏼 Finally then we have epytope 3.3.1 in there 🙌🏼

@jonasscheid jonasscheid closed this Oct 3, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@christopher-mohr christopher-mohr christopher-mohr left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@jonasscheid @christopher-mohr