inject study_type in EBI and improvements to current automatic processing pipeline #3023

antgonza · 2020-08-03T21:39:05Z

…

…sing pipeline

codecov-commenter · 2020-08-03T22:03:37Z

Codecov Report

Merging #3023 into dev will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##              dev    #3023   +/-   ##
=======================================
  Coverage   94.94%   94.94%           
=======================================
  Files          74       74           
  Lines       14263    14263           
=======================================
  Hits        13542    13542           
  Misses        721      721

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d9275b7...bf32a46. Read the comment docs.

ElDeveloper

Changes look good, overall. Are the modifications to the automatic processing pipeline a result of failed runs, or what motivated those changes?

ElDeveloper · 2020-08-04T00:21:02Z

qiita_ware/ebi.py

@@ -356,6 +356,11 @@ def generate_study_xml(self):
        study_title = ET.SubElement(descriptor, 'STUDY_TITLE')
        study_title.text = escape(clean_whitespace(self.study_title))

+        # study type is depricated and not displayed anywhere on EBI-ENA;


Suggested change

# study type is depricated and not displayed anywhere on EBI-ENA;

# study type is deprecated and not displayed anywhere on EBI-ENA;

scripts/qiita-auto-processing

ElDeveloper · 2020-08-04T00:24:39Z

scripts/qiita-auto-processing

        # getting all jobs, includen hiddens, in case the job failed
        jobs = a.jobs(cmd=cmd['command'], show_hidden=True)
-        params = [j.parameters.values for j in jobs]
+        params = [{k: str(v) for k, v in j.parameters.values.items()


Why is the string conversion needed here?

Excellent question; for some reason (for old jobs) the parameters are not string and the parameters here are defined as strings; the easiest is to make sure that everything is string

ElDeveloper · 2020-08-04T00:25:51Z

scripts/qiita-auto-processing

+                            k: str(v)
+                            for k, v in cpp.values.items()
+                            if k not in cmd['ignore_parameters']}


This list comprehension pattern is repeated a couple times, any chance this can be put in a single function, or something like that?

antgonza

Good question! The changes are basically cause (1) we needed to reprocess WGS with the new pipeline (had to change versions) while the first time we only did target gene (thus, we were missing some parameters for WGS); (2) realized that we can have the same command with multiple core parameters (so we need the ignore parameters, including that some of the first version did not converted everything to string); and some general clean up.

scripts/qiita-auto-processing

antgonza · 2020-08-04T01:53:44Z

scripts/qiita-auto-processing

        # getting all jobs, includen hiddens, in case the job failed
        jobs = a.jobs(cmd=cmd['command'], show_hidden=True)
-        params = [j.parameters.values for j in jobs]
+        params = [{k: str(v) for k, v in j.parameters.values.items()


Excellent question; for some reason (for old jobs) the parameters are not string and the parameters here are defined as strings; the easiest is to make sure that everything is string

ElDeveloper · 2020-08-04T21:52:21Z

Thanks @antgonza

@ElDeveloper

* inject study_type in EBI and improvements to current automatic processing pipeline (#3023) * inject study_type in ebi and improvements to current automatic proecssing pipeline * addressing @ElDeveloper comments * some general fixes/additions for next release (#3026) * some general fixes/additions for next release * adding test for not None job.release_validator_job * fix #2839 * fix #2868 (#3028) * fix #2868 * 2nd round * fix errors * more changes * fix errors * fix ProcessingJobTest * fix PY_PATCH * add missing TRN.add * encapsulated_query -> perform_as_transaction * fix #3022 (#3030) * fix #3022 * adding tests * fix #2320 (#3031) * fix #2320 * adding prints to debug * children -> 1 * APIArtifactHandlerTest -> APIArtifactHandlerTests * configure_biom * qdb.util.activate_or_update_plugins * improving code * almost there * add values.template * fix filepaths * filepaths -> files * fixing errors * add prep.artifact insertion * addressing @ElDeveloper comments * fix artifact_definition active command * != -> == * Added three tutorial sections to the Qiita documentation (#3032) * Added three tutorial sections to the Qiita documentation: 'Retrieving Public Data for Own Analysis' and 'Processing public data retrieved with redbiom' to the redbiom tab, and 'Statistical Analysis to Justify Clinical Trial Sample Size Tutorial' to the analyzing samples tab. * Update redbiom.rst * Update redbiom.rst * Update redbiom.rst * Further updates to redbiom.rst and the Stats tutorial. * update redbiom.rst * Finished proof-reading * Placed all three tutorials/sections together under Introduction to the download and analysis of public Qiita data * added a new introduction, with links to the three sections * Added figures to stats tutorial and contexts explanation * Added figures to stats tutorial and contexts explanation * Apply suggestions from code review [skip ci] Co-authored-by: Yoshiki Vázquez Baeza <yoshiki@ucsd.edu> Co-authored-by: Antonio Gonzalez <antgonza@gmail.com> Co-authored-by: Yoshiki Vázquez Baeza <yoshiki@ucsd.edu> * 092020 (#3033) * 092020 * connect artifact with job * rm INSERT qiita.artifact_processing_job * Apply suggestions from code review [skip ci] Co-authored-by: Yoshiki Vázquez Baeza <yoshiki@ucsd.edu> Co-authored-by: Yoshiki Vázquez Baeza <yoshiki@ucsd.edu> Co-authored-by: Daniel McDonald <danielmcdonald@ucsd.edu> Co-authored-by: Mirte Kuijpers <67341505+mcmk3@users.noreply.github.com> Co-authored-by: Yoshiki Vázquez Baeza <yoshiki@ucsd.edu>

@ElDeveloper

* Version 092020 (#3034) * inject study_type in EBI and improvements to current automatic processing pipeline (#3023) * inject study_type in ebi and improvements to current automatic proecssing pipeline * addressing @ElDeveloper comments * some general fixes/additions for next release (#3026) * some general fixes/additions for next release * adding test for not None job.release_validator_job * fix #2839 * fix #2868 (#3028) * fix #2868 * 2nd round * fix errors * more changes * fix errors * fix ProcessingJobTest * fix PY_PATCH * add missing TRN.add * encapsulated_query -> perform_as_transaction * fix #3022 (#3030) * fix #3022 * adding tests * fix #2320 (#3031) * fix #2320 * adding prints to debug * children -> 1 * APIArtifactHandlerTest -> APIArtifactHandlerTests * configure_biom * qdb.util.activate_or_update_plugins * improving code * almost there * add values.template * fix filepaths * filepaths -> files * fixing errors * add prep.artifact insertion * addressing @ElDeveloper comments * fix artifact_definition active command * != -> == * Added three tutorial sections to the Qiita documentation (#3032) * Added three tutorial sections to the Qiita documentation: 'Retrieving Public Data for Own Analysis' and 'Processing public data retrieved with redbiom' to the redbiom tab, and 'Statistical Analysis to Justify Clinical Trial Sample Size Tutorial' to the analyzing samples tab. * Update redbiom.rst * Update redbiom.rst * Update redbiom.rst * Further updates to redbiom.rst and the Stats tutorial. * update redbiom.rst * Finished proof-reading * Placed all three tutorials/sections together under Introduction to the download and analysis of public Qiita data * added a new introduction, with links to the three sections * Added figures to stats tutorial and contexts explanation * Added figures to stats tutorial and contexts explanation * Apply suggestions from code review [skip ci] Co-authored-by: Yoshiki Vázquez Baeza <yoshiki@ucsd.edu> Co-authored-by: Antonio Gonzalez <antgonza@gmail.com> Co-authored-by: Yoshiki Vázquez Baeza <yoshiki@ucsd.edu> * 092020 (#3033) * 092020 * connect artifact with job * rm INSERT qiita.artifact_processing_job * Apply suggestions from code review [skip ci] Co-authored-by: Yoshiki Vázquez Baeza <yoshiki@ucsd.edu> Co-authored-by: Yoshiki Vázquez Baeza <yoshiki@ucsd.edu> Co-authored-by: Daniel McDonald <danielmcdonald@ucsd.edu> Co-authored-by: Mirte Kuijpers <67341505+mcmk3@users.noreply.github.com> Co-authored-by: Yoshiki Vázquez Baeza <yoshiki@ucsd.edu> * fix #3036 Co-authored-by: Daniel McDonald <danielmcdonald@ucsd.edu> Co-authored-by: Mirte Kuijpers <67341505+mcmk3@users.noreply.github.com> Co-authored-by: Yoshiki Vázquez Baeza <yoshiki@ucsd.edu>

@ElDeveloper

…3040) * Version 092020 (#3034) * inject study_type in EBI and improvements to current automatic processing pipeline (#3023) * inject study_type in ebi and improvements to current automatic proecssing pipeline * addressing @ElDeveloper comments * some general fixes/additions for next release (#3026) * some general fixes/additions for next release * adding test for not None job.release_validator_job * fix #2839 * fix #2868 (#3028) * fix #2868 * 2nd round * fix errors * more changes * fix errors * fix ProcessingJobTest * fix PY_PATCH * add missing TRN.add * encapsulated_query -> perform_as_transaction * fix #3022 (#3030) * fix #3022 * adding tests * fix #2320 (#3031) * fix #2320 * adding prints to debug * children -> 1 * APIArtifactHandlerTest -> APIArtifactHandlerTests * configure_biom * qdb.util.activate_or_update_plugins * improving code * almost there * add values.template * fix filepaths * filepaths -> files * fixing errors * add prep.artifact insertion * addressing @ElDeveloper comments * fix artifact_definition active command * != -> == * Added three tutorial sections to the Qiita documentation (#3032) * Added three tutorial sections to the Qiita documentation: 'Retrieving Public Data for Own Analysis' and 'Processing public data retrieved with redbiom' to the redbiom tab, and 'Statistical Analysis to Justify Clinical Trial Sample Size Tutorial' to the analyzing samples tab. * Update redbiom.rst * Update redbiom.rst * Update redbiom.rst * Further updates to redbiom.rst and the Stats tutorial. * update redbiom.rst * Finished proof-reading * Placed all three tutorials/sections together under Introduction to the download and analysis of public Qiita data * added a new introduction, with links to the three sections * Added figures to stats tutorial and contexts explanation * Added figures to stats tutorial and contexts explanation * Apply suggestions from code review [skip ci] Co-authored-by: Yoshiki Vázquez Baeza <yoshiki@ucsd.edu> Co-authored-by: Antonio Gonzalez <antgonza@gmail.com> Co-authored-by: Yoshiki Vázquez Baeza <yoshiki@ucsd.edu> * 092020 (#3033) * 092020 * connect artifact with job * rm INSERT qiita.artifact_processing_job * Apply suggestions from code review [skip ci] Co-authored-by: Yoshiki Vázquez Baeza <yoshiki@ucsd.edu> Co-authored-by: Yoshiki Vázquez Baeza <yoshiki@ucsd.edu> Co-authored-by: Daniel McDonald <danielmcdonald@ucsd.edu> Co-authored-by: Mirte Kuijpers <67341505+mcmk3@users.noreply.github.com> Co-authored-by: Yoshiki Vázquez Baeza <yoshiki@ucsd.edu> * adding job submit ENVIRONMENT and minor improvement to runWorkflow * fix test Co-authored-by: Daniel McDonald <danielmcdonald@ucsd.edu> Co-authored-by: Mirte Kuijpers <67341505+mcmk3@users.noreply.github.com> Co-authored-by: Yoshiki Vázquez Baeza <yoshiki@ucsd.edu>

@ElDeveloper

* Version 092020 (#3034) * inject study_type in EBI and improvements to current automatic processing pipeline (#3023) * inject study_type in ebi and improvements to current automatic proecssing pipeline * addressing @ElDeveloper comments * some general fixes/additions for next release (#3026) * some general fixes/additions for next release * adding test for not None job.release_validator_job * fix #2839 * fix #2868 (#3028) * fix #2868 * 2nd round * fix errors * more changes * fix errors * fix ProcessingJobTest * fix PY_PATCH * add missing TRN.add * encapsulated_query -> perform_as_transaction * fix #3022 (#3030) * fix #3022 * adding tests * fix #2320 (#3031) * fix #2320 * adding prints to debug * children -> 1 * APIArtifactHandlerTest -> APIArtifactHandlerTests * configure_biom * qdb.util.activate_or_update_plugins * improving code * almost there * add values.template * fix filepaths * filepaths -> files * fixing errors * add prep.artifact insertion * addressing @ElDeveloper comments * fix artifact_definition active command * != -> == * Added three tutorial sections to the Qiita documentation (#3032) * Added three tutorial sections to the Qiita documentation: 'Retrieving Public Data for Own Analysis' and 'Processing public data retrieved with redbiom' to the redbiom tab, and 'Statistical Analysis to Justify Clinical Trial Sample Size Tutorial' to the analyzing samples tab. * Update redbiom.rst * Update redbiom.rst * Update redbiom.rst * Further updates to redbiom.rst and the Stats tutorial. * update redbiom.rst * Finished proof-reading * Placed all three tutorials/sections together under Introduction to the download and analysis of public Qiita data * added a new introduction, with links to the three sections * Added figures to stats tutorial and contexts explanation * Added figures to stats tutorial and contexts explanation * Apply suggestions from code review [skip ci] Co-authored-by: Yoshiki Vázquez Baeza <yoshiki@ucsd.edu> Co-authored-by: Antonio Gonzalez <antgonza@gmail.com> Co-authored-by: Yoshiki Vázquez Baeza <yoshiki@ucsd.edu> * 092020 (#3033) * 092020 * connect artifact with job * rm INSERT qiita.artifact_processing_job * Apply suggestions from code review [skip ci] Co-authored-by: Yoshiki Vázquez Baeza <yoshiki@ucsd.edu> Co-authored-by: Yoshiki Vázquez Baeza <yoshiki@ucsd.edu> Co-authored-by: Daniel McDonald <danielmcdonald@ucsd.edu> Co-authored-by: Mirte Kuijpers <67341505+mcmk3@users.noreply.github.com> Co-authored-by: Yoshiki Vázquez Baeza <yoshiki@ucsd.edu> * fix #2920 * fix test Co-authored-by: Daniel McDonald <danielmcdonald@ucsd.edu> Co-authored-by: Mirte Kuijpers <67341505+mcmk3@users.noreply.github.com> Co-authored-by: Yoshiki Vázquez Baeza <yoshiki@ucsd.edu>

@ElDeveloper

* Version 092020 (#3034) * inject study_type in EBI and improvements to current automatic processing pipeline (#3023) * inject study_type in ebi and improvements to current automatic proecssing pipeline * addressing @ElDeveloper comments * some general fixes/additions for next release (#3026) * some general fixes/additions for next release * adding test for not None job.release_validator_job * fix #2839 * fix #2868 (#3028) * fix #2868 * 2nd round * fix errors * more changes * fix errors * fix ProcessingJobTest * fix PY_PATCH * add missing TRN.add * encapsulated_query -> perform_as_transaction * fix #3022 (#3030) * fix #3022 * adding tests * fix #2320 (#3031) * fix #2320 * adding prints to debug * children -> 1 * APIArtifactHandlerTest -> APIArtifactHandlerTests * configure_biom * qdb.util.activate_or_update_plugins * improving code * almost there * add values.template * fix filepaths * filepaths -> files * fixing errors * add prep.artifact insertion * addressing @ElDeveloper comments * fix artifact_definition active command * != -> == * Added three tutorial sections to the Qiita documentation (#3032) * Added three tutorial sections to the Qiita documentation: 'Retrieving Public Data for Own Analysis' and 'Processing public data retrieved with redbiom' to the redbiom tab, and 'Statistical Analysis to Justify Clinical Trial Sample Size Tutorial' to the analyzing samples tab. * Update redbiom.rst * Update redbiom.rst * Update redbiom.rst * Further updates to redbiom.rst and the Stats tutorial. * update redbiom.rst * Finished proof-reading * Placed all three tutorials/sections together under Introduction to the download and analysis of public Qiita data * added a new introduction, with links to the three sections * Added figures to stats tutorial and contexts explanation * Added figures to stats tutorial and contexts explanation * Apply suggestions from code review [skip ci] Co-authored-by: Yoshiki Vázquez Baeza <yoshiki@ucsd.edu> Co-authored-by: Antonio Gonzalez <antgonza@gmail.com> Co-authored-by: Yoshiki Vázquez Baeza <yoshiki@ucsd.edu> * 092020 (#3033) * 092020 * connect artifact with job * rm INSERT qiita.artifact_processing_job * Apply suggestions from code review [skip ci] Co-authored-by: Yoshiki Vázquez Baeza <yoshiki@ucsd.edu> Co-authored-by: Yoshiki Vázquez Baeza <yoshiki@ucsd.edu> Co-authored-by: Daniel McDonald <danielmcdonald@ucsd.edu> Co-authored-by: Mirte Kuijpers <67341505+mcmk3@users.noreply.github.com> Co-authored-by: Yoshiki Vázquez Baeza <yoshiki@ucsd.edu> * rm create_qiime_mapping_file * fixing some tests * fixing more tests * fix even more tests * rm npt.assert_warns * qiime-map -> sample-file * update not_merged_samples.txt * adding @ElDeveloper changes Co-authored-by: Daniel McDonald <danielmcdonald@ucsd.edu> Co-authored-by: Mirte Kuijpers <67341505+mcmk3@users.noreply.github.com> Co-authored-by: Yoshiki Vázquez Baeza <yoshiki@ucsd.edu>

@ElDeveloper

* Version 092020 (#3034) * inject study_type in EBI and improvements to current automatic processing pipeline (#3023) * inject study_type in ebi and improvements to current automatic proecssing pipeline * addressing @ElDeveloper comments * some general fixes/additions for next release (#3026) * some general fixes/additions for next release * adding test for not None job.release_validator_job * fix #2839 * fix #2868 (#3028) * fix #2868 * 2nd round * fix errors * more changes * fix errors * fix ProcessingJobTest * fix PY_PATCH * add missing TRN.add * encapsulated_query -> perform_as_transaction * fix #3022 (#3030) * fix #3022 * adding tests * fix #2320 (#3031) * fix #2320 * adding prints to debug * children -> 1 * APIArtifactHandlerTest -> APIArtifactHandlerTests * configure_biom * qdb.util.activate_or_update_plugins * improving code * almost there * add values.template * fix filepaths * filepaths -> files * fixing errors * add prep.artifact insertion * addressing @ElDeveloper comments * fix artifact_definition active command * != -> == * Added three tutorial sections to the Qiita documentation (#3032) * Added three tutorial sections to the Qiita documentation: 'Retrieving Public Data for Own Analysis' and 'Processing public data retrieved with redbiom' to the redbiom tab, and 'Statistical Analysis to Justify Clinical Trial Sample Size Tutorial' to the analyzing samples tab. * Update redbiom.rst * Update redbiom.rst * Update redbiom.rst * Further updates to redbiom.rst and the Stats tutorial. * update redbiom.rst * Finished proof-reading * Placed all three tutorials/sections together under Introduction to the download and analysis of public Qiita data * added a new introduction, with links to the three sections * Added figures to stats tutorial and contexts explanation * Added figures to stats tutorial and contexts explanation * Apply suggestions from code review [skip ci] Co-authored-by: Yoshiki Vázquez Baeza <yoshiki@ucsd.edu> Co-authored-by: Antonio Gonzalez <antgonza@gmail.com> Co-authored-by: Yoshiki Vázquez Baeza <yoshiki@ucsd.edu> * 092020 (#3033) * 092020 * connect artifact with job * rm INSERT qiita.artifact_processing_job * Apply suggestions from code review [skip ci] Co-authored-by: Yoshiki Vázquez Baeza <yoshiki@ucsd.edu> Co-authored-by: Yoshiki Vázquez Baeza <yoshiki@ucsd.edu> Co-authored-by: Daniel McDonald <danielmcdonald@ucsd.edu> Co-authored-by: Mirte Kuijpers <67341505+mcmk3@users.noreply.github.com> Co-authored-by: Yoshiki Vázquez Baeza <yoshiki@ucsd.edu> * adding qiita.study autoloaded column * cleaning tests * fix redbiom test * avoiding clog submissions * Apply suggestions from code review Co-authored-by: Yoshiki Vázquez Baeza <yoshiki@ucsd.edu> Co-authored-by: Daniel McDonald <danielmcdonald@ucsd.edu> Co-authored-by: Mirte Kuijpers <67341505+mcmk3@users.noreply.github.com> Co-authored-by: Yoshiki Vázquez Baeza <yoshiki@ucsd.edu>

inject study_type in ebi and improvements to current automatic proecs…

0b5e61e

…sing pipeline

antgonza requested a review from ElDeveloper August 3, 2020 21:39

ElDeveloper suggested changes Aug 4, 2020

View reviewed changes

antgonza commented Aug 4, 2020

View reviewed changes

addressing @ElDeveloper comments

bf32a46

ElDeveloper merged commit 68ffc5a into qiita-spots:dev Aug 4, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

inject study_type in EBI and improvements to current automatic processing pipeline #3023

inject study_type in EBI and improvements to current automatic processing pipeline #3023

Uh oh!

antgonza commented Aug 3, 2020

Uh oh!

codecov-commenter commented Aug 3, 2020 •

edited

Loading

Uh oh!

ElDeveloper left a comment

Uh oh!

ElDeveloper Aug 4, 2020

Uh oh!

Uh oh!

ElDeveloper Aug 4, 2020

Uh oh!

antgonza Aug 4, 2020

Uh oh!

ElDeveloper Aug 4, 2020

Uh oh!

antgonza left a comment

Uh oh!

Uh oh!

antgonza Aug 4, 2020

Uh oh!

ElDeveloper commented Aug 4, 2020

Uh oh!

Uh oh!

	# study type is depricated and not displayed anywhere on EBI-ENA;
	# study type is deprecated and not displayed anywhere on EBI-ENA;

inject study_type in EBI and improvements to current automatic processing pipeline #3023

inject study_type in EBI and improvements to current automatic processing pipeline #3023

Uh oh!

Conversation

antgonza commented Aug 3, 2020

Uh oh!

codecov-commenter commented Aug 3, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

ElDeveloper left a comment

Choose a reason for hiding this comment

Uh oh!

ElDeveloper Aug 4, 2020

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ElDeveloper Aug 4, 2020

Choose a reason for hiding this comment

Uh oh!

antgonza Aug 4, 2020

Choose a reason for hiding this comment

Uh oh!

ElDeveloper Aug 4, 2020

Choose a reason for hiding this comment

Uh oh!

antgonza left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

antgonza Aug 4, 2020

Choose a reason for hiding this comment

Uh oh!

ElDeveloper commented Aug 4, 2020

Uh oh!

Uh oh!

codecov-commenter commented Aug 3, 2020 •

edited

Loading