VS-280 Create a VAT intermediary #7657
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
RoriCremer
force-pushed
the
rc-vat-intermediary
branch
2 times, most recently
from
91a0ecc to
7050d09
Compare
RoriCremer
force-pushed
the
rc-vat-intermediary
branch
from
fcf0379 to
2b9b88a
Compare
RoriCremer
force-pushed
the
rc-vat-intermediary
branch
from
53e35ac to
cdd13d3
Compare
rsasch
suggested changes
Feb 14, 2022
RoriCremer
force-pushed
the
rc-vat-intermediary
branch
from
031fe5f to
841d7bd
Compare
rsasch
reviewed
Feb 16, 2022
rsasch
suggested changes
Feb 16, 2022
RoriCremer
force-pushed
the
rc-vat-intermediary
branch
2 times, most recently
from
2282d75 to
ad6455f
Compare
rsasch
suggested changes
Feb 25, 2022
RoriCremer
force-pushed
the
rc-vat-intermediary
branch
from
ad6455f to
52735f5
Compare
RoriCremer
force-pushed
the
rc-vat-intermediary
branch
from
52735f5 to
006c61a
Compare
rsasch
reviewed
Feb 25, 2022
Comment on lines
39
to
40
| | workspace_namespace | name of the current workspace namespace | ## is this still needed? | ||
| | workspace_name | name of the current workspace | ## is this still needed? |
There was a problem hiding this comment.
I don't see either of these inputs in GvsAssignIds, so feel free to remove them.
rsasch
suggested changes
Feb 25, 2022
| The first two of these inputs are two files — one of the file/vcf/shards you want to use for the VAT, and their corresponding index files. These are labelled as `inputFileofFileNames` and `inputFileofIndexFileNames` and need to be copied into a GCP bucket that this pipeline will have access to (eg. this bucket: `gs://aou-genomics-curation-prod-processing/vat/`) for easy access during the workflow. | ||
| The third input is the ancestry file from the ancestry pipeline which will be used to calculate AC, AN and AF for all subpopulations. It needs to be copied into a GCP bucket that this pipeline will have access to. This input has been labelled as the `ancestry_file`. | ||
|
|
||
| Most of the other files are specific to where the VAT will live, like the project_id and dataset_name and the table_suffix which will name the VAT itself as vat_`table_suffix` as well as a GCP bucket location, the output_path, for the intermediary files and the VAT export in tsv form. |
There was a problem hiding this comment.
I thought the export for the VAT was CSV not TSV?
|
|
||
| These numbers are cumulative. Also the names of these json files are retained from the original shard names so as to not cause collisions. If you run the same shards through the VAT twice, the second runs should overwrite the first and the total number of jsons should not change. | ||
| Once the shards have make it into the /genes/ and /vt/ directories, the majority of the expense and transformations needed for that shard are complete. | ||
| They are ready to be loaded into BQ. You will notice that past this step, all there is to do is create the BQ tables, load the BQ tables, run a join query and then the remaining steps are all validations or an export into tsv. |
rsasch
approved these changes
Feb 25, 2022
This was referenced Mar 17, 2023
Closed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add an extensive and instructional ReadMe
Move the expensive step and the saving data step into a subworkflow to that they can complete their mission together in harmony even when a fellow shard has failed.