Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Clean-up and generalize TDR source specs #6426

Closed
hannes-ucsc opened this issue Jul 19, 2024 · 1 comment
Closed

Clean-up and generalize TDR source specs #6426

hannes-ucsc opened this issue Jul 19, 2024 · 1 comment
Assignees
@nadove-ucsc
Labels
- [priority] Medium code [subject] Production code debt [type] A defect incurring continued engineering cost demo [process] To be demonstrated at the end of the sprint demoed [process] Successfully demonstrated to team enh [type] New feature or request orange [process] Done by the Azul team

Comments

@hannes-ucsc
Copy link
Member

Current spec format:

tdr:datarepo-dev-6fdac3db:snapshot/hca_dev_e8808cc84ca0409680f2bba73600cba6__20210902_20210907:/1

  1. Rename TDRSourceSpec.project to TDRSourceSpec.subdomain.

  2. Remove is_snapshot field (dataset/ or snapshot/) and remove support for /dataset:

    tdr:datarepo-dev-6fdac3db:hca_dev_e8808cc84ca0409680f2bba73600cba6__20210902_20210907:/1

  3. Add type field (enum of bigquery or parquet, but reject parquet for now):

    tdr:bigquery:datarepo-dev-6fdac3db:hca_dev_e8808cc84ca0409680f2bba73600cba6__20210902_20210907:/1

  4. Add domain field (enum of gcp or azure, but reject azure for now) and reject the combination of type bigquery and provider azure:

    tdr:bigquery:gcp:datarepo-dev-6fdac3db:hca_dev_e8808cc84ca0409680f2bba73600cba6__20210902_20210907:/1

The order of the attributes of TDRSourceSpec should be: prefix (inherited), type, domain, subdomain and name.
@hannes-ucsc hannes-ucsc added orange [process] Done by the Azul team enh [type] New feature or request debt [type] A defect incurring continued engineering cost code [subject] Production code - [priority] Medium labels Jul 19, 2024
@hannes-ucsc hannes-ucsc mentioned this issue Jul 19, 2024
Open
@nadove-ucsc nadove-ucsc mentioned this issue Jul 24, 2024
Merged
nadove-ucsc added a commit that referenced this issue Jul 25, 2024
@nadove-ucsc
Rename TDRSourceSpec.project to subdomain (#6426)
f516a0c
nadove-ucsc added a commit that referenced this issue Jul 25, 2024
@nadove-ucsc
[u] Remove support for indexing TDR datasets (#6426)
072125f
nadove-ucsc added a commit that referenced this issue Jul 25, 2024
@nadove-ucsc
[u] Add type and domain fields to TDRSourceSpec (#6426)
262353f
nadove-ucsc added a commit that referenced this issue Jul 27, 2024
@nadove-ucsc
Rename TDRSourceSpec.project to subdomain (#6426)
4dfb3fd
nadove-ucsc added a commit that referenced this issue Jul 27, 2024
@nadove-ucsc
[u r] Remove support for indexing TDR datasets (#6426)
e358cdf
nadove-ucsc added a commit that referenced this issue Jul 27, 2024
@nadove-ucsc
[u r] Add type and domain fields to TDRSourceSpec (#6426)
4589e62
nadove-ucsc added a commit that referenced this issue Jul 31, 2024
@nadove-ucsc
Rename TDRSourceSpec.project to subdomain (#6426)
05e524a
nadove-ucsc added a commit that referenced this issue Jul 31, 2024
@nadove-ucsc
[u r] Remove support for indexing TDR datasets (#6426)
0fde525
nadove-ucsc added a commit that referenced this issue Jul 31, 2024
@nadove-ucsc
[u r] Add type and domain fields to TDRSourceSpec (#6426)
b85e4c3
nadove-ucsc added a commit that referenced this issue Aug 1, 2024
@nadove-ucsc
Rename TDRSourceSpec.project to subdomain (#6426)
94525aa
nadove-ucsc added a commit that referenced this issue Aug 1, 2024
@nadove-ucsc
[u r] Remove support for indexing TDR datasets (#6426)
3f16fdf
nadove-ucsc added a commit that referenced this issue Aug 1, 2024
@nadove-ucsc
fixup! [u r] Remove support for indexing TDR datasets (#6426)
1a0a33a
nadove-ucsc added a commit that referenced this issue Aug 1, 2024
@nadove-ucsc
[u r] Add type and domain fields to TDRSourceSpec (#6426)
7fd5e03
nadove-ucsc added a commit that referenced this issue Aug 1, 2024
@nadove-ucsc
fixup! [u r] Add type and domain fields to TDRSourceSpec (#6426)
c2885a5
nadove-ucsc added a commit that referenced this issue Aug 1, 2024
@nadove-ucsc
fixup! [u r] Add type and domain fields to TDRSourceSpec (#6426)
192d97a
nadove-ucsc added a commit that referenced this issue Aug 1, 2024
@nadove-ucsc
[u r] Remove support for indexing TDR datasets (#6426)
b2705b5
nadove-ucsc added a commit that referenced this issue Aug 1, 2024
@nadove-ucsc
[u r] Add type and domain fields to TDRSourceSpec (#6426)
b5617d8
nadove-ucsc added a commit that referenced this issue Aug 1, 2024
@nadove-ucsc
fixup! [u r] Add type and domain fields to TDRSourceSpec (#6426)
a5f7692
@nadove-ucsc nadove-ucsc added the demo [process] To be demonstrated at the end of the sprint label Aug 3, 2024
@nadove-ucsc
Copy link
Contributor

For demo, show new source specs in the service response. Show that the update_subgraph_counts.py script still works. Try filtering an index endpoint using a source spec.

achave11-ucsc pushed a commit that referenced this issue Aug 12, 2024
@nadove-ucsc @achave11-ucsc
Rename TDRSourceSpec.project to subdomain (#6426)
f4de91b
achave11-ucsc pushed a commit that referenced this issue Aug 12, 2024
@nadove-ucsc @achave11-ucsc
[u r] Remove support for indexing TDR datasets (#6426)
7b5a849
achave11-ucsc pushed a commit that referenced this issue Aug 12, 2024
@nadove-ucsc @achave11-ucsc
[u r] Add type and domain fields to TDRSourceSpec (#6426)
be2dfe3
achave11-ucsc added a commit that referenced this issue Aug 13, 2024
@achave11-ucsc
[u r] Clean-up and generalize TDR source specs (#6426, PR #6445)
c3ef634
@nadove-ucsc nadove-ucsc added the demoed [process] Successfully demonstrated to team label Aug 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@nadove-ucsc nadove-ucsc

Labels
- [priority] Medium code [subject] Production code debt [type] A defect incurring continued engineering cost demo [process] To be demonstrated at the end of the sprint demoed [process] Successfully demonstrated to team enh [type] New feature or request orange [process] Done by the Azul team
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@hannes-ucsc @nadove-ucsc