Skip to content

SEA: re-fetch links in case of expiry #635

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
wants to merge 380 commits into
base: less-defensive-download
Choose a base branch
from
Draft

SEA: re-fetch links in case of expiry #635

wants to merge 380 commits into from

Conversation

varun-edachali-dbx
Copy link
Collaborator

WIP

What type of PR is this?

  • Refactor
  • Feature
  • Bug Fix
  • Other

Description

How is this tested?

  • Unit tests
  • E2E Tests
  • Manually
  • N/A

Related Tickets & Documents

@varun-edachali-dbx
remove reliance on schema_bytes in SEA
015fb76 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
remove redundant note on arrow_schema_bytes
0385ffb 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
use more fetch methods
5380c7a 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
remove redundant schema_bytes from parent constructor
27b781f 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
only call get_chunk_link with non null chunk index
238dc0a 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
align SeaResultSet structure with ThriftResultSet
b3bb07e 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
remvoe _fill_result_buffer from SeaResultSet
13e6346 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
reduce code repetition
f90b4d4 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
align SeaResultSet with ext-links-sea
23963fc 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
remove redundant methods
dd43715 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
update unit tests
34a7f66 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
remove accidental venv changes
715cc13 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
pre-fetch next chunk link on processing current
fb53dd9 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
reduce nesting
d893877 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
line break after multi line pydoc
a165f1c 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
re-introduce schema_bytes for better abstraction (likely temporary)
d68e4ea 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
Merge branch 'fetch-json-inline' into ext-links-sea
be17812
@varun-edachali-dbx
add fetchmany_arrow and fetchall_arrow
a0705bc 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
Merge branch 'metadata-sea' into fetch-json-inline
1b90c4a
@varun-edachali-dbx
remove accidental changes in sea backend tests
f7c11b9 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
Merge branch 'exec-phase-sea' into metadata-sea
349c021
@varun-edachali-dbx
remove irrelevant changes
6229848 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
remove un-necessary test changes
fd52356 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
remove un-necessary changes in thrift backend tests
64e58b0 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
remove unimplemented methods test
2903473 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
Merge branch 'metadata-sea' into fetch-json-inline
b300709
@varun-edachali-dbx
remove unimplemented method tests
021ff4c 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
modify example scripts to include fetch calls
adecd53 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
Merge branch 'fetch-json-inline' into ext-links-sea
d33e5fd
@varun-edachali-dbx
add GetChunksResponse
e3cef5c 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
account for empty response in LinkFetcher init
23dba80 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
make get_chunk_link return mandatory ExternalLink
26aa45c 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
set shutdown_event instead of breaking on completion so get_chunk_lin…
9fd57d8 
…k is informed

Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
docstrings, logging, pydoc
dd009df 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
use total_chunk_cound > 0
7c360d1 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
clarify that link has already been submitted on getting row_offset
6b1b972 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
return None for out of range
2ca7c2b 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
default link_fetcher to None
00db613 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
Merge branch 'sea-decouple-link-fetch' into sea-link-expiry
47bb758
@varun-edachali-dbx
reduce repeated init
2cd802e 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx varun-edachali-dbx changed the base branch from sea-decouple-link-fetch to sea-migration July 21, 2025 03:05
@varun-edachali-dbx
add_link -> add_links for less heavy link additions
b114599 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@databricks databricks deleted a comment from github-actions bot Jul 21, 2025
@databricks databricks deleted a comment from github-actions bot Jul 21, 2025
@databricks databricks deleted a comment from github-actions bot Jul 21, 2025
@varun-edachali-dbx
Merge branch 'less-defensive-download' into sea-link-expiry
38c9343
@varun-edachali-dbx
move param order
92821e7 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx varun-edachali-dbx changed the base branch from sea-migration to less-defensive-download July 21, 2025 07:11
@varun-edachali-dbx
type issues
63deb04 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@databricks databricks deleted a comment from github-actions bot Jul 21, 2025
@databricks databricks deleted a comment from github-actions bot Jul 21, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@varun-edachali-dbx