Skip to content

SEA: Decouple Link Fetching #632

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 360 commits into
base: sea-migration
Choose a base branch
from
Open

SEA: Decouple Link Fetching #632

wants to merge 360 commits into from
+299 −75

Conversation

varun-edachali-dbx
Copy link
Collaborator

@varun-edachali-dbx varun-edachali-dbx commented Jul 9, 2025 •
edited
Loading

What type of PR is this?

  • Feature

Description

For efficiency, we fetch the links for data in the SeaCloudFetchQueue in a separate thread. Link fetches are slow because they come from the bottlenecked control plane, so coupling the fetching logic with the main process slows down our fetch phase. Instead, we proactively fetch links in the background and use them when required.

Note: this PR does not account for link expiry. This will be handled separately in #635.

How is this tested?

  • Unit tests
  • E2E Tests
  • Manually
  • N/A

Related Tickets & Documents

N/A

@varun-edachali-dbx
Merge branch 'sea-migration' into exec-phase-sea
699942d
@varun-edachali-dbx
remove fetch phase methods
af8f74e 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
reduce code repetititon + introduce gaps after multi line pydocs
5540c5c 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
remove unused imports
efe3881 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
move description extraction to helper func
36ab59b 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
formatting (black)
1d57c99 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
add more unit tests
df6dac2 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
streamline unit tests
ad0e527 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
test getting the list of allowed configurations
ed446a0 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
reduce diff
38e4b5c 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
reduce diff
94879c0 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
house constants in enums for readability and immutability
1809956 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
add note on hybrid disposition
da5260c 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
[squashed from cloudfetch-sea] introduce external links + arrow funct…
6ec265f 
…ionality

Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
reduce responsibility of Queue
b2ad5e6 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
reduce repetition in arrow tablee creation
66d0df6 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
reduce redundant code in CloudFetchQueue
eb7ec80 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
move chunk link progression to separate func
a3a8a4a 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
remove redundant log
ea79bc8 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
improve logging
5b49405 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
remove reliance on schema_bytes in SEA
015fb76 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
remove redundant note on arrow_schema_bytes
0385ffb 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
use more fetch methods
5380c7a 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
remove redundant schema_bytes from parent constructor
27b781f 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
only call get_chunk_link with non null chunk index
238dc0a 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
align SeaResultSet structure with ThriftResultSet
b3bb07e 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
remvoe _fill_result_buffer from SeaResultSet
13e6346 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
reduce code repetition
f90b4d4 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
align SeaResultSet with ext-links-sea
23963fc 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
remove redundant methods
dd43715 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@databricks databricks deleted a comment from github-actions bot Jul 16, 2025
@databricks databricks deleted a comment from github-actions bot Jul 16, 2025
@databricks databricks deleted a comment from github-actions bot Jul 16, 2025
@databricks databricks deleted a comment from github-actions bot Jul 16, 2025
@varun-edachali-dbx varun-edachali-dbx changed the base branch from sea-hybrid to sea-migration July 17, 2025 04:29
@varun-edachali-dbx
add helper for link addition, check for edge case to prevent inf wait
76fd190 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
add unit tests for LinkFetcher
ab2c2c4 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@databricks databricks deleted a comment from github-actions bot Jul 17, 2025
@databricks databricks deleted a comment from github-actions bot Jul 17, 2025
@varun-edachali-dbx
remove un-necessary download manager check
3af2333 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx
remove un-necessary string literals around param type
480528e 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@databricks databricks deleted a comment from github-actions bot Jul 17, 2025
@databricks databricks deleted a comment from github-actions bot Jul 17, 2025
@databricks databricks deleted a comment from github-actions bot Jul 17, 2025
@varun-edachali-dbx
remove duplicate download_manager init
fba1812 
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@github-actions GitHub Actions
Copy link

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).

@varun-edachali-dbx varun-edachali-dbx marked this pull request as ready for review July 17, 2025 05:29
@github-actions GitHub Actions
Copy link

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jayantsing-db jayantsing-db Awaiting requested review from jayantsing-db

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@varun-edachali-dbx