Skip to content

Enhance SEA HTTP Client #618

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 447 commits into
base: main
Choose a base branch
from
Open

Enhance SEA HTTP Client #618

wants to merge 447 commits into from

Conversation

varun-edachali-dbx
Copy link
Collaborator

@varun-edachali-dbx varun-edachali-dbx commented Jun 27, 2025

What type of PR is this?

  • Feature

Description

Enhance the current, simplified SEA Http Client by introducing retries. Most of the auth related functionality is pulled from the Thrift backend. All of the retry-related tests are expected to pass since they represent user-facing functionality (in terms of exceptions to be thrown in particular cases).

We continue using a low-level connection pool instead of using the higher level requests module because the latter does not have first-class support for passing a password to a client certificate, which we want to support. There are workarounds involving subclassing the HttpAdapter and invoking init_poolmanager, but the docstring of the method states that it should not be invoked from client code, so we do not consider it a long term solution.

The exceptions thrown by the SEA and Thrift backends are not completely the same (yet), because this PR passes None to the context and session id of the RequestError object. This misparity will be logged and addressed later.

How is this tested?

  • Unit tests
  • E2E Tests
  • Manually
  • N/A

Related Tickets & Documents

Design Doc

varun-edachali-dbx and others added 30 commits June 18, 2025 04:15
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
* [squash from exec-sea] bring over execution phase changes

Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>

* remove excess test

Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>

* add docstring

Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>

* remvoe exec func in sea backend

Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>

* remove excess files

Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>

* remove excess models

Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>

* remove excess sea backend tests

Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>

* cleanup

Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>

* re-introduce get_schema_desc

Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>

* remove SeaResultSet

Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>

* clean imports and attributes

Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>

* pass CommandId to ExecResp

Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>

* remove changes in types

Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>

* add back essential types (ExecResponse, from_sea_state)

Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>

* fix fetch types

Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>

* excess imports

Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>

* reduce diff by maintaining logs

Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>

* fix int test types

Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>

* [squashed from exec-sea] init execution func

Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>

* remove irrelevant changes

Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>

* remove ResultSetFilter functionality

Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>

* remove more irrelevant changes

Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>

* remove more irrelevant changes

Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>

* even more irrelevant changes

Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>

* remove sea response as init option

Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>

* exec test example scripts

Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>

* formatting (black)

Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>

* [squashed from sea-exec] merge sea stuffs

Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>

* remove excess changes

Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>

* remove excess removed docstring

Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>

* remove excess changes in backend

Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>

* remove excess imports

Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>

* remove accidentally removed _get_schema_desc

Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>

* remove unnecessary init with sea_response tests

Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>

* rmeove unnecessary changes

Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>

* formatting (black)

Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>

* improved models and filters from cloudfetch-sea branch

Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>

* filters stuff (align with JDBC)

Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>

* backend from cloudfetch-sea

Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>

* remove filtering, metadata ops

Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>

* raise NotImplementedErrror for metadata ops

Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>

* change to valid table name

Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>

* remove un-necessary changes

covered by #588

Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>

* simplify test module

Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>

* logging -> debug level

Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>

* change table name in log

Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>

* remove un-necessary changes

Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>

* remove un-necessary backend cahnges

Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>

* remove un-needed GetChunksResponse

Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>

* remove un-needed GetChunksResponse

only relevant in Fetch phase

Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>

* reduce code duplication in response parsing

Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>

* reduce code duplication

Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>

* more clear docstrings

Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>

* introduce strongly typed ChunkInfo

Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>

* remove is_volume_operation from response

Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>

* add is_volume_op and more ResultData fields

Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>

* add test scripts

Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>

* Revert "Merge branch 'exec-models-sea' into exec-phase-sea"

This reverts commit be1997e, reversing
changes made to 37813ba.

* change logging level

Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>

* remove un-necessary changes

Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>

* remove excess changes

Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>

* remove excess changes

Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>

* remove _get_schema_bytes (for now)

Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>

* redundant comments

Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>

* remove fetch phase methods

Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>

* reduce code repetititon + introduce gaps after multi line pydocs

Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>

* remove unused imports

Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>

* move description extraction to helper func

Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>

* formatting (black)

Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>

* add more unit tests

Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>

* streamline unit tests

Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>

* test getting the list of allowed configurations

Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>

* reduce diff

Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>

* reduce diff

Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>

* house constants in enums for readability and immutability

Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>

* add note on hybrid disposition

Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>

* remove redundant note on arrow_schema_bytes

Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>

* remove invalid import

Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>

* add strong typing for manifest in _extract_description

Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>

* remove un-necessary column skipping

Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>

* remove parsing in backend

Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>

* fix: convert sea statement id to CommandId type

Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>

* make polling interval a separate constant

Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>

* align state checking with Thrift implementation

Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>

* update unit tests according to changes

Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>

* add unit tests for added methods

Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>

* add spec to description extraction docstring, add strong typing to params

Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>

* add strong typing for backend parameters arg

Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>

---------

Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
…nd forward refs, remove some unused imports

Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
Co-authored-by: jayant <167047871+jayantsing-db@users.noreply.github.com>
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
…ions

Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
This reverts commit dabba55, reversing
changes made to dd7dc6a.

Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
This reverts commit 3a999c0, reversing
changes made to a1f9b9c.
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@varun-edachali-dbx varun-edachali-dbx changed the base branch from sea-migration to main July 28, 2025 06:04
@databricks databricks deleted a comment from github-actions bot Jul 28, 2025
@databricks databricks deleted a comment from github-actions bot Jul 28, 2025
@databricks databricks deleted a comment from github-actions bot Jul 28, 2025
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
Signed-off-by: varun-edachali-dbx <varun.edachali@databricks.com>
@databricks databricks deleted a comment from github-actions bot Jul 28, 2025
@databricks databricks deleted a comment from github-actions bot Jul 28, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants