Added heartbeat to cloud download #173
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
While downloading cloud files a heartbeat will query operation status at regular intervals to keep the connection alive. Implemented heartbeat type which takes an instance of driver.Pinger and calls it at a regular interval. Implemented statusGetter type which implements driver.Pinger by calling GetOperationStatus() Updated NewBatchLoader() to take an argument of type driver.Pinger and updated batchLoader to instantiate a heartbeat instance using the pinger Updated NewRows() to create an instance of statusGetter and updated the various constructor functions to take a diver.Pinger argument. Added interface fetcher.Overwatch. NewConcurrentFetcher() now takes an Overwatch argument. An instance of Overwatch will be started/stopped in conjunction with fetcher start/stop. Signed-off-by: Raymond Cypher <raymond.cypher@databricks.com>
rcypher-databricks
requested review from
susodapop,
yunbodeng-db,
andrefurlan-db and
jadewang-db
as code owners
During cloud fetch, the client downloads cloud files without the involvement of DBSQL or cluster so the server may shut down due to inactivity. Heartbeat thread would keep it alive. |
|
But as I said before. The operation/session are open until the client reads the entire result set or manually closes. So the server shouldn't be shutting down due to inactivity. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
While downloading cloud files a heartbeat will query operation status at regular intervals to keep the connection alive.
Implemented heartbeat type which takes an instance of driver.Pinger and calls it at a regular interval.
Implemented statusGetter type which implements driver.Pinger by calling GetOperationStatus()
Updated NewBatchLoader() to take an argument of type driver.Pinger and updated batchLoader to instantiate a heartbeat instance using the pinger.
Updated NewRows() to create an instance of statusGetter and updated the various constructor functions to take a diver.Pinger argument.
Added interface fetcher.Overwatch. NewConcurrentFetcher() now takes an Overwatch argument. An instance of Overwatch will be started/stopped in conjunction with fetcher start/stop.