Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use a single JDBC layer per read/write #323

Merged
merged 6 commits into from
Feb 10, 2022
Merged

Use a single JDBC layer per read/write #323

merged 6 commits into from
Feb 10, 2022

Conversation

jeremyprime
Copy link
Collaborator

Summary

Use a single JDBC layer, which holds the actual connection, for read or write operations.

Description

We create a new JDBC layer, and thus a new connection, each time a read or write pipe is called for. This results in additional connections at runtime, some of which are not closed when the operation completes. This fix recreates the JDBC layer each time it is called, if the underlying connection was closed.

Further work is still needed to avoid opening and closing multiple connections, which will be performed under another ticket.

Related Issue

Closes #314.

Additional Reviewers

@alexey-temnikov
@alexr-bq
@Aryex
@jonathanl-bq

Comment on lines 40 to 42
// Maintain a single copy of the read and write JDBC layers
private var readLayer: Option[VerticaJdbcLayer] = None;
private var writeLayer: Option[VerticaJdbcLayer] = None;
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was unable to create a singleton of the connection or JDBC layer due to our use of a lazy connection and the fact the JDBC layer may contain a closed connection (cannot be reopened, must create a new one).

These vars hold a single JDBC layer per operation (read and write), and the logic below will recreate them (and their underlying connection) if the connection was closed.

In the future we should properly refactor our use of the JDBC layer and connection to avoid closing and opening connections unnecessarily (will create a separate ticket for that).

@Aryex Aryex merged commit ca32d6f into main Feb 10, 2022
@Aryex Aryex deleted the singleton-connection branch February 10, 2022 20:04
@jeremyprime jeremyprime restored the singleton-connection branch February 10, 2022 20:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[BUG] Spark connector leaves multiple sessions open
3 participants