fix(csharp/src/Drivers/BigQuery): fix support for large results #1507
Conversation
davidhcoe
commented
Feb 1, 2024
davidhcoe
commented
- Adds value for a DestinationTable, which is needed for large results
- Removes old Arrow references in test projects
There was a problem hiding this comment.
It seems unfortunate for the caller to have to handle this. How does it work via the ODBC or JDBC drivers? Do the callers similarly have to know that they're fetching large data? Does the driver catch that error and then automatically create a temporary table? (And if so, how does the cleanup happen?)
I don't think it's necessarily a problem to commit the current logic and enable a new scenario, but this doesn't seem like the right longer-term approach.
| @@ -1014,6 +1032,10 @@ public override AdbcStatement CreateStatement() | |||
| { | |||
| options[keyValuePair.Key] = keyValuePair.Value; | |||
| } | |||
| if(keyValuePair.Key == BigQueryParameters.LargeResultsDestinationTable) | |||
There was a problem hiding this comment.
nit: space before paren to match the other tests
Unrelated to this specific change, does the ADBC spec say anything about case-sensitivity of options? For ODBC/JDBC/ADO.NET, connection string keys are usually case-insensitive.
There was a problem hiding this comment.
~it's up to the driver, but mostly we've been case sensitive so far
There was a problem hiding this comment.
~it's up to the driver, but mostly we've been case sensitive so far
This seems like something we'd want to provide guidance around to improve standardization e.g. "case-sensitive but consistently lower-case keys"
| string datasetId = string.Empty; | ||
| string tableId = string.Empty; | ||
|
|
||
| List<string> segments = destinationTable.Split('.').Where(x => x.Length > 0).ToList(); |
There was a problem hiding this comment.
I don't think "a....b....c" should be allowed.
In principle, should this follow BigQuery identifier rules for e.g. escaping? Is there a function in the Google libraries to parse table names.
There was a problem hiding this comment.
I don't see a method to parse the table names.
BigQuery separates the full table names by periods as well, ie,
SELECT FROM pq-test-398321.AdbcTests.adbc_all_types LIMIT 1000
There was a problem hiding this comment.
Sorry, I didn't do a great job explaining this feedback. What I meant was that this test would allow me to pass a..b..c (that is a dot dot be dot dot c) and have it be treated the same way as a.b.c (a dot b dot c) and that seems wrong.
| @@ -185,6 +186,33 @@ static IArrowReader ReadChunk(BigQueryReadClient readClient, string streamName) | |||
| { | |||
| options.AllowLargeResults = true ? keyValuePair.Value.ToLower().Equals("true") : false; | |||
| } | |||
| if(keyValuePair.Key == BigQueryParameters.LargeResultsDestinationTable) | |||
There was a problem hiding this comment.
nit: space before paren
