feat(go/adbc/driver/snowflake): use vectorized scanner for bulk ingest #2025
Conversation
| @@ -45,7 +45,7 @@ import ( | |||
|
|
|||
| const ( | |||
| bindStageName = "ADBC$BIND" | |||
| createTemporaryStageStmt = "CREATE OR REPLACE TEMPORARY STAGE " + bindStageName + " FILE_FORMAT = (TYPE = PARQUET USE_LOGICAL_TYPE = TRUE BINARY_AS_TEXT = FALSE)" | |||
| createTemporaryStageStmt = "CREATE OR REPLACE TEMPORARY STAGE " + bindStageName + " FILE_FORMAT = (TYPE = PARQUET USE_LOGICAL_TYPE = TRUE BINARY_AS_TEXT = FALSE USE_VECTORIZED_SCANNER=TRUE REPLACE_INVALID_CHARACTERS = TRUE)" | |||
There was a problem hiding this comment.
Is it necessary to set REPLACE_INVALID_CHARACTERS? I guess Snowflake recommends it "as a general rule of thumb" but I wonder why...
There was a problem hiding this comment.
I don't know, maybe just to reduce failures caused by the characters? Not sure myself.
There was a problem hiding this comment.
It's a best practice as invalid UTF8 can cause query failures or unexpected results:
https://docs.snowflake.com/en/release-notes/bcr-bundles/un-bundled/bcr-1013-1014
It depends on the configuration of the warehouse, some configurations (encryption, security settings, etc.) are not yet supported, while others are. So if your configuration is supported, you'll get benefits immediately while other configurations will need to wait until snowflake makes some more changes. |
Closes #2005