go/adbc/driver/snowflake: add use_vectorized_scanner
flag to bulk ingest
#2005
Labels
Type: enhancement
New feature or request
Milestone
use_vectorized_scanner
flag to bulk ingest
#2005
What feature or improvement would you like to see?
https://github.com/apache/arrow-adbc/blob/main/go/adbc/driver/snowflake/bulk_ingestion.go#L48C2-L48C164
Change to
createTemporaryStageStmt = "CREATE OR REPLACE TEMPORARY STAGE " + bindStageName + " FILE_FORMAT = (TYPE = PARQUET USE_LOGICAL_TYPE = TRUE BINARY_AS_TEXT = FALSE USE_VECTORIZED_SCANNER=TRUE REPLACE_INVALID_CHARACTERS = TRUE)"
See details about new option: https://medium.com/snowflake/faster-parquet-data-ingestion-with-snowflake-use-vectorized-scanner-28679bcff450
Temporary internal stages have SNOWFLAKE_FULL encryption by default which is not yet supported by USE_VECTORIZED_SCANNER. However it will be supported in the next month, at which point, the vectorized scanner performance improvement will kick in.
It's still possible to set this option now but it won't take effect as it will fall back to the old scanner. When the encryption support is added, it will work automatically.
The text was updated successfully, but these errors were encountered: