-
Notifications
You must be signed in to change notification settings - Fork 98
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support micro seconds timestamp precision with copy unload #491
Comments
@arthurli1126 Could you please try SC 2.6.0 or newer version? |
@sfc-gh-mrui thanks for your reply, the problem is with the simpleDateTime(https://github.com/snowflakedb/spark-snowflake/blob/master/src/main/scala/net/snowflake/spark/snowflake/Conversions.scala#L66) used for parse timestamp during copy unload. The format only support millisecond precision and would give wrong timestamp if the timestamp carries micro seconds. For instance, for string |
Created a draft PR for this: #492 Please let me know if I misunderstand anything or if you have any concerns about the approach. |
Hi folks, due to high NAT gateway cost, we have to use copy load when reading from snowflake, but currently copy unload doesn't support micro second level precision(only at mills). I can work on the PR to add it. But wondering if you have any concerns about this.
The text was updated successfully, but these errors were encountered: