-
Notifications
You must be signed in to change notification settings - Fork 7
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Remove SerializebleLock
in to_snowflake
#39
Conversation
With the larger dataframe, the partition sizes when we read back from snowflake are coming back a bit erratically sized. This is without any changes to core.py Locally, I get:
|
Push a commit to remove the SerializableLock. The difference in test durations is substantial: Without 399da14 (status quo):
With 399da14 (this PR):
|
Similar performance boost in CI: With Lock: https://github.com/coiled/dask-snowflake/actions/runs/3535816356/jobs/5934242005#step:5:70 |
@jrbourbeau is this something we should revisit? |
If the threading problem is resolved, I'd recommend removing the |
SerializebleLock
to speed up writesSerializebleLock
in to_snowflake
@jrbourbeau are you still on this? |
Heyyy look that it. Finally merged! 🎉 |
You just gotta plant a lotta seeds 🙂 |
Fixes #29
Just testing things out for now