Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added worker for Snowflake's Snowpark Container Services #16393

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

bjorhn
Copy link
Contributor

@bjorhn bjorhn commented Dec 15, 2024

This PR adds a new worker to the prefect-snowflake integration. The worker can run service jobs in Snowflake's Snowpark Container Services. Closes #15674

A few things worth mentioning:

  • Snowflake's Python API currently doesn't support running async service jobs, so I'm temporarily running jobs using SQL commands instead. I will submit a PR to resolve this as soon as Snowflake adds the required features to their Python API.
  • Most of the code was hacked together based on the official "custom worker" tutorial and copying things over from the Azure and Kubernetes workers. The same goes for the unit tests.
  • Since a lot of the structure was copied over, let me know if I misunderstood how something should be done. For example, am I using run_sync_in_worker_thread correctly?
  • I don't have a lot of experience creating unit tests in Python, so I've done the bare minimum with regards to unit testing.

I still need to figure out how to test the worker after moving the modules over from our company repo to the Prefect repo.

There are two bugs I've bumped into along the way which I don't think are caused by my code. I don't know if these issues only affect custom workers, but I think so:

Finally, I got a few "minimum Python version errors" during pre-commit, which I am not sure how to interpret. I'm think I'm targeting 3.9 and later, which I assume is fine. Perhaps someone could point me in the right direction?

src/integrations/prefect-snowflake/prefect_snowflake/workers/spcs.py:6:5:!2:3.11:'datetime.UTC' member
src/integrations/prefect-snowflake/prefect_snowflake/workers/spcs.py:::!2:3.11:
:::!2:3.11:

@github-actions github-actions bot added the enhancement An improvement of an existing feature label Dec 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement An improvement of an existing feature
Projects
None yet
Development

Successfully merging this pull request may close these issues.

New worker for running job services in Snowpark Container Services
1 participant