Feature request: package custom Python libraries and its PyPI dependencies

[n-batalha]

Context

We are using dbt for Python "models", which handles Python stored procedures to create tables and abstracts that from users.

However they are leaving it to users (at least for now) to handle the pushing of code outside the model into a stage.

Although I think the feature requested here applies more broadly to reuse logic between stored procedures/UDFs.

Request

To be able to create a package of a user defined Python library, that depends on a number of external libs (in Anaconda or alternatively in PyPI), with the packaging of the library and PyPI dependencies being handled and submitted to a Snowflake stage.

E.g. snow package lib <lib location>, which packages a local library defined in <lib location> alongside its dependencies.

Example

I managed to do one using your wrapper for packaging PyPI dependencies:

Then:

snow package create pyjokes
snow package upload -f pyjokes.zip -s packages --overwrite

And in dbt, a model is written as:

import pandas as pd

def model(dbt, session):
    dbt.config(
        packages = ["pandas"],
        imports = [
            "@packages/pyjokes.zip",
        ]
    )

    from mylib.mylib import get_some_joke

    message = get_some_joke()

    return pd.DataFrame({"joke": [message]})

But I miused the snow package create <package> by depending only on a single external package (pyjokes), and "knowing" your implementation compresses the local folder (which includes my lib) in a zip and then conveniently submits it to a stage. Ideally the user lists a number of dependencies (e.g. pyproject.toml), Anaconda ones are ignored and PyPI only ones are pulled and compressed alongside our lib in a zip, and pushed to a stage.

[n-batalha]

Actually inspecting the code, I found that there is a route, using the package command on a wheel with the user created library (which has 3rd party dependencies - the wheel file knows the dependencies of the lib, you seem to invoke pip which pulls the dependencies of the wheel, and your CLI takes care of the rest). E.g.:

snow package create example-1.0.0-py3-none-any.whl
mv example-1.0.0-py3-none-any.whl.zip example.zip
snow package upload -f example.zip -s packages --overwrite

Not sure if this was considered or is supported, so keeping it open.

[recursive-automata]

Starting from your solution, @n-batalha, ... package create ... can be replaced with pip install -t and zip -r. This puts the top level module (named example below) in the right format for ... package upload ....

pip install -U -t $TARGET_DIR $PACKAGE_DIR
cd $TARGET_DIR
zip -r example.zip example/
snow snowpark package upload -f example.zip -s packages --overwrite -c $CONNECTION_NAME

[sfc-gh-turbaszek]

@n-batalha I'm closing this one as there were numerous fixes in package commands in 2.0 version. If this still persists in 2.0 please reopen this issue.