Optional output from the node

[IanVlasov]

Hello!
I am curious if it is possible to make part of node outputs optional. I have several functions with pretty similar interface (feature selectors), but I would like to return slightly different metadata from them. Here is my node definition:

node(
    COLLECTION_OF_FUNCS[string_repr_of_func],
    inputs={
        "dataset",
        "params"
    },
    outputs=[
        "updated_dataset",
        "feature_importance.csv",
        "importance_picture.png",
        "metrics_of_importance_algorithm"
    ],
    name="feature_selection_node"
)

string_repr_of_func comes from the parameters' file which is defined by user, but not all of these functions can produce complete metadata (e.g., picture or metrics). I would like to return None instead of them and simply do nothing with the absence of metadata (just skip saving process). Could I handle such a case somehow?

Thanks in advance for all your help.

[datajoely]

Hi @IanVlasov this is a tricky one as in general we don't love dynamic pipelines like this due to the combinatorial complexity it can introduce when maintaining and supporting things.

Our modular pipelines pattern is the our modern way of defining pipelines where you can re-use the same node logic over and over with dynamic inputs/outputs/parameters.

The other way to do this is to define a custom DataSet that inherits from the one you are already using which includes the logic to not perform the save() operation if the data provided is None.