Parsers / loaders are a bit limited

[redhog]

Limitations:

• Can not use multiple input fields / the entire item
• Can not take additional parameters from config
• Can only output a single field
  • Use-cases for multiple fields:
    • Parse some text out of a document and additionally some metadata (set of name/value pairs)
    • Parse all columns of e.g. a csv into separate fields

How can we redesign this to make it more flexible?

[shreyashankar]

Agreed. One idea is for the custom parsers to accept a document and return a list of documents/dictionaries, which get merged into the original documents, instead of a list of strings. Then the user doesn't have to specify input_key or output_key.

[redhog]

I think that, coupled with arbitrary arguments from the config yaml to the parser, would be good enough. Parsers that wants to use an input key, would just take that as an argument.

We'd just change the yaml to be:

datasets:
  my_dataset:
    type: file
    source: local
    path: "data.json"
    parsing:
      - function: my_parser
        argument_name_1: value
        argument_name_2: value
        argument_name_3: value

And for the existing parsers, we add the parameters input_key and output_key and it will be compatible with old pipeline yaml files too :)

Signature for a parser function would then be:

def my_parser(
    item: dict[str, Any],
    **kw) -> list[dict[str, Any]]:

[shreyashankar]

I like this idea! I'm on board.

Feel free to take it if you'd like--I'm trying to get a v0 of the UI out this week :-) But i can also get to it later no worries

[redhog]

I'd be happy to do this one, but I want the rate_limit stuff merged first (with the runner argument), to limit merge conflicts...