Clean up and streamline pyproject.toml

[ajbozarth]

As discussed in #430 (comment) and #430 (comment) the pyproject.toml need some looking at.

We should discuss and update:

1. How we want to handle project dependencies vs optional dependencies vs dev dependencies
2. The project authors section
3. Should all the ruff configs be moved into a separate ruff.toml
4. Should all the pytest configs be moved into a separate pytest.toml

These can be split out into separate sub-issues or PRs after discussion

[ajbozarth]

I'm willing to do the dev work on this once there's some discussion and consensus

[psschwei]

+1 for the general idea, need some time to think through specific recommendations

[ajbozarth]

Also worth noting as mention in #537 once we tackle sorting out the dependencies we would also want to go through the code base and make updates based on our decisions. We currently have quite a few work around in the code around optional dependencies that could be updated or removed based on what we change.

[jakelorocco]

An additional concern we should try to fix with our package dependencies: some dependencies are not explicitly needed (except for tests / type checking). We should determine how to handle those packages / optional dependencies.

For instance, none of our actual code requires lanchain-community or ddgs to be installed, but some of our examples do.

[ajbozarth]

My pitch for the dependence would be to group them into the following groups:

• dependencies - any dependencies required for the core library (essentially anything that gets pip installed from pypi)
• examples - any dependencies required only in the examples (its arguable that this could be grouped with dev)
• dev - any dependencies only required in tests
• various optional - dependencies that are used in core, but include heavy overhead that some users might not want, should be group by feature and limited to a particular sub package in core

We should also make sure that our documented install steps for contributing installs the dev dependencies

[ajbozarth]

The project authors section

For this I would suggest we switch to project mellea contributors and use the contact email we made rather than listing off all the individual people, but I understand if the current authors listed would want to keep their name on the pypi release

[psschwei]

The project authors section

For this I would suggest we switch to project mellea contributors and use the contact email we made rather than listing off all the individual people, but I understand if the current authors listed would want to keep their name on the pypi release

we could also add a CITATION.cff file to the repo

[ajbozarth]

we could also add a CITATION.cff file to the repo

Just did a quick google and that's an awesome feature, I've never used it before

+1 on this idea

[nrfulton]

For this I would suggest we switch to project mellea contributors and use the contact email we made rather than listing off all the individual people, but I understand if the current authors listed would want to keep their name on the pypi release

For sure we should do this. The old authors list is unhelpful and wrong/out-of-date. Maintaining a full authors list in the pyproject is also kind of pointless -- operationally, it makes a lot more sense for that line to be just contact line that redirects to someone(s) who's okay wearing a "pager" and has admin to pypi/github.

(We could separately do the "Contributors" section in a README with all of the little circle faces, but even that is kind of obviated by the Contributors feature on GitHub anyways, which self updates...)

[nrfulton]

My pitch for the dependence would be to group them into the following groups:

* `dependencies` - any dependencies required for the core library (essentially anything that gets pip installed from pypi)

* `examples` - any dependencies required only in the examples (its arguable that this could be grouped with `dev`)

* `dev` - any dependencies only required in tests

* various `optional` - dependencies that are used in core, but include heavy overhead that some users might not want, should be group by feature and limited to a particular sub package in core

We should also make sure that our documented install steps for contributing installs the dev dependencies

This seems reasonable. I think most of the "optional" are backend-related, or docling, and probably stuff like audio in the future.

This is probably roughly what we were going for originally

[ajbozarth]

This seems reasonable. I think most of the "optional" are backend-related, or docling, and probably stuff like audio in the future.

This is probably roughly what we were going for originally

In this case the follow up question would be what optional dependencies should we keep? this is our current list:

• watsonx
• docling
• hf
• vllm
• litellm
• tools
• telemetry

From this list we can combine or separate some, or move them to example or dev (eg tools can be renamed examples)