First attempt at a parametrized JobCreate

[javiermtorres]

What's changing

The JobCreate schema is changed to include a separate specific job_config. The openapi produced includes a oneOf constraint:

      "JobCreate": {
        "properties": {
          "name": {
            "type": "string",
            "title": "Name"
          },
[...]
          "job_config": {
            "oneOf": [
              {
                "$ref": "#/components/schemas/JobEvalConfig"
              },
              {
                "$ref": "#/components/schemas/JobEvalLiteConfig"
              },
              {
                "$ref": "#/components/schemas/JobInferenceConfig"
              },
              {
                "$ref": "#/components/schemas/JobAnnotateConfig"
              }
            ],
            "title": "Job Config",
            "discriminator": {
              "propertyName": "job_type",
              "mapping": {
                "annotate": "#/components/schemas/JobAnnotateConfig",
                "eval_lite": "#/components/schemas/JobEvalLiteConfig",
                "evaluate": "#/components/schemas/JobEvalConfig",
                "inference": "#/components/schemas/JobInferenceConfig"
              }
            }
          }
        },

The jobs and experiments services are changed accordingly.

Closes #706

How to test it

Tests should run correctly.

Additional notes for reviewers

N/A

I already...

• Tested the changes in a working environment to ensure they work as expected
• Added some tests for any new functionality
• Updated the documentation (both comments in code and product documentation under /docs)
• Checked if a (backend) DB migration step was required and included it if required
  • No DB migration needed

[njbrake]

Looking good so far! Made a code suggestion but looks like a very logical refactor. My only question would be whether you plan on addressing the custom logic of _get_job_params in this PR. If you don't plan on addressing it here, can you make a separate issue to track elevating that out of the service layer?

[javiermtorres]

The SDK needs to be updated. I have checked the backend unit and integration tests locally and they seem to work.

[javiermtorres]

The SDK and notebook tests have been updated. @veekaybee @aittalam I've changed the code of the notebook slightly. One important difference is that I have removed the model param in the eval lite job. AFAICT, it's not needed there. The notebook takes it from the initial model spec in the notebook and not from the output of the summarization job. Since the output is a csv, it didn't make sense to put the model there, but I'll check the results metadata.

[njbrake]

My concern is that this PR is dropping support for the JobType.EVALUATION, which is needed to support the current frontend design. I may misunderstand the code. Other than that, only minor comments. Thanks for the work on this! (Let me know about JobType.EVALUATION and then I'll approve once that's worked out).

[ividal]

Thanks for this! One note, @javiermtorres this PR should still be in sync with the UI and keep an eye on how it interacts with /experiments.

[ividal]

Just a note that uploading a file w/o ground-truth and then clicking on "generate ground-truth" generates a "job not found error".

See log:
2025-02-10 - PR740.log

[ividal]

Tested #847 in the context of reviewing #740 . I've only looked at the backend code (740), not the frontend code. But on the testing front, I did a demo:

• uploading a dataset with and w/o gt
• annotating one w/o gt
• launching an experiment with both local and API-based models

Overall, works as expected (🥳 ). There are a number of smaller issues, but let's get 847 in here and then 740 into main - and iterate on smaller separate issues :)

[ividal]

With #847 in here and checks green, we have ourselves a usable new experiments workflow :)

THANK YOU for the effort @javiermtorres @khaledosman