Python entrypoints defined with Click don't work

[astrojuanlu]

Because of pallets/click#1054, pallets/click#2249, Python entrypoints defined with Click don't work.

Simple reproducer: https://github.com/astrojuanlu/simple-click-entrypoint

Job definition:

Result:

---

Document Details

⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

• ID: 3dd43252-46d7-3022-60b4-a81f844ec3b6
• Version Independent ID: 3dd43252-46d7-3022-60b4-a81f844ec3b6
• Content: Use a Python wheel in an Azure Databricks job - Azure Databricks
• Content Source: databricks/workflows/jobs/how-to/use-python-wheels-in-workflows.md
• Service: azure-databricks
• GitHub Login: @mssaperla
• Microsoft Alias: saperla

[SaibabaBalapur-MSFT]

@astrojuanlu
Thanks for your feedback! We will investigate and update as appropriate.

[kateglee-db]

Thanks for providing feedback that helps improve our documentation. We've created an internal work item (DOC-11889) to address your feedback. The timeline for resolution varies based on resourcing.

#please-close

[astrojuanlu]

Thanks - do we get a notification here when it's resolved (if ever)?

[kateglee-db]

@astrojuanlu I sent your feedback to the writer who recommends that if you're blocked from using a library you need, reach out to support at https://help.databricks.com/. If the intent is to add this to the docs, this is an edge case, which we don't document in the tech docs. Support might document it in the KB.

[astrojuanlu]

If I can emit a recommendation, it would be nice to document in the same page how the entry point execution is performed in better detail. If I squint, I see two hints, one in the page itself:

Both positional and keyword arguments are passed to the Python wheel task as command-line arguments.

And in the traceback:

     16 if entry:
     17   # Load and execute the entrypoint, assumes no parameters
---> 18   entry[0].load()()
     19 else:

This tells me that, internally, Databricks has some script that loads the entry point and executes it, and expects said entry point to read the parameters from the sys.argv (because the function is called without arguments).