[ENH] - Support Apache Flink for baking Pangeo-Forge recipes with Beam

[rsignell-usgs]

Feature description

It would be nice to support Apache Flink for (among other things), baking Pangeo-Forge recipes with Beam, as described here:
https://github.com/pangeo-forge/pangeo-forge-runner/blob/main/docs/tutorial/flink.md#run-a-recipe-on-a-flink-cluster-on-aws

Value and/or benefit

Currently we are limited to Dask executors but some workflows really benefit from Beam, like the pangeo-forge recipes

Anything else?

No response

[iameskild]

Hey @rsignell-usgs, thanks for the suggestion! This sounds like this might be a good candidate to add as a Nebari extension #865. Adding a more robust extension mechanism is something we're exploring at the moment but we can circle back to this once we have some more concrete ideas of how it would work.

[iameskild]

Thanks again @rsignell-usgs for the suggestion. Now that we have implemented the Nebari extension mechanism, I think this would be a good candidate for a new extension. Closing this as not planned but welcome the community to develop this should there be a demand for it.