DatabricksSqlSensor deferrable

[fjmacagno]

Description

Make DatabricksSqlSensor deferrable.

Use case/motivation

Much like making any sensor deferrable, it would substantially reduce worker load. But, would it be too intense for the triggerer?

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[jroachgolf84]

@fjmacagno, I'd suggest taking a look at using mode="reschedule" for Sensors (see the docs here: https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/sensors.html). This frees up the Worker slot when it's "sleeping" between checks. It does not run on the Triggerer.

[fjmacagno]

I know, and I use it exclusively over poke mode for the bulk of cases, but in k8s executor world it can still be intense to be spinning up hundreds of pods every 5 minutes, distracting the executor from handling actually productive tasks. Thus, this seems like a prime use-case for deferrable.

[jroachgolf84]

Fair enough! Are you interested in submitting a PR? If not, I'll see what I can do in terms of a PR (I've got another one assigned to me).

[jroachgolf84]

@rawwar, do you mind throwing me on as the assignee?

[fjmacagno]

Tragically i am buried with other work and wouldn't want to commit to it and then not have time. If you have the time it is all yours.

[jroachgolf84]

@fjmacagno, would it make more sense to take a closer look at making the DatabricksSqlOperator deferrable? Just thinking out loud, and curious if that lack of functionality today in the Operator itself was why you were reaching for the Sensor.

cc: @eladkal

[fjmacagno]

Honestly i think both would be useful, a common use-case of ours is "wait for data to land, then run a query to generate a new table". I think though it would be much harder to make the operator deferrable, since it relies on a persistent http connection to run. If I understand correctly its basically a jdbc connection, and terminating it will terminate the job, unless there is something fancy we could do.

I would guess the easiest way would be to write an operator on top of the DatabricksSubmitRunOperator or something like it which wraps the sql in a generated spark job. But, thats pretty different compute-wise.

[jroachgolf84]

Sounds good - I'm going to start with the deferrable Sensor, then work with the Databricks community and see if we want a more targeted approach to deferrable tools.