Allow manual trigger arguments to be passed to dbt nodes #151
Comments
|
@MikeWallis42 not sure how this would work since our global args and command-specific args are parameters on the operators. Can you clarify what the vision is here? |
|
@pohek321 it's possible to pass in JSON configuration to an Airflow DAG when manually triggering it. {
"run": [
{
"model_name": "orders",
"args": [
"--full-refresh"
]
}
]
}Within the operator then we can get these from the DAG run context and merge them with the others. run_args = context["dag_run"].conf.get("run")We should stipulate what can and can't be added which might inform the JSON structure. I'm just trying to limit the amount of DAG authoring we might have to do because quite a lot of us don't have production access so would have to dev and release to be able to do and ad-hoc process like a full refresh. |
|
Passing configuration through the DAG configuration is super useful and I would love to see that. I will try to make a PR that allows templating of the operator_args, so that we can do something like the following example: operator_args={"full-refresh": True if "{{ dag_run.conf.get('full_refresh') }}" == 'True' else False}, |
This allows you to fully refresh a model from the console.
Full-refresh/backfill is a common task. Using Airflow parameters makes
this easy. Without this, you'd have to trigger an entire deployment. In our
setup, company analysts manage their models without modifying
the DAG code. This empowers such users.
Example of usage:
```python
with DAG(
dag_id="jaffle",
params={"full_refresh": Param(default=False, type="boolean")},
render_template_as_native_obj=True
):
task = DbtTaskGroup(
operator_args={"full_refresh": "{{ params.get('full_refresh') }}", "install_deps": True},
)
```
Closes: #151
This allows you to fully refresh a model from the console.
Full-refresh/backfill is a common task. Using Airflow parameters makes
this easy. Without this, you'd have to trigger an entire deployment. In our
setup, company analysts manage their models without modifying
the DAG code. This empowers such users.
Example of usage:
```python
with DAG(
dag_id="jaffle",
params={"full_refresh": Param(default=False, type="boolean")},
render_template_as_native_obj=True
):
task = DbtTaskGroup(
operator_args={"full_refresh": "{{ params.get('full_refresh') }}", "install_deps": True},
)
```
Closes: astronomer#151
Things like dbt's full-refresh are likely to be run in exceptional circumstances and should not need a deployment to trigger.
In the style of a user story:
The text was updated successfully, but these errors were encountered: