You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Having the external log's configured to be stored in AWS S3. After updating from 2.2.x version to the 2.3.x, the backward compatibility of fetching task log entries made with the earlier version (2.2.5), fails because the default log template has changed and are not found with that.
The log template with airflow version before 2.3.x was "'{{ ti.dag_id }}/{{ ti.task_id }}/{{ ts }}/{{ try_number }}.log'"
With version 2.3.x it is "{{ ti.dag_id }}/run_id={{ ti.run_id }}/task_id={{ ti.task_id }}/{% if ti.map_index >= 0 %}map_index={{ ti.map_index }}/{% endif %}attempt={{ try_number }}.log"
There was discussion as a workaround (#22992) to add the old template to airflow template table: INSERT INTO log_template (id, filename, elasticsearch_id, created_at) VALUES (0, '{{ ti.dag_id }}/{{ ti.task_id }}/{{ ts }}/{{ try_number }}.log', '{dag_id}_{task_id}_{run_id}_{try_number}', NOW());
That did not change the behaviour. Old task log entries still searched by the new template.
There is a reference log_template_id from the airflow table dag_run to templates. That reference is null for the old entries.
Tested to update those records to use the old template: UPDATE dag_run set log_template_id = 0 WHERE log_template_id IS NULL;
But that did not change the behaviour either.
What you think should happen instead
Task log entries made with the airflow version 2.2.x (Or pre 2.3.x) should be still opened with the log template they were made with.
How to reproduce
Try to open task log entries made with airflow version 2.2.x with the airflow 2.3.x.
Apache Airflow version
2.3.2 (latest released)
What happened
Having the external log's configured to be stored in AWS S3. After updating from 2.2.x version to the 2.3.x, the backward compatibility of fetching task log entries made with the earlier version (2.2.5), fails because the default log template has changed and are not found with that.
The log template with airflow version before 2.3.x was "'{{ ti.dag_id }}/{{ ti.task_id }}/{{ ts }}/{{ try_number }}.log'"
With version 2.3.x it is "{{ ti.dag_id }}/run_id={{ ti.run_id }}/task_id={{ ti.task_id }}/{% if ti.map_index >= 0 %}map_index={{ ti.map_index }}/{% endif %}attempt={{ try_number }}.log"
There was discussion as a workaround (#22992) to add the old template to airflow template table:
INSERT INTO log_template (id, filename, elasticsearch_id, created_at) VALUES (0, '{{ ti.dag_id }}/{{ ti.task_id }}/{{ ts }}/{{ try_number }}.log', '{dag_id}_{task_id}_{run_id}_{try_number}', NOW());
That did not change the behaviour. Old task log entries still searched by the new template.
There is a reference log_template_id from the airflow table dag_run to templates. That reference is null for the old entries.
Tested to update those records to use the old template:
UPDATE dag_run set log_template_id = 0 WHERE log_template_id IS NULL;
But that did not change the behaviour either.
What you think should happen instead
Task log entries made with the airflow version 2.2.x (Or pre 2.3.x) should be still opened with the log template they were made with.
How to reproduce
Try to open task log entries made with airflow version 2.2.x with the airflow 2.3.x.
Operating System
Linux, Centos
Versions of Apache Airflow Providers
apache-airflow-providers-snowflake==2.6.0
apache-airflow-providers-amazon==3.3.0
apache-airflow-providers-google==6.1.0
apache-airflow-providers-microsoft-azure==3.1.1
apache-airflow-providers-odbc==2.0.1
apache-airflow-providers-postgres==2.2.0
Deployment
Other Docker-based deployment
Deployment details
The official Airflow docker image hosted in the AWS ECS Cluster.
apache/airflow:2.3.2-python3.7
Anything else
This happen's every time older task log's is fetched.
Are you willing to submit PR?
Code of Conduct
The text was updated successfully, but these errors were encountered: