Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

After migrating from version 2.2.5 to 2.3.0 airflow is unable to view the external scheduler logs #24244

Closed
1 of 2 tasks
Haapalaj opened this issue Jun 6, 2022 · 5 comments
Closed
1 of 2 tasks
Labels
area:core kind:bug This is a clearly a bug

Comments

@Haapalaj
Copy link

Haapalaj commented Jun 6, 2022

Apache Airflow version

2.3.2 (latest released)

What happened

Having the external log's configured to be stored in AWS S3. After updating from 2.2.x version to the 2.3.x, the backward compatibility of fetching task log entries made with the earlier version (2.2.5), fails because the default log template has changed and are not found with that.
The log template with airflow version before 2.3.x was "'{{ ti.dag_id }}/{{ ti.task_id }}/{{ ts }}/{{ try_number }}.log'"
With version 2.3.x it is "{{ ti.dag_id }}/run_id={{ ti.run_id }}/task_id={{ ti.task_id }}/{% if ti.map_index >= 0 %}map_index={{ ti.map_index }}/{% endif %}attempt={{ try_number }}.log"

There was discussion as a workaround (#22992) to add the old template to airflow template table:
INSERT INTO log_template (id, filename, elasticsearch_id, created_at) VALUES (0, '{{ ti.dag_id }}/{{ ti.task_id }}/{{ ts }}/{{ try_number }}.log', '{dag_id}_{task_id}_{run_id}_{try_number}', NOW());
That did not change the behaviour. Old task log entries still searched by the new template.

There is a reference log_template_id from the airflow table dag_run to templates. That reference is null for the old entries.
Tested to update those records to use the old template:
UPDATE dag_run set log_template_id = 0 WHERE log_template_id IS NULL;
But that did not change the behaviour either.

What you think should happen instead

Task log entries made with the airflow version 2.2.x (Or pre 2.3.x) should be still opened with the log template they were made with.

How to reproduce

Try to open task log entries made with airflow version 2.2.x with the airflow 2.3.x.

Operating System

Linux, Centos

Versions of Apache Airflow Providers

apache-airflow-providers-snowflake==2.6.0
apache-airflow-providers-amazon==3.3.0
apache-airflow-providers-google==6.1.0
apache-airflow-providers-microsoft-azure==3.1.1
apache-airflow-providers-odbc==2.0.1
apache-airflow-providers-postgres==2.2.0

Deployment

Other Docker-based deployment

Deployment details

The official Airflow docker image hosted in the AWS ECS Cluster.
apache/airflow:2.3.2-python3.7

Anything else

This happen's every time older task log's is fetched.

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

@Haapalaj Haapalaj added area:core kind:bug This is a clearly a bug labels Jun 6, 2022
@boring-cyborg
Copy link

boring-cyborg bot commented Jun 6, 2022

Thanks for opening your first issue here! Be sure to follow the issue template!

@uranusjr
Copy link
Member

uranusjr commented Jun 6, 2022

See #24153. We'll fix it in 2.3.3, I think.

@Haapalaj
Copy link
Author

Haapalaj commented Jun 6, 2022

Wonderful! We will test this then.

@Haapalaj
Copy link
Author

Updated from airflow 2.3.2 to 2.3.3 and after doing this in the airflow db:

INSERT INTO log_template (id, filename, elasticsearch_id, created_at) VALUES (0, '{{ ti.dag_id }}/{{ ti.task_id }}/{{ ts }}/{{ try_number }}.log', '{dag_id}{task_id}{run_id}_{try_number}', NOW());

Fetching old logs that previously did not open, started now to work!

@eladkal
Copy link
Contributor

eladkal commented Jul 20, 2022

Closed as fixed in 2.3.3

@eladkal eladkal closed this as completed Jul 20, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:core kind:bug This is a clearly a bug
Projects
None yet
Development

No branches or pull requests

3 participants