Skip to content

Conversation

@jason810496
Copy link
Member

related: #56191

The test_remote_logging e2e test is flaky because some of the logs are not yet uploaded to S3 before we validating log file names.

Example fail run: https://github.com/apache/airflow/actions/runs/18613327530/job/53075269841?pr=55962

=========================== short test summary info ============================
FAILED tests/airflow_e2e_tests/remote_log_tests/test_remote_logging.py::TestRemoteLogging::test_remote_logging_s3 - AssertionError: None of the log sources ['/opt/airflow/logs/dag_id=example_xcom_test/run_id=manual__2025-10-18T09:13:28.389807+00:00/task_id=bash_pull/attempt=1.log'] were found in S3 bucket logs ['s3://test-airflow-logs/dag_id=example_xcom_test/run_id=manual__2025-10-18T09:13:28.389807+00:00/task_id=bash_push/attempt=1.log', 's3://test-airflow-logs/dag_id=example_xcom_test/run_id=manual__2025-10-18T09:13:28.389807+00:00/task_id=pull_value_from_bash_push/attempt=1.log', 's3://test-airflow-logs/dag_id=example_xcom_test/run_id=manual__2025-10-18T09:13:28.389807+00:00/task_id=puller/attempt=1.log', 's3://test-airflow-logs/dag_id=example_xcom_test/run_id=manual__2025-10-18T09:13:28.389807+00:00/task_id=push/attempt=1.log', 's3://test-airflow-logs/dag_id=example_xcom_test/run_id=manual__2025-10-18T09:13:28.389807+00:00/task_id=push_by_returning/attempt=1.log']
assert False
 +  where False = any(<generator object TestRemoteLogging.test_remote_logging_s3.<locals>.<genexpr> at 0x7f6276c88dd0>)
=================== 1 failed, 1 passed, 1 warning in 46.40s ====================

What

Add a retry mechanism to verify that a minimum number of TaskInstance logs are present in S3 before proceeding with validation.

@jason810496 jason810496 changed the title Add retry for test_remote_logging_s3 e2e test Add retry for e2e test_remote_logging_s3 Oct 18, 2025
Copy link
Member

@potiuk potiuk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fantastic!

@potiuk potiuk merged commit 52250f6 into apache:main Oct 18, 2025
51 checks passed
abdulrahman305 bot pushed a commit to abdulrahman305/airflow that referenced this pull request Oct 19, 2025
* Add retry for test_remote_logging_s3 e2e test

* Print list of contents found

* Rename retry_delay_seconds as retry_interval_in_seconds
nailo2c pushed a commit to nailo2c/airflow that referenced this pull request Oct 20, 2025
* Add retry for test_remote_logging_s3 e2e test

* Print list of contents found

* Rename retry_delay_seconds as retry_interval_in_seconds
TyrellHaywood pushed a commit to TyrellHaywood/airflow that referenced this pull request Oct 22, 2025
* Add retry for test_remote_logging_s3 e2e test

* Print list of contents found

* Rename retry_delay_seconds as retry_interval_in_seconds
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants