TypeError in _handle_heartbeat_failures logging

[karenbraganz]

Apache Airflow version

3.1.0

If "Other Airflow 2/3 version" selected, which one?

No response

What happened?

Supervision of a task instance failed after a single failed heartbeat attempt even though the max_failed_heartbeats is set to 3. This happened because an exception was raised when the _handle_heartbeat_failures function was called.

During the first failed heartbeat attempt, the _handle_heartbeat_failures function logs a message by calling log.warning(), which accepts an exception parameter that expects a string type object. However, in the source code, an exception type object is passed instead of a string type object. This results in a TypeError (like below) which causes task supervision to fail.

TypeError: can only concatenate str (not "RemoteProtocolError") to str

I have attached the stack trace from the worker logs.

worker_typeerror.rtf

What you think should happen instead?

I believe a string should be passed (like below) instead of an exception object here.

exception=str(exc)

Alternatively, I think we could pass exc_info=True instead of the exception parameter. This is what was done until 3.0.4. I am not sure if there was a specific reason for changing this.

How to reproduce

Run a task instance and kill the API server once the task instance begins running.

You will notice that the task supervision fails upon the first failed heartbeat.

Operating System

Debian GNU/Linux

Versions of Apache Airflow Providers

No response

Deployment

Astronomer

Deployment details

No response

Anything else?

This also affects Airflow 3.0.5 and 3.0.6. Prior to that, exc_info=True was used.

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[amoghrajesh]

Alternatively, I think we could pass exc_info=True instead of the exception parameter. This is what was done until 3.0.4. I am not sure if there was a specific reason for changing this.

The reason for that @karenbraganz was ruff! The version of ruff we currently use objects it and complains

Run 'ruff' for extremely fast Python linting.....................................Failed
- hook id: ruff
- exit code: 1
  LOG014 `exc_info=` outside exception handlers
      --> task-sdk/src/airflow/sdk/execution_time/supervisor.py:1140:13
       |
  1138 |             ti_id=self.id,
  1139 |             max_retries=MAX_FAILED_HEARTBEATS,
  1140 |             exc_info=True,
       |             ^^^^^^^^^^^^^
  1141 |         )
  1142 |         # If we've failed to heartbeat too many times, kill the process
       |
  help: Remove `exc_info=`