Task-level audit logs missing SUCCESS/RUNNING events in Airflow 3.1.x (only FAILED and state mismatch recorded)

[pluto0007]

Apache Airflow Version

3.1.0

Python 3.x

Postgres metadata DB

AIRFLOW_HOME = /opt/airflow

What happened

After upgrading to Airflow 3.1.0, the system no longer writes task-level SUCCESS or RUNNING events into the log table (and therefore /api/v2/eventLogs and the UI Audit Log show no success/running entries).

Only failed and state mismatch events are written for tasks.

DAG-level SUCCESS events are logged, but task-level ones are not.

This is a regression from Airflow 2.10.x, where task-level success and running events were always logged.

Reproducible example

A DAG (hiport_v2) with a single task (hiport_v2_1) is run multiple times.
The tasks complete successfully.

Then we query the audit log table:

SELECT dttm, event, dag_id, task_id, run_id
FROM log
WHERE dag_id = 'hiport_v2'
ORDER BY dttm DESC
LIMIT 20;

Actual output (sample):

dttm | event | dag_id | task_id | run_id
------------------------------+-----------------+------------+--------------+----------------------------------------
2025-10-22 14:19:53.705162+0 | trigger_dagrun | hiport_v2 | |
2025-10-16 12:30:02.359807+0 | failed | hiport_v2 | hiport_v2_1 | manual__2025-10-16T12:18:43.261104+00:00
2025-10-16 12:30:02.355367+0 | state mismatch | hiport_v2 | hiport_v2_1 | manual__2025-10-16T12:18:43.261104+00:00
...

There are no success or running events at all.

A global event summary confirms this:

SELECT event, COUNT(*)
FROM log
GROUP BY event
ORDER BY 2 DESC;

Output:

failed | 4102
state mismatch | 4102
trigger_dag_run | 38
cli_api_server | 51
...
(no "success")
(no "running")

REST API evidence

Calling:

GET /api/v2/eventLogs?dag_id=hiport_v2&order_by=-when

Returns only failed/state_mismatch, never success.

Even DAG runs that succeed show empty task-level events.

Expected Behavior

Task-level audit logging should:

Insert success events for every task that finishes successfully

Insert running events for every task that starts

Show these in:

UI → “Audit Log Events”

/api/v2/eventLogs

The log table

This was the behavior in Airflow 2.10.x and is also suggested in the 3.x audit log documentation (task lifecycle events include success/running/failure).

Actual Behavior

Only task-level failure events are recorded (failed, state mismatch).

No task-level success or running events are ever written.

/api/v2/eventLogs cannot be used to consume full lifecycle events anymore.

DAG-level success events still appear.

Why this matters

We use the official REST API (/api/v2/eventLogs) to drive an external orchestration/audit service that depends on complete event history.
With Airflow 3.1.x, task-level success events disappear completely, breaking downstream integrations.

How to Reproduce

Create any DAG with one or more tasks.

Run it successfully.

Query the audit log table:

SELECT * FROM log WHERE dag_id='<your_dag>' ORDER BY dttm DESC;

Observe that:

failed and state mismatch are logged,

but no success/running events appear.

/api/v2/eventLogs also shows no success events.

Additional Notes

Metadata backend: Postgres

Scheduler/Webserver run as root and use:

AIRFLOW__DATABASE__SQL_ALCHEMY_CONN=postgresql://airflowdbo:****@localhost:5432/airflow_db

Behavior is the same for all DAGs, not only one.

Request

Please confirm whether this is:

An intentional change in Airflow 3.x audit logging,
or

A regression/bug where task lifecycle events (success/running) are no longer logged.

If this is not intentional, can task-level success/running audit logging be restored in a 3.1.x+ patch?

We can provide more logs or a minimal DAG if needed.

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[sjyangkevin]

I am interested in looking into this issue.

[potiuk]

feel free

[sjyangkevin]

Hi @pluto0007 ,

I attempted to re-produce the issue but I would like to confirm if it exact the same as what you get.

Airflow 2 Test

Run airflow using breeze with the following command to observe the audit logs:

breeze start-airflow --python 3.12 --backend postgres --use-airflow-version 2.11.0 --db-reset

As you mentioned, the task lifecycle state transition (queued, running, success, failed) is logged.

Airflow 3 Test

Run Airflow 3.1.5 with

breeze start-airflow --python 3.12 --backend postgres --use-airflow-version 3.1.5 --auth-manager FabAuthManager --db-reset --mount-sources providers-and-tests

The same DAG that I used in Airflow 2 is ran, but I only observed user action, as shown below.

I triggered the DAG and it resulted two events 1.) unpause through PATCH, 2.) trigger through POST. However, I cannot see any task lifecycle state transition being logged. You mentioned:

failed and state mismatch are logged

I wonder if there is additional setting I should add to it, or if my set up result in some issues that these states are not logged. It will be helpful if you could provide some examples, or steps on how your DAG is run such that the logs for failed task appears.

I did some research over the issues and PRs and find the followings:

1. AIP-84 Log User Actions / Server Accesses #44057
2. AIP-84 Log User Actions #45609
3. feat(hitl): record HITLDetail action in audit log #54767

From my understanding, it looks like these user actions are logged through the action_logging. I wonder if now the log table only tracks for user interactions. As mentioned in the document, system generated events include task lifecycle state transition should be in the scope of audit log.

airflow/airflow-core/src/airflow/api_fastapi/logging/decorators.py

Line 77 in 8d28fe8

def action_logging(event: str | None = None):

I would like to also check if I am on the right track, and would appreciate if anyone with more experience in this area could provide some insights. Please correct me if my set up is wrong. Meanwhile, I will continue to explore to see if I could better understand it. Thanks!

Edited:

I also noticed ti_update_state is updating the task state, wonder if it can be the place I start to look into.

airflow/airflow-core/src/airflow/api_fastapi/execution_api/routes/task_instances.py

Lines 340 to 345 in 8d28fe8

	def ti_update_state(
	task_instance_id: UUID,
	ti_patch_payload: Annotated[TIStateUpdate, Body()],
	session: SessionDep,
	dag_bag: DagBagDep,
	):

Meanwhile understanding what are actually logged by the followings:

airflow/airflow-core/src/airflow/jobs/scheduler_job_runner.py

Line 1064 in 8d28fe8

session.add(Log(event="state mismatch", extra=msg, task_instance=ti.key))

airflow/airflow-core/src/airflow/jobs/scheduler_job_runner.py

Lines 880 to 882 in 8d28fe8

	def _process_task_event_logs(log_records: deque[Log], session: Session):
	objects = (log_records.popleft() for _ in range(len(log_records)))
	session.bulk_save_objects(objects=objects, preserve_order=False)

airflow/airflow-core/src/airflow/executors/base_executor.py

Lines 215 to 217 in 8d28fe8

	def log_task_event(self, *, event: str, extra: str, ti_key: TaskInstanceKey):
	"""Add an event to the log table."""
	self._task_event_logs.append(Log(event=event, task_instance=ti_key, extra=extra))

airflow/airflow-core/src/airflow/models/taskinstance.py

Lines 1139 to 1140 in 8d28fe8

	if not test_mode:
	session.add(Log(TaskInstanceState.RUNNING.value, ti))

airflow/airflow-core/src/airflow/models/taskinstance.py

Lines 1464 to 1465 in 8d28fe8

	if not test_mode:
	session.add(Log(TaskInstanceState.FAILED.value, ti))

airflow/airflow-core/src/airflow/utils/cli_action_loggers.py

Lines 132 to 136 in 8d28fe8

	session.bulk_insert_mappings(
	Log, # type: ignore[arg-type]
	[
	{
	"event": f"cli_{sub_command}",

[Lee-W]

I also noticed ti_update_state is updating the task state, wonder if it can be the place I start to look into.

Yes

Meanwhile understanding what are actually logged by the followings:

I think this is the right track. The easiest way would probably be adding Log in ti_update_state. but there might be some existing utility function you can reuse. action_logging "might" not be the thing we want to reuse. This pattern is not used in execution_api (only in core_api for the time being)

[sjyangkevin]

I also noticed ti_update_state is updating the task state, wonder if it can be the place I start to look into.
Yes

Thanks for the feedback. I will try out adding the logging into ti_update_state. Meanwhile, I am also looking into the 2.11.0 code and trying to better understand how those states are actually logged, I feel that would also help me to find out the proper place to implement the fix.

airflow/airflow/models/taskinstance.py

Lines 369 to 372 in d9ed7b9

	if not test_mode:
	_add_log(event=ti.state, task_instance=ti, session=session)
	if ti.state == TaskInstanceState.SUCCESS:
	ti._register_dataset_changes(events=context["outlet_events"], session=session)

action_logging "might" not be the thing we want to reuse. This pattern is not used in execution_api (only in core_api for the time being)

Yeah, agree on this. From my understanding, it looks more like to log user actions instead of system-generated events (e.g., task lifecycle state transition).