Restarting a single mapped task restarts all sibling tasks as well

[MarcusCramer91]

Apache Airflow version

3.0.4

If "Other Airflow 2 version" selected, which one?

No response

What happened?

Cannot restart a single mapped task. It always restarts all sibling mapped tasks as well. I found this old issue #24699 so I thought it should be working, but doesn't for me. We have multiple issues here:

1. When clearing the individual task it shows all siblings in the "Downstream" selection.
2. It re-runs all sibling tasks. Cannot tell for certain this isn't just a UI bug in this simple example. But in my real-world use case, where Airflow triggers Google Cloud Run Jobs, it does spawn n Cloud Run Jobs upon restart.
3. It also re-runs all downstream tasks for all map indices.
4. For previous task executions, upon UI reload, I only get "::group::Log message source details: sources=["Could not read served logs: 'TaskInstanceHistory' object has no attribute 'triggerer_job'"]" and not the actual logs.

Or am I misunderstanding this feature?

What you think should happen instead?

Only the selected mapped task should be restarted.

The logs should properly show up in the UI for previous task tries.

Only downstream tasks with the same map index should be re-run (not sure this is easily possible or intended, but without it I don't see how restarting individual tasks would be of much use).

How to reproduce

@task
def task1(val: str):
    print("Task 2" + val)

@task
def task2(val: str):
    print("Task 2" + val)

@dag(dag_id="test")
def get_dag() -> None:
    t1 = task1.expand(val=["foo", "bar"])
    t2 = task2.expand(val=["foo", "bar"])
    t1 >> t2

get_dag()

Run once. Then re-run an individual task of task1.

Operating System

Debian GNU/Linux 12 (bookworm)

Versions of Apache Airflow Providers

No response

Deployment

Official Apache Airflow Helm Chart

Deployment details

No response

Anything else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[jroachgolf84]

I am able to reproduce this behavior. However, I'd like to address this point below:

Only downstream tasks with the same map index should be re-run (not sure this is easily possible or intended, but without it I don't see how restarting individual tasks would be of much use).

In this case, if I cleared the state of task1[0], I would expect both task2[0] and task1[0] to be re-run, not just task2[0].

[MarcusCramer91]

I am able to reproduce this behavior. However, I'd like to address this point below:

Only downstream tasks with the same map index should be re-run (not sure this is easily possible or intended, but without it I don't see how restarting individual tasks would be of much use).

In this case, if I cleared the state of task1[0], I would expect both task2[0] and task1[0] to be re-run, not just task2[0].

Agree! But I wouldn't expect task2[1] and task1[1] to be re-run, right?

[MarcusCramer91]

Any updates here? :) This is really making production use for us more cumbersome right now.

[kevinhongzl]

Hi @MarcusCramer91
I tried restarting a single mapped task by unselecting all the options (past, future, upstream, etc.) above:

and it worked as expected. No other task instances were restarted except the one I clicked on. Would you mind checking this to see if it works for you?

[matthieuauger]

Hello,

I think the initial issue is related to "Downstream" related tasks and depth-first workflows.

Use case is when you have multiple mapped tasks in a workflow, for example in a group.
My personal is use case is that I download data for each of my users with mapped tasks.

My DAG looks like this

• TaskGroup (mapped to 10 users)
  • Task 1 : create_job (mapped to 10 users with the group)
  • Task 2 : download_data (mapped to 10 users with the group)
  • Task 3 : validate_data (mapped to 10 users with the group)

Let's assume everything went well for 9 out of 10 users (index 0 to 8) but failed for last user (mapped task index 9).

I would like to be able to restart task instance create_job[9], click downstream and see download_data[9] and validate_date[9].

This is really useful because currently if a mapped task fails, you need to re-run everything.

I tried again with new version but issue still persist

[MarcusCramer91]

@kevinhongzl Exactly what @matthieuauger said :)