Implement XComArg concat()

[uranusjr]

This is useful when you want to do a thing against more than one list of things. You can sort of already do this with an extra task, but this is lazier and saves some resources both in the XCom storage and memory needed to run tasks.

[hussein-awala]

To be honest, I'm not completely convinced of the importance of this feature, but it might be useful in some case.

Could you update the documentation by describing the new XCom operation chain?

[josh-fell]

My initial reaction was the semantics could be confused with a TaskFlow-specific implementation of airflow.models.baseoperator.chain but it doesn't function the same way. This seems to mirror itertools.chain(), but because there is already a chain() method at the operator level we might need a different semantic? Maybe .combine() is another option?

Curious what others think too.

This is cool, and I would definitely use this personally.

[josh-fell]

Oddly enough, this seems useful for the discussion in #40124?

[uranusjr]

I briefly searched how other ecosystems call this and concat seems popular (behind chain). combine is more often like join but I think it still is an option in this case since the joining semantic does not make sense.

[potiuk]

I briefly searched how other ecosystems call this and concat seems popular (behind chain). combine is more often like join but I think it still is an option in this case since the joining semantic does not make sense.

Maybe append is a better name ? I agree with @hussein-awala -> we need some docs to make it discoverable, it's a rather useful feature, but having a description and example of use is pretty much a "MUST HAVE" here.

[uranusjr]

I don’t like append personally since in Python it’s a method that modifies the object in-place, not creating a new object. (Same for extend although nobody mentions it yet.) I plan to add docs after we agree on the interface.

[potiuk]

Maybe just xcom_chain() or chain_xcom() ? That would remove confusion and keep itertools relation - at the expense of somewhat redundancy.

[uranusjr]

I think I’m going to go with concat. This is used in JavaScript. Java has Stream.concat (which is more like itertools.chain, close enough). Various languages also have concat for strings. Probably among the most solid choices beside chain.

[potiuk]

SGTM

[uranusjr]

Alright, I’ve made the name change and added docs.

[eladkal]

This is useful when you want to do a thing against more than one list of things.

Very nice! BTW I think it's also interesting to be able to do thing against specific items in the XCOM rather than all of it.

[josh-fell]

Looks great!

Just curious if DAG authors would also be able to concat XComArgs from two sets of mapped tasks? If yes, then maybe worth adding to the docs too?

[uranusjr]

concat XComArgs from two sets of mapped tasks

Not sure I follow this

[josh-fell]

concat XComArgs from two sets of mapped tasks

Not sure I follow this

Related to a question I could think of DAG authors asking: "Does this work with mapped-task inputs?".

  graph LR;
      mapped_task_1-->aggregate_mapped_tasks;
      mapped_task_2-->aggregate_mapped_tasks;

Loading

meaning

aggregate_mapped_tasks(input=mapped_task_1.concat(mapped_task_2))

Looking again the "... concat function takes arbitrary positional arguments ..." statement covers this so ignore. Still looks great.

[uranusjr]

I added some text to call out chaining calls.