[core][compiled-graphs] Revisit the definition of upstream / downstream definitions in DAG and RayCG

[kevin85421]

Description

Currently, the DAG assumes that all DAGNodes in args, kwargs, and other_args_to_resolve are upstream nodes (code). However, Ray Compiled Graphs builds the upstream/downstream relationship based only on args.

If other_args_to_resolve contains DAGNodes which don't belong to args, traditional DAG and RayCG will have different understanding about the DAG graph.

• Solution
  • Currently, we avoid persisting DAGNodes inside other_args_to_resolve if they don't belong to args (example: [core][compiled-graphs] Don't persist input_nodes in _CollectiveOperation to avoid wrong understanding about DAGs #48463) because it is possible to cause issues if we change some assumptions in RayCG in the future (example: [core][compiled-graph] Support a list of DAGNodes as an arg #48045 (comment)).
  • If we observe the inconsistency causing more and more issues, we can consider defining a new field which will not be serialized in dag_node.py, so all DAGNodes inside the new field will not be considered as upstream nodes.

Use case

No response