-
Notifications
You must be signed in to change notification settings - Fork 2.7k
Rework graph.py #8642
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Rework graph.py #8642
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Code is clean; tested with workflows involving multiple iterators and collections and produced expected output. I have a few minor code comments regarding handling of edge conditions.
Cleaned up code Co-authored-by: Lincoln Stein <lincoln.stein@gmail.com>
lstein
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
All issues previously rasied have been resolved.
Summary
This new
graph.pykeeps the same DAG model and validation but restructures runtime scheduling and readability: ready queues moved from publicdequefields to internal, non-serialized structures grouped by node class, with a configurable class priority and optional batch cap;indegreeis now seeded as unmet inputs (fixing iterator stalls) and children are enqueued via a single_enqueue_if_ready()helper;next()drains the active class queue, reseeds when empty, and_prepare()still expands iterators but with clearer predicates; minor fixes include catchingValueErrorindelete_edge, replacing magic port strings withITEM_FIELD/COLLECTION_FIELD, and renaming helper methods for clarity. Functionality and performance are preserved aside from the scheduling policy change and the indegree correctness fix.Additionally, there is a new
README.mdfile that explains a bit about the classes ingraph.pyso future developers can get a head start.Related Issues / Discussions
All of this was an effort to understand the execution engine better to support different types in the future - if, loops, and the like.
QA Instructions
I've run a number of workflows with these changes and all seems good.
Merge Plan
Checklist
What's Newcopy (if doing a release after this PR)