Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add support for mapping over remote launch plans #2761

Merged
merged 9 commits into from
Oct 10, 2024

Conversation

pvditt
Copy link
Contributor

@pvditt pvditt commented Sep 19, 2024

Tracking issue

fixes: https://linear.app/unionai/issue/COR-1686/support-mapping-over-flytelaunchplan-remote

Why are the changes needed?

Mapping over remote launch plans doesn't work

What changes were proposed in this pull request?

  • add support for mapping over remote launch plans by using create_and_link_node_from_remote
  • updated models TypedInterface to transform variables of a type or variables of a collection of type

How was this patch tested?

Setup process

remote = UnionRemote(config=Config.auto(), default_project="flytesnacks", default_domain="development")
lp1 = remote.fetch_launch_plan(name="advanced_composition.subworkflow.regression_line_wf",
                               version="v0.3.288")


@workflow
def test_wf1_remote():

    val = [1, 2, 3]
    x = [[-3, 0, 3], [-8, 2, 4], [7, 3, 1]]
    y = [[7, 4, -2], [-2, 4, 7], [3, 6, 4]]
    map_task(lp1)(val=val, x=x, y=y)

Screenshots

Check all the applicable boxes

  • I updated the documentation accordingly.
  • All new and existing tests passed.
  • All commits are signed-off.

Related PRs

Docs link

Signed-off-by: Paul Dittamo <pvdittamo@gmail.com>
Signed-off-by: Paul Dittamo <pvdittamo@gmail.com>
Signed-off-by: Paul Dittamo <pvdittamo@gmail.com>
Signed-off-by: Paul Dittamo <pvdittamo@gmail.com>
Copy link
Collaborator

@eapolinario eapolinario left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for fixing this. Let's try to keep the interface clean by not using Any at this point, unless we have a very good reason.

@@ -27,7 +31,7 @@
class ArrayNode:
def __init__(
self,
target: LaunchPlan,
target: Any,
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we use Union to describe the types? I'd rather keep ArrayNode well-typed.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yup - getting an update for this. Was working around a circular imports error. Can use if TYPE_CHECKING

Signed-off-by: Paul Dittamo <pvdittamo@gmail.com>
@wild-endeavor
Copy link
Contributor

I think there's some circular dependency that needs to happen but I think maybe what's here right now is more than necessary. let's chat about this some time today or tomorrow?

Signed-off-by: Paul Dittamo <pvdittamo@gmail.com>
Signed-off-by: Paul Dittamo <pvdittamo@gmail.com>
Signed-off-by: Paul Dittamo <pvdittamo@gmail.com>
@pvditt pvditt merged commit cd8216a into master Oct 10, 2024
104 checks passed
otarabai pushed a commit to otarabai/flytekit that referenced this pull request Oct 15, 2024
* add support for mapping over remote launch plans

Signed-off-by: Paul Dittamo <pvdittamo@gmail.com>

* add unit tests

Signed-off-by: Paul Dittamo <pvdittamo@gmail.com>

* lint

Signed-off-by: Paul Dittamo <pvdittamo@gmail.com>

* update var name

Signed-off-by: Paul Dittamo <pvdittamo@gmail.com>

* utilize TYPE_CHECKING to check for FlyteLaunchPlan

Signed-off-by: Paul Dittamo <pvdittamo@gmail.com>

* utilize adding remote node function

Signed-off-by: Paul Dittamo <pvdittamo@gmail.com>

* add unit tests

Signed-off-by: Paul Dittamo <pvdittamo@gmail.com>

* revert changes to local execute

Signed-off-by: Paul Dittamo <pvdittamo@gmail.com>

---------

Signed-off-by: Paul Dittamo <pvdittamo@gmail.com>
kumare3 pushed a commit that referenced this pull request Nov 8, 2024
* add support for mapping over remote launch plans

Signed-off-by: Paul Dittamo <pvdittamo@gmail.com>

* add unit tests

Signed-off-by: Paul Dittamo <pvdittamo@gmail.com>

* lint

Signed-off-by: Paul Dittamo <pvdittamo@gmail.com>

* update var name

Signed-off-by: Paul Dittamo <pvdittamo@gmail.com>

* utilize TYPE_CHECKING to check for FlyteLaunchPlan

Signed-off-by: Paul Dittamo <pvdittamo@gmail.com>

* utilize adding remote node function

Signed-off-by: Paul Dittamo <pvdittamo@gmail.com>

* add unit tests

Signed-off-by: Paul Dittamo <pvdittamo@gmail.com>

* revert changes to local execute

Signed-off-by: Paul Dittamo <pvdittamo@gmail.com>

---------

Signed-off-by: Paul Dittamo <pvdittamo@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants