Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Initial Workflow SDK #857

Merged
merged 22 commits into from
Aug 11, 2023
Merged

Initial Workflow SDK #857

merged 22 commits into from
Aug 11, 2023

Conversation

bderusha
Copy link
Contributor

@bderusha bderusha commented May 11, 2023

Description

This PR covers just enough of the authoring and management APIs to allow for anyone to be able to add the rest of them without requiring much/any coordination around structure.

Design decisions of note:

  • sdk-workflows is implemented as a separate optional package (like actors)
  • Runtime registration follows conventions/patterns used by sdk-actors with WorkflowRuntime.getInstance().registerWorkflow(...)

Warning
Bug Alert:
Ran into a panic: runtime error: invalid memory address or nil pointer dereference error while using durabletask-java@v1.1.0 with dapr runtime 1.10.7. It seems to be related to some mismatch between the client and the durabletask-go@v0.1.3 that comes with dapr 1.10. When run against dapr runtime 1.11.0-rc.1 everything works nicely.

Issue reference

#839

Checklist

Please make sure you've completed the relevant tasks for this PR, out of the following list:

  • Code compiles correctly
  • Created/updated tests
  • Extended the documentation

@nyemade-uversky
Copy link

Putting this on our radars @cgillum , @DeepanshuA , @RyanLettieri

@bderusha
Copy link
Contributor Author

@cgillum , @DeepanshuA , @RyanLettieri, @nyemade-uversky This PR is almost done in terms of foundations. Would love a quick review before we add a bunch more methods if you have time this week.

@cgillum cgillum mentioned this pull request May 26, 2023
@bderusha bderusha changed the title [WIP] Initial Workflow SDK design Initial Workflow SDK May 26, 2023
Signed-off-by: Bill DeRusha <billderusha@microsoft.com>
@bderusha bderusha marked this pull request as ready for review May 26, 2023 21:11
@bderusha bderusha requested review from a team as code owners May 26, 2023 21:12
Hannah Kennedy added 2 commits May 26, 2023 17:18
Signed-off-by: Hannah Kennedy <hakenned@microsoft.com>
Signed-off-by: Hannah Kennedy <hakenned@microsoft.com>
Copy link

@cgillum cgillum left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a great start! Some comments below.

bderusha added 2 commits June 1, 2023 12:55
Signed-off-by: Bill DeRusha <444835+bderusha@users.noreply.github.com>
Signed-off-by: Bill DeRusha <444835+bderusha@users.noreply.github.com>
@bderusha
Copy link
Contributor Author

bderusha commented Jun 1, 2023

@cgillum Thanks for the feedback! I pushed up a new round of changes to address them

Copy link

@cgillum cgillum left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few additional comments/questions.

Signed-off-by: Bill DeRusha <444835+bderusha@users.noreply.github.com>
@bderusha
Copy link
Contributor Author

bderusha commented Jun 2, 2023

Thanks for the quick turnaround, @cgillum. I believe I have addressed all active comments. Let me know if you have any more suggestions.

bderusha added 2 commits June 2, 2023 11:59
Signed-off-by: Bill DeRusha <444835+bderusha@users.noreply.github.com>
Signed-off-by: Bill DeRusha <444835+bderusha@users.noreply.github.com>
Copy link

@cgillum cgillum left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a few small things/questions.

@bderusha
Copy link
Contributor Author

bderusha commented Jun 6, 2023

Thanks @cgillum Pushed up a quick set of changes based on your feedback.

Is there anyone else we need to loop into this PR for it to land? It feels like we're getting close here

@cgillum
Copy link

cgillum commented Jun 6, 2023

@bderusha this PR looks good to me. However, I don't have permissions to merge it as I'm not a maintainer for the Dapr Java SDK (my signoffs don't count either). We'll need @dapr/approvers-java-sdk and @dapr/maintainers-java-sdk to take a look.

FYI @nyemade-uversky in case we need to ping relevant folks directly.

Copy link
Contributor

@johnewart johnewart left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like a good start! I just had a few comments about dependencies that were added to POMs (looks like perhaps dependencies got added in places they don't need to be).

examples/pom.xml Outdated Show resolved Hide resolved
pom.xml Outdated Show resolved Hide resolved
Signed-off-by: Bill DeRusha <444835+bderusha@users.noreply.github.com>
@bderusha
Copy link
Contributor Author

bderusha commented Jun 7, 2023

Looks like a good start! I just had a few comments about dependencies that were added to POMs (looks like perhaps dependencies got added in places they don't need to be).

Thanks @johnewart. Good catch on those dependencies. Should be all set now.

@nyemade-uversky
Copy link

@dapr/maintainers-java-sdk would you be able to take a look at this PR?

@olitomlinson
Copy link

@cgillum I don't know if this is the time or the place, but can I ask that code examples are included that demonstrate how to unit test a Workflow from an end users perspective i.e how to mock Activity calls, timers, continueAsNew etc etc

@bderusha
Copy link
Contributor Author

@artursouza @mukundansundar Could you help trigger build again, please? I think I have resolved all the build issues now

@nyemade-uversky
Copy link

@artursouza, @mukundansundar and @bderusha would it be possible to get this PR merged this week? Looks like we're waiting on the build to pass, is that the only ting blocking this pr rn?

@bderusha
Copy link
Contributor Author

@artursouza, @mukundansundar and @bderusha would it be possible to get this PR merged this week? Looks like we're waiting on the build to pass, is that the only ting blocking this pr rn?

@nyemade-uversky There is a single failing test for Build jdk:11 sb:2.5.7 around pub/sub. I have some time carved out to investigate tomorrow, but if anyone has time or insights to share I would welcome them.

@bderusha
Copy link
Contributor Author

Thank you for rerunning the tests @mukundansundar Looks like that failure was intermittent. All tests are passing now.

FYI @nyemade-uversky

Copy link
Member

@artursouza artursouza left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is great. I have provided some feedback.

Overall, I think it should be built using more of the main SDK connectivity - otherwise it will require duplication of code to handle connection to sidecar.

Last but not least, please, consider making this a separate project under dapr-sandbox org - instructions at https://github.com/dapr-sandbox/

I do think I should continue helping in the review to keep consistent user experience for the Java SDKs since not all decisions are documented.

@bderusha
Copy link
Contributor Author

This is great. I have provided some feedback.

Overall, I think it should be built using more of the main SDK connectivity - otherwise it will require duplication of code to handle connection to sidecar.

Last but not least, please, consider making this a separate project under dapr-sandbox org - instructions at https://github.com/dapr-sandbox/

I do think I should continue helping in the review to keep consistent user experience for the Java SDKs since not all decisions are documented.

@artursouza Thank you for the feedback. The code comments all make sense. I am curious to learn more about dapr-sandbox
though. Can you help me understand more about why this feature is a good fit for a sandbox project?

The context I am thinking about here is that other SDKs have early workflow support in their main repos (python-sdk, dotnet-sdk) and of course the workflow engine itself is in the mainline dapr/dapr.

At first blush it feels like this feature is no different from other alpha APIs introduced to the dapr runtime and then supported by the SDKs. But I am admittedly just learning about the dapr-sandbox so you may have additional context that draws a clearer line to why the sandbox is a good home. Regardless of where this code lands, I am interested in learning more so that we can streamline future contributions :) Thanks!

FYI I will be on vacation most of July and slow to respond, so I am tagging others who might be interested in this while I'm out.
@nyemade-uversky @cgillum @NarmathaBala

@artursouza
Copy link
Member

This is great. I have provided some feedback.
Overall, I think it should be built using more of the main SDK connectivity - otherwise it will require duplication of code to handle connection to sidecar.
Last but not least, please, consider making this a separate project under dapr-sandbox org - instructions at https://github.com/dapr-sandbox/
I do think I should continue helping in the review to keep consistent user experience for the Java SDKs since not all decisions are documented.

@artursouza Thank you for the feedback. The code comments all make sense. I am curious to learn more about dapr-sandbox though. Can you help me understand more about why this feature is a good fit for a sandbox project?

The context I am thinking about here is that other SDKs have early workflow support in their main repos (python-sdk, dotnet-sdk) and of course the workflow engine itself is in the mainline dapr/dapr.

At first blush it feels like this feature is no different from other alpha APIs introduced to the dapr runtime and then supported by the SDKs. But I am admittedly just learning about the dapr-sandbox so you may have additional context that draws a clearer line to why the sandbox is a good home. Regardless of where this code lands, I am interested in learning more so that we can streamline future contributions :) Thanks!

FYI I will be on vacation most of July and slow to respond, so I am tagging others who might be interested in this while I'm out. @nyemade-uversky @cgillum @NarmathaBala

Making this a dapr-sandbox repo would need to be consistent with other SDKs, so all of them would move the WF part to it. In this case, there is a different since it is for composing workflow and not simply an Alpha API. The wf invocation is a simple Alpha API and it should still be here.

The main driving factor for this is that this is at a different maturity level and a new artifact. The fact that it is versioned differently from the other artifacts of the SDK says it all. That is why I am leaning torwards this being a separate SDK. For this to be merged in this repo, we would also need to update all the release scripts to handle specific versioning for this artifact. See https://github.com/dapr/java-sdk/wiki/Release-Process

@artursouza
Copy link
Member

/cc @brunoborges do you mind taking a look at this PR too? It is from Msft.

@NarmathaBala
Copy link

NarmathaBala commented Jul 11, 2023

This is great. I have provided some feedback.
Overall, I think it should be built using more of the main SDK connectivity - otherwise it will require duplication of code to handle connection to sidecar.
Last but not least, please, consider making this a separate project under dapr-sandbox org - instructions at https://github.com/dapr-sandbox/
I do think I should continue helping in the review to keep consistent user experience for the Java SDKs since not all decisions are documented.

@artursouza Thank you for the feedback. The code comments all make sense. I am curious to learn more about dapr-sandbox though. Can you help me understand more about why this feature is a good fit for a sandbox project?
The context I am thinking about here is that other SDKs have early workflow support in their main repos (python-sdk, dotnet-sdk) and of course the workflow engine itself is in the mainline dapr/dapr.
At first blush it feels like this feature is no different from other alpha APIs introduced to the dapr runtime and then supported by the SDKs. But I am admittedly just learning about the dapr-sandbox so you may have additional context that draws a clearer line to why the sandbox is a good home. Regardless of where this code lands, I am interested in learning more so that we can streamline future contributions :) Thanks!
FYI I will be on vacation most of July and slow to respond, so I am tagging others who might be interested in this while I'm out. @nyemade-uversky @cgillum @NarmathaBala

Making this a dapr-sandbox repo would need to be consistent with other SDKs, so all of them would move the WF part to it. In this case, there is a different since it is for composing workflow and not simply an Alpha API. The wf invocation is a simple Alpha API and it should still be here.

The main driving factor for this is that this is at a different maturity level and a new artifact. The fact that it is versioned differently from the other artifacts of the SDK says it all. That is why I am leaning torwards this being a separate SDK. For this to be merged in this repo, we would also need to update all the release scripts to handle specific versioning for this artifact. See https://github.com/dapr/java-sdk/wiki/Release-Process

@artursouza - @nyemade-uversky shared with us re: letting this PR merge into the Dapr repo. We have Bill out of office for the month of July. Considering the feedback suggested will involved some deep changes, we (Bill and I) agreed that it was best for Bill to pick this back up in August. I hope that works for the team.

@brunoborges
Copy link
Contributor

/cc @brunoborges do you mind taking a look at this PR too? It is from Msft.

Hey Artur, sorry but I'm afraid I don't have the necessary knowledge of the Dapr SDK, nor the time, to provide meaningful feedback here.

@craprivatebeta
Copy link

This is great. I have provided some feedback.
Overall, I think it should be built using more of the main SDK connectivity - otherwise it will require duplication of code to handle connection to sidecar.
Last but not least, please, consider making this a separate project under dapr-sandbox org - instructions at https://github.com/dapr-sandbox/
I do think I should continue helping in the review to keep consistent user experience for the Java SDKs since not all decisions are documented.

@artursouza Thank you for the feedback. The code comments all make sense. I am curious to learn more about dapr-sandbox though. Can you help me understand more about why this feature is a good fit for a sandbox project?
The context I am thinking about here is that other SDKs have early workflow support in their main repos (python-sdk, dotnet-sdk) and of course the workflow engine itself is in the mainline dapr/dapr.
At first blush it feels like this feature is no different from other alpha APIs introduced to the dapr runtime and then supported by the SDKs. But I am admittedly just learning about the dapr-sandbox so you may have additional context that draws a clearer line to why the sandbox is a good home. Regardless of where this code lands, I am interested in learning more so that we can streamline future contributions :) Thanks!
FYI I will be on vacation most of July and slow to respond, so I am tagging others who might be interested in this while I'm out. @nyemade-uversky @cgillum @NarmathaBala

Making this a dapr-sandbox repo would need to be consistent with other SDKs, so all of them would move the WF part to it. In this case, there is a different since it is for composing workflow and not simply an Alpha API. The wf invocation is a simple Alpha API and it should still be here.
The main driving factor for this is that this is at a different maturity level and a new artifact. The fact that it is versioned differently from the other artifacts of the SDK says it all. That is why I am leaning torwards this being a separate SDK. For this to be merged in this repo, we would also need to update all the release scripts to handle specific versioning for this artifact. See https://github.com/dapr/java-sdk/wiki/Release-Process

@artursouza - @nyemade-uversky shared with us re: letting this PR merge into the Dapr repo. We have Bill out of office for the month of July. Considering the feedback suggested will involved some deep changes, we (Bill and I) agreed that it was best for Bill to pick this back up in August. I hope that works for the team.

That works for me. Yes, the PR can be added here.

@mukundansundar
Copy link
Contributor

@NarmathaBala @bderusha
Do we have an ETA for updates in this PR? Thanks.

@bderusha
Copy link
Contributor Author

bderusha commented Aug 2, 2023

@NarmathaBala @bderusha Do we have an ETA for updates in this PR? Thanks.

@mukundansundar I just got back from some time off and am planning to make progress on the PR this week.

Signed-off-by: Bill DeRusha <444835+bderusha@users.noreply.github.com>
@bderusha
Copy link
Contributor Author

bderusha commented Aug 3, 2023

@mukundansundar @artursouza I believe I have addressed all outstanding comments. Please take a look when you get a chance.

Signed-off-by: Bill DeRusha <444835+bderusha@users.noreply.github.com>
Copy link
Member

@skyao skyao left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it is a gread PR!

I put some very detail comments about the code implementation.

Let's make it perfect.

Signed-off-by: Bill DeRusha <444835+bderusha@users.noreply.github.com>
examples/pom.xml Outdated Show resolved Hide resolved
sdk-workflows/pom.xml Outdated Show resolved Hide resolved
}

@Override
public TaskOrchestration create() {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If all the API is blocking for DurableTask, then ProjectReactor does not add any value and makes things more complicated by giving the illusion of async model. Does DurableTask have an async API or is it all blocking?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@artursouza DurableTask does not expose any async methods in the client or runtime. The only async API is for some of the Orchestration and Activity Context classes which return custom Task classes which wrap java.util.concurrent.CompletableFuture objects.

I am new to ProjectReactor, so I don't have a good sense for effort VS value of trying to convert those context methods to mono. What are your thoughts?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

DurableTask does not expose any async methods in the client or runtime.

This is so bad! So since DurableTask is blocking, then we use reactor in java sdk won't bring any value but make it complex.

It seems that we need to give up Mono and use normal blocking call.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree. We should keep it blocking until the DurableTask SDK addresses this: microsoft/durabletask-java#160

Signed-off-by: Bill DeRusha <444835+bderusha@users.noreply.github.com>
Signed-off-by: Bill DeRusha <444835+bderusha@users.noreply.github.com>
Signed-off-by: Bill DeRusha <444835+bderusha@users.noreply.github.com>
@codecov
Copy link

codecov bot commented Aug 10, 2023

Codecov Report

Merging #857 (904e41e) into master (9dc842f) will increase coverage by 0.18%.
The diff coverage is 79.61%.

@@             Coverage Diff              @@
##             master     #857      +/-   ##
============================================
+ Coverage     77.31%   77.49%   +0.18%     
- Complexity     1268     1304      +36     
============================================
  Files           117      123       +6     
  Lines          3927     4013      +86     
  Branches        468      472       +4     
============================================
+ Hits           3036     3110      +74     
- Misses          652      662      +10     
- Partials        239      241       +2     
Files Changed Coverage Δ
...dapr/workflows/runtime/WorkflowRuntimeBuilder.java 45.45% <45.45%> (ø)
...io/dapr/workflows/runtime/OrchestratorWrapper.java 64.70% <64.70%> (ø)
...ava/io/dapr/workflows/runtime/WorkflowRuntime.java 69.23% <69.23%> (ø)
sdk/src/main/java/io/dapr/utils/NetworkUtils.java 88.46% <81.25%> (-11.54%) ⬇️
...a/io/dapr/workflows/client/DaprWorkflowClient.java 91.30% <91.30%> (ø)
...ava/io/dapr/workflows/DaprWorkflowContextImpl.java 100.00% <100.00%> (ø)
...lows/src/main/java/io/dapr/workflows/Workflow.java 100.00% <100.00%> (ø)
...rc/main/java/io/dapr/client/DaprClientBuilder.java 71.42% <100.00%> (+6.72%) ⬆️

... and 1 file with indirect coverage changes

@artursouza artursouza merged commit 6d65991 into dapr:master Aug 11, 2023
@mukundansundar mukundansundar added this to the v1.10 milestone Sep 20, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.