workflows: update durabletask dependency

[famarting]

Description

Update durabletask dependency to use new dapr fork

Also implements support for purge API :#711

Implements support for reuse id policy and set custom status: #739

Adds support for the recursive parameter on terminate workflow: #774

Issue reference

We strive to have all PR being opened based on an issue, where the problem or feature have been discussed prior to implementation.

Please reference the issue this PR will close: #[issue number]

Checklist

Please make sure you've completed the relevant tasks for this PR, out of the following list:

• Code compiles correctly
• Created/updated tests
• Extended the documentation

[elena-kolevska]

Some comments. Please run tox -e ruff and tox -e flake8 to fix linter errors.

We'll also need to update the grpc library to >= 1.67.0, for compatibility with durabletask.

And finally, we'll need to update the tests in test_dapr_workflow_context.py to cover set-custom_status. Something like

class FakeOrchestrationContext:
    def __init__(self):
        self.instance_id = mock_instance_id
        self.custom_status = None. # <- adding this
    def set_custom_status(self, custom_status):
        self.custom_status = custom_status
        
...

class DaprWorkflowContextTest(unittest.TestCase):
...
def test_workflow_context_functions(self):
...
            dapr_wf_ctx.set_custom_status(mock_status)
            assert fakeContext.custom_status == mock_status            

[elena-kolevska]

@berndverst any objections on bumping the grpc library version?
This has been updated in the durable task fork and it's a dependency here.

[elena-kolevska]

Version 1.37.0 is very old, it's from April 2021. I think we're safe to update.

[berndverst]

This does not mean it would use 1.37 but that 1.37 would be possible to be used. Pip by default will always install the latest version matching the constraint unless an older version is cached (that behavior can be changed by the way). But if another library a customer is using required grpcio==1.47.0 for example it would be possible for that specific version to be used.

In Python two versions cannot coexist unlike in other languages. So it's really really important not to force the minimum version to something recent unless this is absolutely necessary.

To provide a specific example, at this time the tensorflow development test container pins grpcio 1.42.0 and this is common for folks using Python 3.9 with compatible tensorflow packages for example. A lot of people use libraries in production that pin specific grpcio versions and they want to use Dapr for added functionality.

Pinning a recent grpcio as the minimum version prevents a lot of compatibility with other packages.

For this reason I must strongly disagree with pinning 1.67.0 as the minimum version. You could argue it is the fault of those other packages for constraining the maximum version of grpcio - and that would be true, but that is a pedantic viewpoint. We need to ensure a good experience for customers.

Note - I have been here before: I made this very mistake as a Dapr maintainer - then several ML users complained about incompatibilities between Dapr and their ML package and a hotfix release had to be made.

Now from a security perspective it is wrong to say that a low minimum version (which includes vulnerable package versions) is a security issue in Dapr. Python resolves dependency versions at install time.

Installing dapr via pip install dapr --no-cache-dir would ensure to install the latest version of all dependencies from pypi. Folks who do not want this can manually add specific dependencies to their own requirements.txt to control the transitive dependency versions.

[berndverst]

Suggested change

	grpcio >= 1.67.0
	grpcio >= 1.42.0

Short of auditing the Dapr community to figure out what libraries everyone uses and what the dependencies are pinning... here is a quick thing I identified. This is evidence of at least 1.42.0 being used for something.

https://github.com/tensorflow/tensorflow/blob/8fd1cd0826ef2b12c00540f3acc5a6e3c249f5b2/ci/official/containers/linux_arm64/devel.usertools/test.requirements.txt#L2

If as a project we want to say that we do not care about ML users - or that ML users must be on the latest official grpcio packages (highly unlikely because ML package authors are very slow to update packages) then we could consider increasing the minimum version of grpcio.

[elena-kolevska]

Unfortunately, this version is imposed by the protos used in the durable task library.

@cgillum and @famarting, can we regenerate the protos with an older grpcio version? I suggest we pin it to a version we agree on in the requirements file. I'll check for any CVEs and share here, so we can decide which version would make sense.

[berndverst]

Yes let's please do that -- you should not use the latest grpcio-tools to generate the protos -- instead use an older version for wider compatibility with more versions of grpcio.

Is this only an issue in the "fork" or also the upstream version?

[elena-kolevska]

It's coming from the upstream, here's the commit: microsoft/durabletask-python#31.

[famarting]

if we have to update the protos, I would suggest doing it here https://github.com/dapr/durabletask-protobuf , since dapr is already based on that one

[elena-kolevska]

Here's the PR: dapr/durabletask-python#6
Once that's merged I'll publish a new version and we can update this PR.

[codecov]

Codecov Report

Attention: Patch coverage is 96.77419% with 1 line in your changes missing coverage. Please review.

Project coverage is 86.17%. Comparing base (bffb749) to head (faebc7a).
Report is 20 commits behind head on main.

Files with missing lines	Patch %	Lines
...ext-workflow/dapr/ext/workflow/workflow_context.py	66.66%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #757      +/-   ##
==========================================
- Coverage   86.63%   86.17%   -0.46%     
==========================================
  Files          84       89       +5     
  Lines        4473     4999     +526     
==========================================
+ Hits         3875     4308     +433     
- Misses        598      691      +93     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[elena-kolevska]

LGTM, pending @berndverst 's approval on the grpcio version bump

The base branch was changed.

[berndverst]

Suggested change

	grpcio >= 1.67.0
	grpcio >= 1.42.0

Short of auditing the Dapr community to figure out what libraries everyone uses and what the dependencies are pinning... here is a quick thing I identified. This is evidence of at least 1.42.0 being used for something.

https://github.com/tensorflow/tensorflow/blob/8fd1cd0826ef2b12c00540f3acc5a6e3c249f5b2/ci/official/containers/linux_arm64/devel.usertools/test.requirements.txt#L2

If as a project we want to say that we do not care about ML users - or that ML users must be on the latest official grpcio packages (highly unlikely because ML package authors are very slow to update packages) then we could consider increasing the minimum version of grpcio.

[berndverst]

Careful. Changing this version is prone to huge changes to the resulting compiled protos, which therefore requires new versions of grpcio.

New versions of grpcio can load compiled protos from older grpcio-tools. But older grpcio versions cannot load compiled protos from the newest grpcio-tools.

I strongly recommend carefully testing this compatibility matrix - it is best not to change this version here (which is only used once as part of the release process -- it is a build dependency for maintainers only).

Unless you have a reason to change this, don't.