Ensure backwards compat in upperSchemaVersion handling

[davidmrdavid]

Currently, Python SDK versions 1.1.0 to 1.1.3 only handle upperSchemaVersion values of 0 (V1) and 1 (V2). This is a problem, because upperSchemaVersion was supposed to support arbitrary integers to denote replay protocol versions and we just implemented upperSchemaVersion.V3 in the Extension.

Technically speaking, the error is due to a bug in the Python SDK. However, due to the auto-upgrading nature of Bundles, I think it's important to consider how to avoid impacting existing SDKs when we're making a minor Extension release. This PR proposes one way to do that.

Essentially, we're "deprecating" the upperSchemaVersion field sent to the SDKs and instead utilizing a new field newUpperSchemaVersion to include protocol versions larger than V2. This will give the Python SDK the possibility of supporting new replay schemas without prevent older SDK releases from working with Extension Bundles.

Since this change is so minimal, I think we could include this change without needing to re-run all the validation work for Extension release 2.7.0. The only affected path should be the OOProc smoke tests, which can be executed easily. Alternatively, we can always make a 2.7.1 hotpatch release with this change after 2.7.0 is out.

Issue describing the changes in this PR

Related to # Azure/azure-functions-durable-python#368 and #2138

Pull request checklist

• My changes do not require documentation changes
  • Otherwise: Documentation PR is ready to merge and referenced in pending_docs.md
• My changes should not be added to the release notes for the next release
  • Otherwise: I've added my notes to release_notes.md
• My changes do not need to be backported to a previous version
  • Otherwise: Backport tracked by issue/PR #issue_or_pr
• I have added all required tests (Unit tests, E2E tests)
• My changes do not require any extra work to be leveraged by OutOfProc SDKs
  • Otherwise: That work is being tracked here: #issue_or_pr_in_each_sdk
• My changes do not change the version of the WebJobs.Extensions.DurableTask package
  • Otherwise: major or minor version updates are reflected in /src/Worker.Extensions.DurableTask/AssemblyInfo.cs

[davidmrdavid]

I have validated that the JS and Python smoke tests pass with these changes.

[cgillum]

What’s the impact to Durable JS when we reduce upperSchemaVersion back to 2? Do any newly introduced features stop working? Do we need a follow-up work item to restore any functionality that looks for upperSchemaVersion 3?

[cgillum]

Got my answer, which is that schema version 3 isn’t used yet.

[davidmrdavid]

@hossam-nasr: let's sync on this later

[hossam-nasr]

@cgillum @davidmrdavid Replay schema v3 was only used for long timers (Azure/azure-functions-durable-js#340) and callHttp polling (Azure/azure-functions-durable-js#346), which afaik haven't been released it. I've created this issue to track supporting this new flag though: Azure/azure-functions-durable-js#350