Add Worker Versioning doc #1533
Conversation
|
|
||
| The primary use case that this feature makes much simpler is the deployment of incompatible changes to short-lived workflows. On Task Queues making use of this feature, the workflow starter does not need to know that there are new versions being introduced. New [Workflow Executions](/concepts/what-is-a-workflow-execution) will run using the new code in the newly deployed workers, and old (but still open) workflows will only be processed by workers with an appropriate version. Old workers may be decommissioned once there are no longer any open workflows using their version. | ||
|
|
||
| For example if you have a workflow which never takes longer than a day to execute, a great strategy is to simply always assign a new Build ID to every new build of your worker and add it to the graph as the new default version. Because you know your workflow never takes more than a day to complete, you know that you'll never have to leave older workers running for more than a day once you've deployed the new version (assuming availability). |
There was a problem hiding this comment.
question I have after reading this: how do I prune those old version from the graph once I decommission the workers?
There was a problem hiding this comment.
You don't! They're expected to just sit around until they eventually page out, if ever. I'll add a bit on that.
|
|
||
| Whether you use `tctl` or an SDK, updating the version graph feels the same. | ||
|
|
||
| You'll specify the task queue you're targeting, the Build ID you're adding (or moving), whether or not it is to become the new default version, and some existing version it should be considered compatible with, if any. |
There was a problem hiding this comment.
here and elsewhere, both "Build ID" and "version" are used and I can't really tell what the difference is. are they synonyms?
I sorta get that a Build ID is a property of a worker, and a version is a thing in the graph, but since there's a 1:1 mapping between them, do we want to use different terms?
There was a problem hiding this comment.
They're effectively synonyms for the user. I think there is some value in using both terms here, distinguished like you said, since it helps contextualize when we're talking about something inside vs outside of the graph... but I'm open to suggestion.
Maybe I can just add a little bit about the two terms near the top?
| to the front (along with its entire compatible branch), as long as you have targeted the most recent | ||
| version in that compatible branch. | ||
|
|
||
| You can't delete versions. This is to help avoid a situation where workflows accidentally become stuck with |
There was a problem hiding this comment.
maybe comment on max size and when things fall out here?
Sushisource
force-pushed
the
worker-versioning
branch
from
d265c0f to
2430145
Compare
|
[note to myself] |
|
I converted this PR to draft to ensure that it doesn't accidentally get merged too early, and I've started a terminology-and-style review. |
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
Sushisource
force-pushed
the
worker-versioning
branch
from
6c92c86 to
a9c8caa
Compare
| receive Tasks as usual. Workers who have opted in don't poll the Task Queue because they can't | ||
| receive Tasks for Workflows that have no associated version. | ||
|
|
||
| :::note Versions don't need to be semver! |
There was a problem hiding this comment.
nit: it's not semver if it's only 2 parts.
There was a problem hiding this comment.
nit on your nit: 2-part version numbers can still be in the general class of semantic versions, even if they're not "semver".
| :::note Versions don't need to be semver! | |
| :::note Versions don't need to follow semver or any other semantic versioning standard! |
| - Add a version to an existing set that is compatible with an existing version. | ||
| - Optionally making it the default for that set | ||
| - Optionally making that set the overall-default set. | ||
| - Promote a version within an existing set to become the default for that set. |
There was a problem hiding this comment.
nit: this doesn't have an example above
| configurable on Temporal Server via the `limit.versionBuildIDPerQueue` dynamic config property. | ||
| Operations to add new Build IDs to the sets will fail if the limit would be exceeded. | ||
|
|
||
| There is also a limit on the number of sets, which defaults to 10. This limit is configurable via |
There was a problem hiding this comment.
Not a docs concern but 10 might not be enough as the default assuming daily deployment and an arbitrary long running workflow.
There was a problem hiding this comment.
I don't care what it is but 10 is what David / Yimin thought was plausible for overhead. Whatever it ends up being we can just adjust this later.
assuming daily deployment and an arbitrary long running workflow
I mean, if it runs arbitrarily long, no limit is sufficient so it's a bit of a moot point.
| You can use this technique with longer-lived Workflows as well. Just be aware that you might need to | ||
| keep multiple versions of Workers running while open Workflows complete. | ||
|
|
||
| You can also use the feature to prevent a buggy codepath from being taken on currently open |
There was a problem hiding this comment.
this is correct, but sort of a weird way to introduce the feature. the normal compatible changes I might want to make are things like adjusting timeouts, changing activities only, changing logs/metrics, etc. where you have a new build, but you wouldn't really say there's a buggy codepath, just that you have new code you want to use instead
There was a problem hiding this comment.
Good point, mentioned that first.
|
|
||
| Continue-as-new from a Workflow on a versioned Task Queue will, by default, start the continued | ||
| Workflow on the same compatible set as the original Workflow. Continuing-as-new onto a different | ||
| task queue will not assign any particular version. |
There was a problem hiding this comment.
are we going to add the feature to c-a-n onto the latest (on the same queue)?
There was a problem hiding this comment.
Yes... but I wanted to avoid having to specify exactly how a user does that b/c we haven't added it to the command yet and it'll probably look slightly different in each SDK. I could say something generic like "Look at your SDK's reference docs for c-a-n" or something
There was a problem hiding this comment.
I think we're going to need this feature for child workflows as well.
This default behavior where everything stays in the same compatible version will make it hard to retire old workers in some scenarios.
There was a problem hiding this comment.
Left it generic for now
| receive Tasks as usual. Workers who have opted in don't poll the Task Queue because they can't | ||
| receive Tasks for Workflows that have no associated version. | ||
|
|
||
| :::note Versions don't need to be semver! |
There was a problem hiding this comment.
nit on your nit: 2-part version numbers can still be in the general class of semantic versions, even if they're not "semver".
| :::note Versions don't need to be semver! | |
| :::note Versions don't need to follow semver or any other semantic versioning standard! |
Sushisource
force-pushed
the
worker-versioning
branch
from
780396a to
10f1b6a
Compare
Sushisource
force-pushed
the
worker-versioning
branch
from
4c2c41e to
183529d
Compare
|
Created a PR off this branch to include it in the docs: |
* Make style changes to 1533 * Add feedback
What does this PR do?
Adds conceptual documenatation on the new Worker Versioning feature
Notes to reviewers
Closes #1387