-
Notifications
You must be signed in to change notification settings - Fork 948
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add System Labels incl. user and correlationId (an execution reference field) #2059
Comments
In the integration world (EAI/ESB/SOA) we refere at this by the name of correlation identifier (correlationId) as it allows to correlate multiple messages (here executions) together. |
ProblemThe user's main problem is the heavy reliance on Use cases
The reference key can be described as "a label identifying particular processed input data". Such key is identified prior feeding the data to Kestra. It is often a "business-relevant ID" contained in the data or a concatenated value from multiple sources. It could be configured manually by the user in the flow definition. Possible SolutionsThe idea of this reference field is to create a surrogate key comprised of multiple fields for easier filtering of relevant operational data to answer questions such as "What are top 10 failing executions in the past 1h containing label env:prod grouped by label x and flowId => Which PROD flows of a "type" specified by the given label x are most troublesome?" |
Since, this is been pushed back in priorities ... anybody have an example work around they use to achieve similar for the time been? |
@damienkilgannon you can leverage the label task and filter executions by that label, e.g.: - id: label
type: io.kestra.plugin.core.execution.Labels
labels:
url: "{{ outputs.some_task.value }}" |
Discussed with Ludo, and we will solve it by introducing System Labels and one of them will be System Labels are all labels that have keys starting with The If some users would like to set a custom correlation ID, it's possible by defining a custom id: flow
namespace: company.team
labels:
system_correlationId: myCustomKey
tasks:
- id: hello
type: io.kestra.plugin.core.log.Log
message: Hello {{ labels.system_correlationId }} The System Labels listed below will be automatically added to relevant executions and wil be hidden in the UI:
IMPORTANT NOTE: those should be read-only (not possible to overwrite those). What about custom labels that users may want to hide in the UI? As an extra scope (if we have time), we can add kestra:
internal-labels:
prefixes:
- internal_
- admin_ |
TBD we likely will not allow user's custom system labels |
Two labels have been added as system labels: Only |
see discussion here
We could add a simple reference on the execution, each subflow, trigger that start from that flow will copy the reference, and we need to have this field visible on the list and to allow search on that one.
The text was updated successfully, but these errors were encountered: