-
Notifications
You must be signed in to change notification settings - Fork 25.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Transform] Log and audit a warning if all the group-by fields are runtime fields #68050
[Transform] Log and audit a warning if all the group-by fields are runtime fields #68050
Conversation
ed93fc1
to
2fe34a8
Compare
Pinging @elastic/ml-core (:ml/Transform) |
@@ -77,6 +79,11 @@ public void validateConfig(ActionListener<Boolean> listener) { | |||
listener.onResponse(true); | |||
} | |||
|
|||
@Override | |||
public List<String> getPerformanceCriticalFields() { | |||
return config.getGroupConfig().getGroups().values().stream().map(SingleGroupSource::getField).collect(toList()); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I am not sure all of these would be performance critical. it almost seems like they would be performance critical IF they didn't support missing buckets right?
If the group config supported missing buckets, we don't get the benefit of filtering for changes or something? I remember there was a trade off.
Right @hendrikmuhs
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If the group config supported missing buckets, we don't get the benefit of filtering for changes or something? I remember there was a trade off.
missing bucket runs a bit slower in composite aggs, but I think we can ignore that, its like 10-20% improvement. Problematic are group_by with scripts, but we already have a warning for that: We warn if change detection is completely disabled, which means we do a full re-run on every checkpoint.
Can you also add a check if the timestamp field configured for |
2fe34a8
to
64adc54
Compare
Done. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
This PR adds logging and auditing for the case when all the fields that transform uses as group-by (in
pivot
) or unique key (inlatest
) are search runtime fields.This PR is marked as
>non-issue
as it enhances an unreleased feature.Relates #65147