-
Notifications
You must be signed in to change notification settings - Fork 3k
Core: Introduce CompositeMetricsReporter #7337
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Core: Introduce CompositeMetricsReporter #7337
Conversation
e08d9b6 to
dbf7181
Compare
| return reporter; | ||
| } | ||
|
|
||
| static class CompositeMetricsReporter implements MetricsReporter { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@rdblue as we discussed I've made this an inner class. However, we could consider moving the class out of CatalogUtil and replace https://github.com/apache/iceberg/blob/master/core/src/main/java/org/apache/iceberg/TableScanContext.java#L359-L366 with a CompositeMetricsReporter.
For cases where we'd like to use multiple reporters during scans
TableScan tableScan = table
.newScan()
.metricsReporter(...)
.metricsReporter(...)
.metricsReporter(...);
we would internally instantiate a CompositeMetricsReporter in TableScanContext#reportWith(..) and then have TableScanContext#metricsReporter() return MetricsReporter rather than Collection<MetricsReporter>.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
dbf7181 to
7d276c5
Compare
|
Why do we need to support multiple reporters? I don't think we need to change the behavior. |
7d276c5 to
af1440f
Compare
13d7859 to
74c3615
Compare
core/src/main/java/org/apache/iceberg/metrics/CompositeMetricsReporter.java
Outdated
Show resolved
Hide resolved
core/src/main/java/org/apache/iceberg/metrics/CompositeMetricsReporter.java
Outdated
Show resolved
Hide resolved
core/src/main/java/org/apache/iceberg/metrics/CompositeMetricsReporter.java
Outdated
Show resolved
Hide resolved
core/src/main/java/org/apache/iceberg/metrics/CompositeMetricsReporter.java
Outdated
Show resolved
Hide resolved
| reporter -> { | ||
| try { | ||
| reporter.report(report); | ||
| } catch (Exception e) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this should be RuntimeException since report doesn't throw a checked exception.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
the reason I'm catching an Exception here is that I think we generally don't want to interfere with actual scans or commits when anything on the reporting side fails. Generally speaking, every reporter should make sure that it doesn't interefere with the scan/commit path when reporting and something fails but I'm not sure we can enforce that.
I'm a bit worried that if we only catch RuntimeException and some reporter misbehaves in an unexpected way, we'd have to go back and fix either the misbehaving reporter or this catch clause.
thoughts?
74c3615 to
8bb4721
Compare
core/src/main/java/org/apache/iceberg/metrics/MetricsReporters.java
Outdated
Show resolved
Hide resolved
core/src/main/java/org/apache/iceberg/metrics/MetricsReporters.java
Outdated
Show resolved
Hide resolved
core/src/main/java/org/apache/iceberg/metrics/MetricsReporters.java
Outdated
Show resolved
Hide resolved
core/src/main/java/org/apache/iceberg/metrics/MetricsReporters.java
Outdated
Show resolved
Hide resolved
db694d4 to
abd5453
Compare
core/src/main/java/org/apache/iceberg/metrics/MetricsReporters.java
Outdated
Show resolved
Hide resolved
core/src/test/java/org/apache/iceberg/metrics/TestMetricsReporters.java
Outdated
Show resolved
Hide resolved
|
Looks good. Just one minor thing to fix with the sets. |
This also switches from a collection of reporters to the `CompositeMetricsReporter` in `TableScanContext` to fix an issue where duplicate reporters were not handled.
abd5453 to
1d49a8d
Compare
This also switches from a collection of reporters to the `CompositeMetricsReporter` in `TableScanContext` to fix an issue where duplicate reporters were not handled.
This also switches from a collection of reporters to the `CompositeMetricsReporter` in `TableScanContext` to fix an issue where duplicate reporters were not handled.

This introduces a
CompositeMetricsReporterthat allows registering multiple (simple)MetricsReporterinstances that all get notified whenever a newMetricsReportis available.This also switches from a collection of reporters to the
CompositeMetricsReporterinTableScanContextto fix an issue where duplicate reporters were not handled (see #7337 (comment) for details).