CommitReport not reported through custom MetricsReporter

[pawel-big-lebowski]

Apache Iceberg version

1.6.0

Query engine

Spark

Please describe the bug 🐞

I am running Spark jobs locally on my laptop with a custom MetricsReporter which does no external connections and is pretty much same to in-memory metrics reporter. I am using Spark Config to specify metrics reporter class.

My SQL is as simple as:

    spark.sql("CREATE TABLE source1 USING iceberg AS SELECT * FROM temp");
    spark.sql("CREATE TABLE target USING iceberg AS SELECT * FROM source1");

with a table having two fields and two rows.

In this case, I see my custom metrics reporter got loaded due to the log:

2024-11-26 15:11:10 INFO  org.apache.iceberg.CatalogUtil - Loading custom MetricsReporter implementation: io.openlineage.spark.agent.vendor.iceberg.metrics.OpenLineageMetricsReporter

I am able to debug that my metrics reporter gets ScanReport. However, it is not getting any CommitReport. Instead, they're logged with LoggingMetricsReporter:

2024-11-26 15:11:12 INFO  org.apache.iceberg.metrics.LoggingMetricsReporter - Received metrics report: CommitReport{tableName=default.source1, snapshotId=2507000462464497081, sequenceNumber=1, ...., metadata={engine-version=3.3.4, app-id=local-1732630268734, engine-name=spark, iceberg-version=Apache Iceberg 1.6.0 (commit 229d8f6fcd109e6c8943ea7cbb41dab746c6d0ed)}}

There are no logs of the form Cannot initialize MetricsReporter.

Moreover, when I run the query:

 spark.sql("INSERT INTO target VALUES (4, 5)");

I see commit reports within my custom reporter. That's why I think it's a bug while not being a configuration issue on my side.

During short debugging, I found out that BaseTransaction object is initialised through a constructor https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/BaseTransaction.java#L82 which injects LoggingMetricsReporter instead of using the one configured.

Willingness to contribute

• I can contribute a fix for this bug independently
• I would be willing to contribute a fix for this bug with guidance from the Iceberg community
• I cannot contribute a fix for this bug at this time

[singhpk234]

Taking a look!

[singhpk234]

I tested this via a unit test looks like it works :

During short debugging, I found out that BaseTransaction object is initialised through a constructor https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/BaseTransaction.java#L82 which injects LoggingMetricsReporter instead of using the one configured.

There are 2 constructors one which doesn't have reporter as the arg for that we wire in LoggingMetricsReporter otherwise we use the 2nd constuctor can you please share the spark command for intialize ?

I do see some missing wiring between spark and iceberg connector

[singhpk234]

Here are some places we miss propagating the reporter :
[1]

iceberg/spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/BaseSparkAction.java

Line 139 in bd7cff1

return new BaseTable(ops, metadata.metadataFileLocation());

[2]

iceberg/core/src/main/java/org/apache/iceberg/SerializableTable.java

Line 143 in bd7cff1

return new BaseTable(ops, tableName);

[nastra]

The main issue is that the metrics reporter isn't being propagated when a table is created/replaced through the Transaction API, which I've fixed in #11671.

To fix metrics reporter propagation through SerializableTable we'll need #8032.