[SPARK-54403][SQL][Metric View] Add YAML serde infrastructure for metric views by linhongliu-db · Pull Request #53146 · apache/spark

linhongliu-db · 2025-11-20T22:11:21Z

What changes were proposed in this pull request?

This PR adds the complete serialization/deserialization infrastructure for parsing metric view YAML definitions:

Add Jackson YAML dependencies to pom.xml
Implement canonical model for metric views:
- Column, Expression (Dimension/Measure), MetricView, Source
- YAMLVersion validation and exception types
Implement version-specific serde (v0.1):
- YAML deserializer/serializer
- Base classes for extensibility
Add JSON utilities for metadata serialization

Why are the changes needed?

SPIP: Metrics & semantic modeling in Spark

Does this PR introduce any user-facing change?

No

How was this patch tested?

build/sbt "catalyst/testOnly org.apache.spark.sql.metricview.serde.MetricViewFactorySuite"

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Code
Co-Authored-By: Claude noreply@anthropic.com

… for metric views This commit adds the complete serialization/deserialization infrastructure for parsing metric view YAML definitions: - Add Jackson YAML dependencies to pom.xml - Implement canonical model for metric views: - Column, Expression (Dimension/Measure), MetricView, Source - YAMLVersion validation and exception types - Implement version-specific serde (v0.1): - YAML deserializer/serializer - Base classes for extensibility - Add JSON utilities for metadata serialization

pan3793 · 2025-11-21T06:24:21Z

pom.xml

+      <dependency>
+        <groupId>com.fasterxml.jackson.dataformat</groupId>
+        <artifactId>jackson-dataformat-yaml</artifactId>
+        <version>${fasterxml.jackson.version}</version>


all jackson deps are managed by jackson-bom now, you don't need to declare it again here

LuciferYang · 2025-11-21T13:47:31Z

sql/catalyst/src/main/scala/org/apache/spark/sql/metricview/serde/JsonUtils.scala

+
+object JsonUtils {
+  // Singleton ObjectMapper that can be used across the project
+  private lazy val mapper: ObjectMapper = {


According to the comments, should we try to use this singleton as much as possible in the subsequent development?

Yes, I'm surprised that there is no such util in the Spark repo. For now, I plan to use it for all the metric view development, but not sure how much other code needs this.

sql/catalyst/src/main/scala/org/apache/spark/sql/metricview/serde/MetricViewCanonical.scala

sql/catalyst/src/main/scala/org/apache/spark/sql/metricview/serde/MetricViewSerDeBase.scala

sql/catalyst/src/main/scala/org/apache/spark/sql/metricview/serde/MetricViewSerDeV01.scala

sql/catalyst/src/main/scala/org/apache/spark/sql/metricview/serde/MetricViewCanonical.scala

linhongliu-db · 2025-11-24T19:47:56Z

cc @cloud-fan to review

sql/catalyst/src/main/scala/org/apache/spark/sql/metricview/serde/MetricViewCanonical.scala

…rde/MetricViewCanonical.scala

cloud-fan · 2025-12-04T19:20:17Z

sql/catalyst/src/main/scala/org/apache/spark/sql/metricview/serde/MetricViewSerDeV01.scala

+  }
+}
+
+object YamlMapperProvider extends YamlMapperProviderBase


do we need to add V01 postfix?

cloud-fan · 2025-12-04T19:21:29Z

sql/catalyst/src/main/scala/org/apache/spark/sql/metricview/serde/MetricViewCanonical.scala

+
+// Trait representing the capability to validate an object
+trait Validatable {
+  def validate(): Try[Unit]


why use Try? Can we fail directly?

cloud-fan · 2025-12-04T19:22:17Z

sql/catalyst/src/main/scala/org/apache/spark/sql/metricview/serde/MetricViewCanonical.scala

+  }
+}
+
+case class Column[T <: Expression](


is this type parameter really useful?

cloud-fan · 2025-12-05T22:35:35Z

sql/catalyst/src/main/scala/org/apache/spark/sql/metricview/serde/MetricViewCanonical.scala

+
+  def validate(): Unit


Suggested change

def validate(): Unit

cloud-fan · 2025-12-05T22:36:29Z

sql/catalyst/src/main/scala/org/apache/spark/sql/metricview/serde/MetricViewCanonical.scala

+    expression: Expression,
+    ordinal: Int) extends Validatable {
+  override def validate(): Unit = {
+    // No validation needed


then shall we remove extends Validatable?

cloud-fan · 2025-12-05T22:37:42Z

sql/catalyst/src/main/scala/org/apache/spark/sql/metricview/serde/MetricViewCanonical.scala

+  }
+
+  def getColumnMetadata: ColumnMetadata = {
+    val truncatedExpr = expression.expr.take(Constants.MAXIMUM_PROPERTY_SIZE)


we can't parse back a truncated expr, shall we just fail here if it's too large?

cloud-fan · 2025-12-05T22:43:11Z

sql/catalyst/src/main/scala/org/apache/spark/sql/metricview/serde/MetricViewCanonical.scala

+
+// Only parse the "version" field and ignore all others
+@JsonIgnoreProperties(ignoreUnknown = true)
+private[sql] case class YAMLVersion(version: String) extends Validatable {


does it need to be Validatable? We only create YAMLVersion in MetricViewFactory.fromYAML which already does validation.

remove all Validatable

linhongliu-db · 2025-12-08T21:58:57Z

cc @cloud-fan updated the PR based on comments, could you please take another look?

cloud-fan · 2025-12-09T07:51:02Z

sql/catalyst/src/main/scala/org/apache/spark/sql/metricview/serde/JsonUtils.scala

+import com.fasterxml.jackson.databind.ObjectMapper
+import com.fasterxml.jackson.module.scala.DefaultScalaModule
+
+private[sql] object JsonUtils {


it's dead code for now?

it will be used at https://github.com/apache/spark/pull/53158/files#diff-1f33f825cb8bc9e947d5f021b7e21ec37996d97b2d65375dd0c430a38f1c7c25R115

sql/catalyst/src/main/scala/org/apache/spark/sql/metricview/serde/MetricViewCanonical.scala

cloud-fan · 2025-12-09T07:53:03Z

sql/catalyst/src/main/scala/org/apache/spark/sql/metricview/serde/MetricViewCanonical.scala

+private[sql] object YAMLVersion {
+  private def validYAMLVersions: Set[String] = Set("0.1")
+
+  def apply(version: String): YAMLVersion = {


dead code for now?

good catch. the actual version validation is at MetricViewFactory and this code is useless. There is also a test that confirm this: https://github.com/apache/spark/pull/53146/files#diff-23bf5ddc582ff6684f7cc8950a12f4d8e745ff3fb7b0142dd00015e1f159fc8aR144

cloud-fan · 2025-12-09T07:55:59Z

sql/catalyst/src/main/scala/org/apache/spark/sql/metricview/serde/MetricViewSerDeBase.scala

+  def dimensions: Seq[ColumnBase]
+  def measures: Seq[ColumnBase]
+
+  def toCanonical: MetricView = {


Can you explain a bit more about how the canonical entities can help with metric view version evolution?

updated the comment

…rde/MetricViewCanonical.scala Co-authored-by: Wenchen Fan <cloud0fan@gmail.com>

linhongliu-db · 2025-12-09T17:11:59Z

@cloud-fan updated. :-)

cloud-fan · 2025-12-10T03:30:20Z

thanks, merging to master!

…ution ### What changes were proposed in this pull request? This PR implements the command to create metric views and the analysis rule to resolve a metric view query: - CREATE Metric view - Add SQL grammar to support `WITH METRIC` when creating a view - Add dollar-quoted string support for YAML definitions - Implement CreateMetricViewCommand to analyze the view body - Use a table property to indicate that the View is a metric view since HIVE has no dedicated table type - SELECT Metric view - Update SessionCatalog to parse metric view definitions on read - Add MetricViewPlanner utility to parse the YAML definition and construct an unresolved plan - Add ResolveMetricView rule to substitute the dimensions and measures reference to actual expressions NOTE: This PR depends on #53146 This PR also marks `org.apache.spark.sql.metricview` as an internal package ### Why are the changes needed? [SPIP: Metrics & semantic modeling in Spark](https://docs.google.com/document/d/1xVTLijvDTJ90lZ_ujwzf9HvBJgWg0mY6cYM44Fcghl0/edit?tab=t.0#heading=h.4iogryr5qznc) ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? ``` build/sbt "hive/testOnly org.apache.spark.sql.execution.SimpleMetricViewSuite" build/sbt "hive/testOnly org.apache.spark.sql.hive.execution.HiveMetricViewSuite" ``` ### Was this patch authored or co-authored using generative AI tooling? No Closes #53158 from linhongliu-db/metric-view-create-and-select. Authored-by: Linhong Liu <linhong.liu@databricks.com> Signed-off-by: Wenchen Fan <wenchen@databricks.com>

github-actions bot added SQL BUILD labels Nov 20, 2025

pan3793 reviewed Nov 21, 2025

View reviewed changes

LuciferYang reviewed Nov 21, 2025

View reviewed changes

sql/catalyst/src/main/scala/org/apache/spark/sql/metricview/serde/MetricViewCanonical.scala Outdated Show resolved Hide resolved

LuciferYang reviewed Nov 21, 2025

View reviewed changes

sql/catalyst/src/main/scala/org/apache/spark/sql/metricview/serde/MetricViewSerDeBase.scala Outdated Show resolved Hide resolved

LuciferYang reviewed Nov 21, 2025

View reviewed changes

sql/catalyst/src/main/scala/org/apache/spark/sql/metricview/serde/MetricViewSerDeBase.scala Outdated Show resolved Hide resolved

LuciferYang reviewed Nov 21, 2025

View reviewed changes

sql/catalyst/src/main/scala/org/apache/spark/sql/metricview/serde/MetricViewSerDeV01.scala Outdated Show resolved Hide resolved

LuciferYang reviewed Nov 21, 2025

View reviewed changes

sql/catalyst/src/main/scala/org/apache/spark/sql/metricview/serde/MetricViewCanonical.scala Outdated Show resolved Hide resolved

linhongliu-db mentioned this pull request Nov 21, 2025

[SPARK-54405][SQL][Metric View] CREATE command and SELECT query resolution #53158

Closed

address comments

84a208a

linhongliu-db commented Dec 4, 2025

View reviewed changes

sql/catalyst/src/main/scala/org/apache/spark/sql/metricview/serde/MetricViewCanonical.scala Outdated Show resolved Hide resolved

Update sql/catalyst/src/main/scala/org/apache/spark/sql/metricview/se…

c62f15c

…rde/MetricViewCanonical.scala

cloud-fan reviewed Dec 4, 2025

View reviewed changes

linhongliu-db added 2 commits December 5, 2025 21:57

address comment

7a3797a

more comments

11023ea

cloud-fan reviewed Dec 5, 2025

View reviewed changes

address comment

4608e0f

cloud-fan reviewed Dec 9, 2025

View reviewed changes

sql/catalyst/src/main/scala/org/apache/spark/sql/metricview/serde/MetricViewCanonical.scala Outdated Show resolved Hide resolved

cloud-fan reviewed Dec 9, 2025

View reviewed changes

linhongliu-db and others added 2 commits December 9, 2025 09:07

Update sql/catalyst/src/main/scala/org/apache/spark/sql/metricview/se…

06d8389

…rde/MetricViewCanonical.scala Co-authored-by: Wenchen Fan <cloud0fan@gmail.com>

remove dead code

804a4cc

cloud-fan approved these changes Dec 10, 2025

View reviewed changes

cloud-fan closed this in dc47def Dec 10, 2025

Conversation

linhongliu-db commented Nov 20, 2025

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

linhongliu-db commented Nov 24, 2025

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

linhongliu-db commented Dec 8, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cloud-fan Dec 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

linhongliu-db commented Dec 9, 2025

Uh oh!

cloud-fan commented Dec 10, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

cloud-fan Dec 9, 2025 •

edited

Loading