GH-825: Add UUID canonical extension type #903

jhrotko · 2025-11-03T14:51:38Z

What's Changed

Add Uuid Extension type canonical implementation in Java. The proposed implementation was validated in Dremio engine

Closes #825

lidavidm

See previous PR. It is not sufficient to move the existing files, the "uuid" type used in tests was never meant to implement the canonical type. #824

jhrotko · 2025-11-07T13:59:16Z

@lidavidm could you take another look? Let me know if anything else is missing

bodduv

Thank you for adding the canonical type.

vector/src/main/java/org/apache/arrow/vector/extension/UuidType.java

vector/src/main/java/org/apache/arrow/vector/complex/impl/UuidWriterImpl.java

vector/src/main/java/org/apache/arrow/vector/FixedSizeExtensionType.java

vector/src/main/java/org/apache/arrow/vector/UuidVector.java

jhrotko · 2025-11-11T12:18:42Z

there are some integration tests failling:

  ################# FAILURES #################
  FAILED TEST: custom_metadata Java producing,  Go consuming
  <class 'RuntimeError'>: Command failed: /root/go/bin/arrow-json-integration-test -arrow /tmp/tmprky9nvsk/020d3b5c_generated_custom_metadata.json_as_file -json /tmp/arrow-integration-ypsmstn1/generated_custom_metadata.json -mode VALIDATE
  With output:
  --------------
  arrow-json: JSON schema:
  schema:
    fields: 4
      - sort_of_pandas: type=int8, nullable
                  metadata: ["pandas": "{}"]
      - lots_of_meta: type=int8, nullable
                metadata: ["a": "{}", "b": "{}", "c": "{}", "d": "{}", "..": "{}", "w": "{}", "x": "{}", "y": "{}", "z": "{}"]
      - unregistered_extension: type=int8, nullable
                          metadata: ["ARROW:extension:name": "!nonexistent", "ARROW:extension:metadata": "", "ARROW:integration:allow_unregistered_extension": "true"]
      - list_with_odd_values: type=list<item: int32, nullable>, nullable
    metadata: ["schema_custom_0": "{}", "schema_custom_1": "{}"]
  Arrow schema:
  schema:
    fields: 4
      - sort_of_pandas: type=int8, nullable
                  metadata: ["pandas": "{}"]
      - lots_of_meta: type=int8, nullable
                metadata: ["..": "{}", "a": "{}", "b": "{}", "c": "{}", "d": "{}", "w": "{}", "x": "{}", "y": "{}", "z": "{}"]
      - unregistered_extension: type=int8, nullable
                          metadata: ["ARROW:extension:name": "!nonexistent", "ARROW:extension:metadata": "", "ARROW:integration:allow_unregistered_extension": "true"]
      - list_with_odd_values: type=list<item: int32, nullable>, nullable
    metadata: ["schema_custom_1": "{}", "schema_custom_0": "{}"]
  arrow-json: schemas did not match
  
  --------------
  
  
  FAILED TEST: flight_sql Java producing,  Go consuming
  <class 'RuntimeError'>: Command failed: /root/go/bin/arrow-flight-integration-client -host localhost -port=43857 -scenario flight_sql
  With output:
  --------------
  gc 1 @3.115s 0%: 0.078+0.59+0.008 ms clock, 0.31+0.37/0.34/0.60+0.032 ms cpu, 3->3->0 MB, 4 MB goal, 0 MB stacks, 0 MB globals, 4 P
  panic: expected: schema:
  	  fields: 1
  	    - id: type=int64, nullable
  	    metadata: ["ARROW:FLIGHT:SQL:TABLE_NAME": "test", "ARROW:FLIGHT:SQL:IS_AUTO_INCREMENT": "1", "ARROW:FLIGHT:SQL:IS_CASE_SENSITIVE": "0", "ARROW:FLIGHT:SQL:TYPE_NAME": "type_test", "ARROW:FLIGHT:SQL:SCHEMA_NAME": "schema_test", "ARROW:FLIGHT:SQL:IS_SEARCHABLE": "1", "ARROW:FLIGHT:SQL:CATALOG_NAME": "catalog_test", "ARROW:FLIGHT:SQL:PRECISION": "100", "ARROW:FLIGHT:SQL:REMARKS": "test column"], got: schema:
  	  fields: 1
  	    - id: type=int64, nullable
  	    metadata: ["ARROW:FLIGHT:SQL:PRECISION": "100", "ARROW:FLIGHT:SQL:REMARKS": "test column", "ARROW:FLIGHT:SQL:IS_AUTO_INCREMENT": "1", "ARROW:FLIGHT:SQL:IS_CASE_SENSITIVE": "0", "ARROW:FLIGHT:SQL:SCHEMA_NAME": "schema_test", "ARROW:FLIGHT:SQL:TABLE_NAME": "test", "ARROW:FLIGHT:SQL:IS_SEARCHABLE": "1", "ARROW:FLIGHT:SQL:TYPE_NAME": "type_test", "ARROW:FLIGHT:SQL:CATALOG_NAME": "catalog_test"]
  
  goroutine 1 [running]:
  main.main()
  	/arrow/go/arrow/internal/flight_integration/cmd/arrow-flight-integration-client/main.go:52 +0x277
  
  --------------
  
  
  FAILED TEST: flight_sql:extension Java producing,  Go consuming
  <class 'RuntimeError'>: Command failed: /root/go/bin/arrow-flight-integration-client -host localhost -port=35551 -scenario flight_sql:extension
  With output:
  --------------
  panic: expected: schema:
  	  fields: 1
  	    - id: type=int64, nullable
  	    metadata: ["ARROW:FLIGHT:SQL:TABLE_NAME": "test", "ARROW:FLIGHT:SQL:IS_AUTO_INCREMENT": "1", "ARROW:FLIGHT:SQL:IS_CASE_SENSITIVE": "0", "ARROW:FLIGHT:SQL:TYPE_NAME": "type_test", "ARROW:FLIGHT:SQL:SCHEMA_NAME": "schema_test", "ARROW:FLIGHT:SQL:IS_SEARCHABLE": "1", "ARROW:FLIGHT:SQL:CATALOG_NAME": "catalog_test", "ARROW:FLIGHT:SQL:PRECISION": "100", "ARROW:FLIGHT:SQL:REMARKS": "test column"], got: schema:
  	  fields: 1
  	    - id: type=int64, nullable
  	    metadata: ["ARROW:FLIGHT:SQL:PRECISION": "100", "ARROW:FLIGHT:SQL:REMARKS": "test column", "ARROW:FLIGHT:SQL:IS_AUTO_INCREMENT": "1", "ARROW:FLIGHT:SQL:IS_CASE_SENSITIVE": "0", "ARROW:FLIGHT:SQL:SCHEMA_NAME": "schema_test", "ARROW:FLIGHT:SQL:TABLE_NAME": "test", "ARROW:FLIGHT:SQL:IS_SEARCHABLE": "1", "ARROW:FLIGHT:SQL:TYPE_NAME": "type_test", "ARROW:FLIGHT:SQL:CATALOG_NAME": "catalog_test"]
  
  goroutine 1 [running]:
  main.main()
  	/arrow/go/arrow/internal/flight_integration/cmd/arrow-flight-integration-client/main.go:52 +0x277
  
  --------------
  
  
  FAILED TEST: flight_sql Go producing,  Java consuming
  <class 'RuntimeError'>: Command failed: java -Dio.netty.tryReflectionSetAccessible=true -Darrow.struct.conflict.policy=CONFLICT_APPEND --add-opens=java.base/java.nio=org.apache.arrow.memory.core,ALL-UNNAMED -XX:-UsePerfData --add-reads=org.apache.arrow.flight.core=ALL-UNNAMED -cp /build/java/flight/flight-integration-tests/target/flight-integration-tests-19.0.0-SNAPSHOT-jar-with-dependencies.jar org.apache.arrow.flight.integration.tests.IntegrationTestClient -port 40707 -scenario flight_sql
  With output:
  --------------
  WARNING: Unknown module: org.apache.arrow.flight.core specified to --add-reads
  WARNING: Unknown module: org.apache.arrow.memory.core specified to --add-opens
  SLF4J(W): No SLF4J providers were found.
  SLF4J(W): Defaulting to no-operation (NOP) logger implementation
  SLF4J(W): See https://www.slf4j.org/codes.html#noProviders for further details.
  Unknown error
  expected: '%!s(bool=true)', got: '%!s(bool=false)'
  
  --------------
  
  
  FAILED TEST: flight_sql:extension Go producing,  Java consuming
  <class 'RuntimeError'>: Command failed: java -Dio.netty.tryReflectionSetAccessible=true -Darrow.struct.conflict.policy=CONFLICT_APPEND --add-opens=java.base/java.nio=org.apache.arrow.memory.core,ALL-UNNAMED -XX:-UsePerfData --add-reads=org.apache.arrow.flight.core=ALL-UNNAMED -cp /build/java/flight/flight-integration-tests/target/flight-integration-tests-19.0.0-SNAPSHOT-jar-with-dependencies.jar org.apache.arrow.flight.integration.tests.IntegrationTestClient -port 32823 -scenario flight_sql:extension
  With output:
  --------------
  WARNING: Unknown module: org.apache.arrow.flight.core specified to --add-reads
  WARNING: Unknown module: org.apache.arrow.memory.core specified to --add-opens
  SLF4J(W): No SLF4J providers were found.
  SLF4J(W): Defaulting to no-operation (NOP) logger implementation
  SLF4J(W): See https://www.slf4j.org/codes.html#noProviders for further details.
  Unknown error
  expected: '%!s(bool=true)', got: '%!s(bool=false)'
  
  --------------
  
  
  FAILED TEST: custom_metadata Java producing,  Go consuming
  <class 'RuntimeError'>: Go C Data Integration call failed: Schemas are different:
  - Json Schema: schema:
    fields: 4
      - sort_of_pandas: type=int8, nullable
                  metadata: ["pandas": "{}"]
      - lots_of_meta: type=int8, nullable
                metadata: ["a": "{}", "b": "{}", "c": "{}", "d": "{}", "..": "{}", "w": "{}", "x": "{}", "y": "{}", "z": "{}"]
      - unregistered_extension: type=int8, nullable
                          metadata: ["ARROW:extension:name": "!nonexistent", "ARROW:extension:metadata": "", "ARROW:integration:allow_unregistered_extension": "true"]
      - list_with_odd_values: type=list<item: int32, nullable>, nullable
    metadata: ["schema_custom_0": "{}", "schema_custom_1": "{}"]
  - Imported Schema: schema:
    fields: 4
      - sort_of_pandas: type=int8, nullable
                  metadata: ["pandas": "{}"]
      - lots_of_meta: type=int8, nullable
                metadata: ["..": "{}", "a": "{}", "b": "{}", "c": "{}", "d": "{}", "w": "{}", "x": "{}", "y": "{}", "z": "{}"]
      - unregistered_extension: type=int8, nullable
                          metadata: ["ARROW:extension:name": "!nonexistent", "ARROW:extension:metadata": "", "ARROW:integration:allow_unregistered_extension": "true"]
      - list_with_odd_values: type=list<item: int32, nullable>, nullable
    metadata: ["schema_custom_1": "{}", "schema_custom_0": "{}"]

It seems the metadata elements when it's a map the order is not maintained. I this a known issue? @lidavidm

lidavidm · 2025-11-11T13:04:32Z

It's apache/arrow-go#571

jhrotko · 2025-11-11T16:07:04Z

@lidavidm it was merged, should this be re-run?

jhrotko · 2025-11-12T08:22:51Z

@lidavidm anything else missing?

bodduv

Thank you for addressing the comments.

github-actions bot added the breaking-change label Nov 3, 2025

This comment has been minimized.

Sign in to view

apacheGH-825: Move UuidType to arrow.vector from test

4e8de67

jhrotko force-pushed the GH-825 branch from 595d2ae to 4e8de67 Compare November 3, 2025 14:53

lidavidm requested changes Nov 3, 2025

View reviewed changes

jhrotko changed the title ~~GH-825: Move UuidType to arrow.vector from test~~ GH-825: Implement Uuid canonical type Nov 7, 2025

jhrotko changed the title ~~GH-825: Implement Uuid canonical type~~ GH-825: Add a canonical extension type UUID Nov 7, 2025

Add canonical implementation

ebca624

jhrotko force-pushed the GH-825 branch from a298920 to ebca624 Compare November 7, 2025 13:58

jhrotko marked this pull request as ready for review November 7, 2025 13:58

jhrotko requested review from jbonofre, laurentgo and wgtmac as code owners November 7, 2025 13:58

jhrotko changed the title ~~GH-825: Add a canonical extension type UUID~~ GH-825: Add UUID canonical extension type Nov 7, 2025

bodduv reviewed Nov 9, 2025

View reviewed changes

vector/src/main/java/org/apache/arrow/vector/extension/UuidType.java Show resolved Hide resolved

vector/src/main/java/org/apache/arrow/vector/extension/UuidType.java Show resolved Hide resolved

lidavidm reviewed Nov 10, 2025

View reviewed changes

jhrotko added 2 commits November 10, 2025 17:02

Address comments

f7880fb

Remove FixedSizeExtensionType

b04af47

lidavidm added enhancement PRs that add or improve features. and removed breaking-change labels Nov 11, 2025

github-actions bot added this to the 18.4.0 milestone Nov 11, 2025

bodduv approved these changes Nov 14, 2025

View reviewed changes

jhrotko requested a review from lidavidm November 14, 2025 14:13

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

GH-825: Add UUID canonical extension type #903

GH-825: Add UUID canonical extension type #903

Uh oh!

jhrotko commented Nov 3, 2025 •

edited

Loading

Uh oh!

This comment has been minimized.

lidavidm left a comment

Uh oh!

jhrotko commented Nov 7, 2025 •

edited

Loading

Uh oh!

bodduv left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jhrotko commented Nov 11, 2025

Uh oh!

lidavidm commented Nov 11, 2025

Uh oh!

jhrotko commented Nov 11, 2025

Uh oh!

jhrotko commented Nov 12, 2025

Uh oh!

bodduv left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

GH-825: Add UUID canonical extension type #903

Are you sure you want to change the base?

GH-825: Add UUID canonical extension type #903

Uh oh!

Conversation

jhrotko commented Nov 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What's Changed

Uh oh!

This comment has been minimized.

lidavidm left a comment

Choose a reason for hiding this comment

Uh oh!

jhrotko commented Nov 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bodduv left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jhrotko commented Nov 11, 2025

Uh oh!

lidavidm commented Nov 11, 2025

Uh oh!

jhrotko commented Nov 11, 2025

Uh oh!

jhrotko commented Nov 12, 2025

Uh oh!

bodduv left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

jhrotko commented Nov 3, 2025 •

edited

Loading

jhrotko commented Nov 7, 2025 •

edited

Loading