chore: Add global order min/max per tile to the fragment metadata #5646

rroelke · 2025-09-22T18:06:15Z

Resolves CORE-321.

Today's fragment metadata contains the minimum bounding rectangle of each tile. This is very useful for determining whether tiles can satisfy spatial queries, but much less useful for determining how different tiles from different fragments may or may not interleave (knowledge of which can be used to implement several possible optimizations). This is because the bounding rectangle can under-estimate the true lower bound of the global order value in a tile, potentially by a lot.

We would like for our queries to start to leverage the interleaving of tiles from different fragments in the ways described in that linked document. First we must extend the fragment metadata to include the tile global order bounds. This pull request implements that:

we update the fragment metadata format version and specification to include a section with the global order lower and upper bounds for each tile in a fragment.
we add C and C++ APIs to query those bounds.
we update the writers to populate the correct values in fragment metadata and test this for the unordered writer, global order writer, and fragment consoliator.

TYPE: C_API | CPP_API | FORMAT
DESC: Add per-tile global order bounds to fragment metadata

…metdata-global-order-bounds

…to load default profile in ordinary execution

…tion

…rlapping single-tile fragment section

rroelke · 2025-09-22T18:16:14Z

RE: CORE-383

Before merging this PR, estimate impact of it on this other issue. Will we make that problem worse?

…metdata-global-order-bounds

format_spec/fragment.md

teo-tsirpanis · 2025-09-24T00:07:22Z

tiledb/api/c_api/fragment_info/fragment_info_api_external.h

We don't have APIs that let users access other tile metadata, and I'm not sure we should add them, mainly because they exist only in new fragment metadata versions. I would be open to changing my mind, if there is an end-to-end scenario for these APIs.

These run alongside the various mbr APIs, e.g. tiledb_fragment_info_get_mbr_from_index.

The end-to-end scenario which I am interested in is enabling these to be queried using tiledb-tables as a potential support tool.

teo-tsirpanis · 2025-09-24T00:08:19Z

tiledb/common/types/untyped_datum.h

+  [[nodiscard]] constexpr inline const void* content() const {
    return datum_content_;
  }
-  [[nodiscard]] inline size_t size() const {
+  [[nodiscard]] constexpr inline size_t size() const {
    return datum_size_;
  }
  template <class T>
-  [[nodiscard]] inline const T& value_as() const {
+  [[nodiscard]] constexpr inline const T& value_as() const {
    return *static_cast<const T*>(datum_content_);
  }


Nit: the [[nodiscard]]s look a bit excessive if you asked me, but I'm OK with keeping them.

teo-tsirpanis · 2025-09-24T00:30:25Z

format_spec/fragment.md

+| Tile global order min coordinates offset for dimension 1 | `uint64_t` | _New in version 23_ For sparse arrays, he offset to the generic tile storing the tile global order mins for dimension 1
+| … | … | … |
+| Tile global order min coordinates offset for dimension N | `uint64_t` | _New in version 23_ For sparse arrays, the offset to the generic tile storing the tile global order mins for dimension N
+| Tile global order max coordinates offset for dimension 1 | `uint64_t` | _New in version 23_ For sparse arrays, the offset to the generic tile storing the tile global order maxes for dimension 1
+| … | … | … |
+| Tile global order max coordinates offset for dimension N | `uint64_t` | _New in version 23_ For sparse arrays, the offset to the generic tile storing the tile global order maxes for dimension N


I would prefer phrasing it like The offset to the […] for dimension N. This field exists only on sparse arrays..

We have yet to decide whether this is how we want to store the tiles, or use the extensible footer design described in CORE-345. If we do the latter, and don't want to change this PR's implementation, one proposed solution was to do it in a subsequent PR that will be merged before the next release.

The fact that these fields are only written in the footer for sparse arrays, is another reason to pursue the extensible footer design.

If we stick with this design, a better and simpler idea would be to always the offsets, and write zeroes for dense arrays.

We have yet to decide whether this is how we want to store the tiles, or use the extensible footer design described in CORE-345

Each time this has been brought up in the meeting the overall consensus has been (1) yes, we do want to implement a more extensible footer design; (2) no, we do not want to hold up existing work so that we can do that. It is settled.

The fact that these fields are only written in the footer for sparse arrays, is another reason to pursue the extensible footer design

I agree with this, it is rather clunky. However see above.

If we stick with this design, a better and simpler idea would be to always the offsets, and write zeroes for dense arrays

It removes a few ifs. Theoretically it might also for some other developer who wanted to read the storage format. I don't really have a preference.

format_spec/fragment.md

tiledb/sm/fragment/fragment_metadata.cc

format_spec/fragment.md

…metdata-global-order-bounds

) This pull request does some refactoring and enhancing of our test data structures `struct Fragment1D` and `struct Fragment2D` which are used as containers for inputs to write queries and outputs of read queries. 1) we move the methods which generically create arrays using these and write fragments using these to a header file instead of `sparse_global_order_reader.cc` 2) we refactor functionality into a `template <typename DimensionTuple, typename AttributeTuple> struct Fragment` which is a common base class for `Fragment1D`, `Fragment2D`, a new `Fragment3D` and can be used with an empty dimension tuple for dense fragments 3) we implement various fixes to enable variable-length dimensions 4) some other incidentals This pull request has been refactored out of #5646 for two reasons: 1) it is all enhancements to test code which may be distracting from the functional components of #5646 ; 2) I anticipate using it to add additional tests to #5655 . Specifically, using the `Fragment` with an empty dimension tuple will be very nice for writing tests on dense arrays. There's no direct testing of this, but since the code was factored out of #5646 all of it was used in testing there. Many components are exercised due to the refactor of the existing tests. --- TYPE: NO_HISTORY DESC: refactors and enhancements to test data structure `struct Fragment`

…metdata-global-order-bounds

) This pull request does some refactoring and enhancing of our test data structures `struct Fragment1D` and `struct Fragment2D` which are used as containers for inputs to write queries and outputs of read queries. 1) we move the methods which generically create arrays using these and write fragments using these to a header file instead of `sparse_global_order_reader.cc` 2) we refactor functionality into a `template <typename DimensionTuple, typename AttributeTuple> struct Fragment` which is a common base class for `Fragment1D`, `Fragment2D`, a new `Fragment3D` and can be used with an empty dimension tuple for dense fragments 3) we implement various fixes to enable variable-length dimensions 4) some other incidentals This pull request has been refactored out of #5646 for two reasons: 1) it is all enhancements to test code which may be distracting from the functional components of #5646 ; 2) I anticipate using it to add additional tests to #5655 . Specifically, using the `Fragment` with an empty dimension tuple will be very nice for writing tests on dense arrays. There's no direct testing of this, but since the code was factored out of #5646 all of it was used in testing there. Many components are exercised due to the refactor of the existing tests. --- TYPE: NO_HISTORY DESC: refactors and enhancements to test data structure `struct Fragment`

…metdata-global-order-bounds

) This pull request does some refactoring and enhancing of our test data structures `struct Fragment1D` and `struct Fragment2D` which are used as containers for inputs to write queries and outputs of read queries. 1) we move the methods which generically create arrays using these and write fragments using these to a header file instead of `sparse_global_order_reader.cc` 2) we refactor functionality into a `template <typename DimensionTuple, typename AttributeTuple> struct Fragment` which is a common base class for `Fragment1D`, `Fragment2D`, a new `Fragment3D` and can be used with an empty dimension tuple for dense fragments 3) we implement various fixes to enable variable-length dimensions 4) some other incidentals This pull request has been refactored out of #5646 for two reasons: 1) it is all enhancements to test code which may be distracting from the functional components of #5646 ; 2) I anticipate using it to add additional tests to #5655 . Specifically, using the `Fragment` with an empty dimension tuple will be very nice for writing tests on dense arrays. There's no direct testing of this, but since the code was factored out of #5646 all of it was used in testing there. Many components are exercised due to the refactor of the existing tests. --- TYPE: NO_HISTORY DESC: refactors and enhancements to test data structure `struct Fragment`

…metdata-global-order-bounds

rroelke added 30 commits July 21, 2025 13:55

Domain option version of get_dimension_index

a187efb

Dimension::cell_size

cb22d35

Add tile global/order min/max, bump format version, not tested

760f643

Stubs for C API retrieving tile global order bounds

fb387bd

Fixed-length simple test passes

d3d9936

Fix test with serialization on

273a0bb

Fix set_num_tiles index for bounds buffers

6ad6ac9

Add tiledb::Array::context

fa5b8ee

Refactor some functions into array_templates.h

01ec15b

Refactor Fragment1D, Fragment2D into common base class

bfbec1f

Fragment metadata global order bounds example passes

a018738

Add unordered 1D test

7a1e6e6

Rapidcheck runs, prints minimum

b639572

Merge remote-tracking branch 'origin/main' into rr/core-321-fragment-…

f642e6e

…metdata-global-order-bounds

Add RestProfile::file_exists to avoid throwing exception for failing …

9c900b3

…to load default profile in ordinary execution

Fix offset by tile_index_base_

99e04db

Rapidcheck passes for fixed 1D no dups

83e2b36

Turn on dups

4947b3a

Fix Fragment2d::d2

f9b93fb

Test Fragment metadata global order bounds: 2D fixed unordered section

b997f22

Test Fragment metadata global order bounds: 2D fixed global order sec…

52672dc

…tion

Add 1D var test with trivial input

72cdd82

Trivial 1D var test passes

23e9420

Minimum write GENERATE

49cd23e

Fix global_cell_cmp_std_tuple for variable-length dimension

5dbee50

Fix set_tile_global_order_bounds_var indexing

23c397f

Nontrival 1D var test

0e25643

Fragment::extend

b7ab7f1

Fix prepare_bound_buffers for empty qb

a0dedb9

Fragment metadata global order bounds: 1D fixed consolidation non-ove…

eed749a

…rlapping single-tile fragment section

3D vcf rapidcheck

6cb0b1d

rroelke marked this pull request as draft September 22, 2025 18:06

rroelke added 12 commits September 22, 2025 16:15

Merge remote-tracking branch 'origin/main' into rr/core-321-fragment-…

722f3e0

…metdata-global-order-bounds

Add C++ API and use it in tests

44469f8

Add 3D vcf consolidation rapidcheck test

d176eb6

Update format spec

e76b0ce

Only write global order min/maxes for sparse array

029592c

Allow dimension_sizes arg to be nullptr

36d624c

Recapture output for 'dump with string dimension' test

369e02a

Fix interleaving triples output

045beb4

Fix cpp context lifetime issue

0e4d960

constexpr UntypedDatumView methods/constructors

4d08b0c

Fix cppapi impl dimension_sizes

5886af9

std::min explicit template to make mac compiler happy

804051e

teo-tsirpanis reviewed Sep 24, 2025

View reviewed changes

format_spec/fragment.md Outdated Show resolved Hide resolved

tiledb/sm/fragment/fragment_metadata.cc Outdated Show resolved Hide resolved

format_spec/fragment.md Show resolved Hide resolved

rroelke added 3 commits October 13, 2025 12:47

Fix format spec typos and version number

3c316af

Separate method for new versioning

0434459

Merge remote-tracking branch 'origin/main' into rr/core-321-fragment-…

47df461

…metdata-global-order-bounds

rroelke mentioned this pull request Oct 29, 2025

chore: refactor and improve test data structure struct Fragment #5675

Merged

Merge remote-tracking branch 'origin/main' into rr/core-321-fragment-…

12fec8b

…metdata-global-order-bounds

Merge remote-tracking branch 'origin/main' into rr/core-321-fragment-…

f187ce3

…metdata-global-order-bounds

rroelke added 2 commits November 6, 2025 14:23

Merge remote-tracking branch 'origin/main' into rr/core-321-fragment-…

5c0b0af

…metdata-global-order-bounds

Restore Dimension<STRING_ASCII> specialization

677fa62

teo-tsirpanis mentioned this pull request Nov 13, 2025

Extensible fragment metadata footer. #5686

Draft

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

chore: Add global order min/max per tile to the fragment metadata #5646

chore: Add global order min/max per tile to the fragment metadata #5646

rroelke commented Sep 22, 2025

Uh oh!

rroelke commented Sep 22, 2025

Uh oh!

Uh oh!

teo-tsirpanis Sep 24, 2025

Uh oh!

rroelke Sep 30, 2025

Uh oh!

teo-tsirpanis Sep 24, 2025

Uh oh!

teo-tsirpanis Sep 24, 2025

Uh oh!

rroelke Sep 30, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

chore: Add global order min/max per tile to the fragment metadata #5646

Are you sure you want to change the base?

chore: Add global order min/max per tile to the fragment metadata #5646

Conversation

rroelke commented Sep 22, 2025

Uh oh!

rroelke commented Sep 22, 2025

Uh oh!

Uh oh!

teo-tsirpanis Sep 24, 2025

Choose a reason for hiding this comment

Uh oh!

rroelke Sep 30, 2025

Choose a reason for hiding this comment

Uh oh!

teo-tsirpanis Sep 24, 2025

Choose a reason for hiding this comment

Uh oh!

teo-tsirpanis Sep 24, 2025

Choose a reason for hiding this comment

Uh oh!

rroelke Sep 30, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants