feat: add a MVP for column statistics at dataset level on Rust side

[HaochengLIU]

design doc ,Related to #4540

Overview

This PR implements dataset-level column statistics for Lance, enabling query optimization through min/max/null/nan statistics stored at the dataset level. The implementation follows a 7-phase design with ~4,200 lines of code.

Default to off.

---

🎯 What This PR Does

Goal: When using Dataset write API, collect per-fragment column statistics at writer level and consolidate them into a single dataset-level file at compaction stage for query optimization.

Key Features:

• Per-fragment statistics collection (min, max, null_count, nan_count)
• Dataset-level consolidation into a single Lance file
• Policy enforcement (all-or-nothing: all fragments must have stats)
• Compaction integration (automatic consolidation)
• Backward/forward compatible format

---

📊 Architecture Summary

┌─────────────────────────────────────────────────────────────┐
│                     Write Operation                          │
│  ┌──────────────┐         ┌──────────────┐                  │
│  │ Data Chunks  │────────▶│ FileWriter   │                  │
│  └──────────────┘         │ + ZoneStats  │                  │
│                            └──────┬───────┘                  │
│                                   │                          │
│                                   ▼                          │
│                         ┌─────────────────────┐              │
│                         │  Fragment.lance     │              │
│                         │  + column_stats     │              │
│                         │    (Arrow IPC)      │              │
│                         └─────────────────────┘              │
└─────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────┐
│                    Compaction Operation                      │
│                                                              │
│  ┌────────────┐   ┌────────────┐   ┌────────────┐          │
│  │ Frag 0     │   │ Frag 1     │   │ Frag 2     │          │
│  │ stats      │   │ stats      │   │ stats      │          │
│  └─────┬──────┘   └─────┬──────┘   └─────┬──────┘          │
│        │                │                │                  │
│        └────────────────┼────────────────┘                  │
│                         │                                   │
│                         ▼                                   │
│              ┌──────────────────────┐                       │
│              │  Consolidate Stats   │                       │
│              │  (Global Offsets)    │                       │
│              └──────────┬───────────┘                       │
│                         │                                   │
│                         ▼                                   │
│              ┌──────────────────────┐                       │
│              │ _stats/              │                       │
│              │  column_stats_v1_... │                       │
│              │  (Lance file)        │                       │
│              └──────────────────────┘                       │
└─────────────────────────────────────────────────────────────┘

---

📁 Files by Phase (Review Order)

Phase 0: Code refactor between index and column stats (Foundation)

1. rust/lance-core/src/utils/zone.rs (NEW, 212 lines)
   • ZoneBound, ZoneProcessor, FileZoneBuilder
   • Reusable zone utilities for both stats and indices

Phase 1: Policy Enforcement at writer level (Consistency)

2. rust/lance/src/dataset/write.rs (MODIFIED, +50 lines)

   • Added enable_column_stats field to WriteParams
   • validate_column_stats_policy(): Errors on mismatch

3. rust/lance/src/dataset/write/insert.rs (MODIFIED, +185 lines)

   • Sets lance.column_stats.enabled in manifest on dataset creation
   • 4 tests for policy enforcement

Phase 2: Per-Fragment Writer (Collection)

4. rust/lance-file/src/writer.rs (MODIFIED, +407 lines)
   • build_column_statistics(): Collects stats using DataFusion accumulators
   • Stores as Arrow IPC in global buffer
   • Use a version number to control upgrade

Phase 3: Per-Fragment Reader (Retrieval)

5. rust/lance-file/src/reader.rs (MODIFIED, +305 lines)
   • has_column_stats(): Quick metadata check
   • read_column_stats(): Deserialize Arrow IPC

Phase 4: Consolidation Core (Aggregation)

6. rust/lance/src/dataset/column_stats.rs (NEW, 1,049 lines)
   • consolidate_column_stats(): Main consolidation logic
   • All-or-nothing policy check
   • Global offset calculation

Manifest Reference:

• Key: lance.column_stats.file
• Value: _stats/column_stats.lance (always the same path)
• Version: Stored as file metadata (lance:dataset:version key in Lance file)

Phase 5: High-Level API (Type Dispatch)

7. rust/lance/src/dataset/column_stats_reader.rs (NEW, 397 lines)
   • ColumnStatsReader: Read with automatic type conversion
   • parse_scalar_value(): String → ScalarValue dispatch
   • Supports Int8-64, UInt8-64, Float32/64, Utf8, LargeUtf8

Phase 6: Compaction Integration (Automation)

8. rust/lance/src/dataset/optimize.rs (MODIFIED, +630 lines)
   • Added consolidate_column_stats to CompactionOptions (default: true)
   • Automatic consolidation after compaction
   • 6 comprehensive tests covering all scenarios

---

🔒 Backward/Forward Compatibility

Backward Compatibility

• Old datasets without stats: ✅ Work normally (consolidation skipped)
• Old readers: ✅ Ignore new metadata (no breaking changes)

Forward Compatibility

• New columns can be added: ✅ Schema is extensible
• New stat types: ✅ Can add columns to consolidated file
• Format versioning: ✅ lance:column_stats:version in metadata

---

🚀 Next Steps

This PR is the foundation. Future work:

1. Query Optimizer Integration: Use stats for predicate pushdown
2. Python Bindings: Expose stats to Python API
3. Performance benchmarks
4. Additional Statistics: Distinct count, histograms
5. Incremental Updates: Update stats without full consolidation
6. Stats-based Query Planning: Cardinality estimation

---

📝 Questions for Reviewers

1. Policy Enforcement: Is the error message clear when policy mismatches?
2. Schema Evolution: Any concerns about adding new stat types later?
3. Compaction Integration: Does it cover all compaction code paths?
4. API Design: Is ColumnStatsReader intuitive enough?
5. Test Coverage: Any missing edge cases?

[github-actions]

ACTION NEEDED
Lance follows the Conventional Commits specification for release automation.

The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification.

For details on the error please inspect the "PR Title Check" action.

[github-actions]

Code Review: WIP Add column stats mvp

P0 Issues (Must Fix)

1. Hardcoded Absolute Path in Test
rust/lance/src/dataset/tests/dataset_io.rs:283

let test_uri = "/Users/haochengliu/Documents/projects/lance/ColStats";

This hardcoded absolute path will cause test failures on CI and other machines. Use TempStrDir::default() like other tests in the file.

2. Min/Max Serialization Format is Fragile
rust/lance-file/src/writer.rs:854-855

mins.push(format!("{:?}", zone.min));
maxs.push(format!("{:?}", zone.max));

Using Debug format ({:?}) for serialization produces non-parseable strings like Int32(5) or Utf8("hello"). This will make deserialization difficult and is not a stable format. Consider:

• Using Arrow's native types (e.g., store min/max in their original Arrow array types)
• Using a well-defined serialization format (JSON, protobuf, etc.)

P1 Issues (Should Fix)

3. No Reader Implementation
Statistics are written to global buffers but there's no code to read them back. Without a reader, this feature cannot be tested end-to-end or used for query optimization.

4. New Dependencies Added to lance-file
rust/lance-file/Cargo.toml now includes full datafusion and datafusion-expr dependencies (not just datafusion-common). This significantly increases the dependency footprint. Consider:

• Moving statistics computation to a separate module that can be optionally included
• Using a lighter-weight min/max computation instead of DataFusion accumulators

5. Test Only Prints Debug Output
The test test_write_with_column_stats only prints metadata to stdout and doesn't verify the statistics were actually computed correctly. Add assertions that verify:

• Statistics exist in the schema metadata
• Min/max values are correct for the test data

Questions

1. How will readers know to look for the statistics? The buffer index is stored in schema metadata, but there's no documentation or constants for the metadata keys.
2. Will this work with nested fields and complex types? The current implementation iterates over top-level fields only.

[codecov]

Codecov Report

❌ Patch coverage is 91.43104% with 297 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...ust/lance/src/dataset/column_stats_consolidator.rs	90.66%	89 Missing and 19 partials ⚠️
rust/lance/src/dataset/column_stats_reader.rs	77.24%	69 Missing and 2 partials ⚠️
rust/lance-file/src/writer.rs	89.16%	36 Missing and 12 partials ⚠️
rust/lance-file/src/reader.rs	87.09%	27 Missing and 1 partial ⚠️
rust/lance/src/dataset/write/insert.rs	95.19%	5 Missing and 5 partials ⚠️
rust/lance-file/src/writer/column_stats.rs	92.38%	0 Missing and 8 partials ⚠️
rust/lance/src/dataset/optimize.rs	99.05%	3 Missing and 3 partials ⚠️
rust/lance/src/dataset/write.rs	89.13%	4 Missing and 1 partial ⚠️
rust/lance-core/src/utils/zone.rs	98.47%	0 Missing and 4 partials ⚠️
rust/lance/src/dataset/transaction.rs	63.63%	4 Missing ⚠️
... and 2 more

📢 Thoughts on this report? Let us know!

[HaochengLIU]

There are some old tests are still failing as I enable the column stats to be ON by default.

I will try my best to fix them before I will be OOO, if not can someone lend me a hand please? @jackye1995 @westonpace ? tyvm