8633: Fix Writer docs and rename AvroBinaryFormat to AvroSoeFormat

[martin-augment]

8633: To review by AI

Summary by CodeRabbit

• New Features

  • Added UTF-8 View support through new public API extension trait.

• Documentation

  • Updated writer encoding documentation to focus on Single Object Encoding (SOE) format.
  • Enhanced Avro stream writing examples and guides with SOE-focused quickstart.
  • Improved error messages for better clarity on SOE streaming context.

• API Changes

  • Renamed binary format to SOE (Single Object Encoding) for consistency with encoding standard.

[coderabbitai]

Walkthrough

The pull request refactors Avro writer functionality by renaming AvroBinaryFormat to AvroSoeFormat (Single Object Encoding) across multiple modules, updates documentation to reflect SOE semantics, and introduces a new public trait AvroFieldExt with a with_utf8view method for field manipulation.

Changes

Cohort / File(s)	Summary
Format naming refactor arrow-avro/src/writer/format.rs	Renamed AvroBinaryFormat struct to AvroSoeFormat; updated impl for AvroFormat; updated error message to reference "Avro SOE streaming" instead of "Avro binary streaming"
Stream writer API updates arrow-avro/src/writer/mod.rs	Changed public type alias AvroStreamWriter to use AvroSoeFormat instead of AvroBinaryFormat; updated impl blocks, imports, builder methods, and documentation to reference AvroSoeFormat and describe SOE semantics
Reader documentation updates arrow-avro/src/reader/mod.rs	Updated doctests and examples to reference AvroSoeFormat instead of AvroBinaryFormat; updated Decoder documentation to describe SOE-framed Avro bytes
Public trait addition & documentation arrow-avro/src/lib.rs	Introduced new public trait AvroFieldExt with with_utf8view method; provided impl for codec::AvroField; updated module documentation to emphasize SOE format and Apicurio support; marked OCF example as runnable

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~8 minutes

The changes are predominantly homogeneous—a consistent refactoring that renames one format type to another across multiple files and updates related documentation. While affecting multiple files, the pattern is repetitive and straightforward. The new trait addition is minimal. No complex logic modifications are present.

Poem

🐰 From Binary to SOE, our Avro streams now flow,
A trait for UTF8 views makes extensions grow,
With single objects encoded so divine,
This refactored format will surely shine! ✨

Pre-merge checks and finishing touches

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Description Check	⚠️ Warning	The PR description provided ("8633: To review by AI") is incomplete and does not follow the repository's description template at all. The template requires addressing key sections including which issue is closed, the rationale for changes, a summary of what changes are included, whether changes are tested, and whether there are user-facing changes. The current description lacks all substantive information required by these sections and appears to be a placeholder rather than a completed submission. This severely limits the ability for reviewers to understand the context, testing approach, and any potential breaking changes associated with this PR.	The description must be expanded to properly address all required sections of the template. At minimum, the author should clarify which issue (likely apache#8633) this PR closes, explain the rationale for the rename from AvroBinaryFormat to AvroSoeFormat, confirm whether tests are included for the new AvroFieldExt trait and the rename changes, and explicitly call out any breaking changes to the public API such as the type alias modification and new trait.

✅ Passed checks (2 passed)

Check name	Status	Explanation
Title Check	✅ Passed	The PR title "8633: Fix Writer docs and rename AvroBinaryFormat to AvroSoeFormat" accurately captures the primary changes across the codebase, which involve renaming the format type and updating writer documentation to reflect SOE (Single Object Encoding) semantics. The title is concise and specific, allowing a reviewer to quickly understand the main focus. However, the title does not mention the introduction of a new public trait AvroFieldExt with a with_utf8view method, which represents an additional public API change present in the changeset. Despite this omission, the title remains closely aligned with the primary refactoring effort described in the commits.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.

✨ Finishing touches

• 📝 Generate docstrings

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch pr-8633-2025-10-24-08-47-47

---

📜 Recent review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 7b3b262 and 9276c6f.

📒 Files selected for processing (4)

• arrow-avro/src/lib.rs (3 hunks)
• arrow-avro/src/reader/mod.rs (10 hunks)
• arrow-avro/src/writer/format.rs (2 hunks)
• arrow-avro/src/writer/mod.rs (11 hunks)

🧰 Additional context used

🧬 Code graph analysis (1)

arrow-avro/src/writer/mod.rs (3)

arrow-avro/src/schema.rs (4)

• new (358-360)
• new (786-788)
• schema (362-365)
• schema (456-460)

arrow-avro/src/writer/encoder.rs (3)

• new (739-745)
• new (1916-1918)
• new (1955-1969)

arrow-avro/src/reader/header.rs (1)

• schema (116-124)

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (5)

• GitHub Check: Clippy
• GitHub Check: Test on Mac
• GitHub Check: Test on Windows
• GitHub Check: Release Audit Tool (RAT)
• GitHub Check: claude-review

🔇 Additional comments (6)

arrow-avro/src/writer/format.rs (1)

107-136: LGTM! Clear rename to align with Avro terminology.

The rename from AvroBinaryFormat to AvroSoeFormat accurately reflects the Single Object Encoding specification, and the documentation correctly describes the per-record framing behavior. The error message at line 127 is appropriately updated.

arrow-avro/src/lib.rs (2)

23-131: LGTM! Documentation updates align with SOE terminology.

The documentation changes consistently reflect the SOE (Single Object Encoding) terminology and provide clear examples for both OCF and SOE usage patterns.

---

198-212: Clarify the purpose and usage of the new AvroFieldExt trait.

The trait appears to wrap an existing method on codec::AvroField without adding clear abstraction value. Consider:

• Documentation gap: The trait lacks usage examples or guidance on when consumers should use it versus calling codec::AvroField::with_utf8view directly.
• Design rationale: If this trait is intended to extend a type from a dependency or provide a stable public API over internal implementation, document this intent.
• Example needed: Add a doctest showing typical usage scenarios.

Can you clarify:

1. Is this trait exposing previously-internal functionality to users?
2. Why wrap the existing method instead of re-exporting it directly?
3. Should the trait be used in the SOE examples added elsewhere in this file?

arrow-avro/src/writer/mod.rs (3)

29-48: LGTM! Clear and accurate documentation updates.

The documentation correctly describes the SOE, Confluent, and Apicurio wire format prefixes, providing clear guidance on when to use each format.

---

71-71: Consistent rename to AvroSoeFormat throughout the module.

The type alias, impl blocks, and constructor all correctly reference AvroSoeFormat. The change maintains backward compatibility for users of the AvroStreamWriter type alias.

Also applies to: 265-265, 303-334

---

504-504: Test updates correctly use AvroSoeFormat.

All test cases have been updated to use the new format type consistently.

Also applies to: 538-538, 1401-1401

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[claude]

Pull Request Review

Thank you for this PR! This is a well-executed refactoring that improves code clarity and documentation. Here's my detailed review:

---

✅ Code Quality & Best Practices

Strengths:

• Excellent naming improvement: Renaming AvroBinaryFormat → AvroSoeFormat significantly improves code clarity. The new name accurately reflects that this format uses Single Object Encoding (SOE), not just "binary" data.
• Consistent renaming: All references have been properly updated across:
  • Type definition in format.rs:115
  • Type alias in mod.rs:265
  • All test cases and documentation examples
  • Reader module documentation
• Documentation improvements: The updated docs are more precise and informative:
  • Clarified that SOE writer adds per-record prefixes automatically
  • Better distinction between OCF, SOE, Confluent, and Apicurio formats
  • Added "runnable" markers to examples

Minor observations:

• Line format.rs:128-129: The blank line removal is fine but purely cosmetic
• The removed Confluent example from lib.rs appears to be intentional consolidation, which is good for reducing duplication

---

🔍 Potential Issues

None identified. The changes are:

• Purely additive/refactoring (no behavioral changes)
• Backward compatible through the type system (compile-time errors guide users)
• Correctly applied across all usages

---

⚡ Performance Considerations

No performance impact. This is a pure refactoring:

• Type renaming has zero runtime cost
• Documentation changes don't affect compiled code
• Test updates maintain identical test coverage

---

🔒 Security Concerns

None. The changes are documentation and naming only—no modifications to:

• Serialization/deserialization logic
• Fingerprint calculations
• Schema validation
• Data encoding/decoding

---

🧪 Test Coverage

Excellent test coverage maintained:

• ✅ Tests updated in mod.rs:504 for Confluent wire format with AvroSoeFormat
• ✅ Tests updated in mod.rs:538 for Apicurio wire format with AvroSoeFormat
• ✅ Tests updated in mod.rs:1401 for capacity configuration with AvroSoeFormat
• ✅ All reader module documentation examples updated to use new type name

Recommendation:
Consider running the full test suite to ensure all doctests pass:

cargo test --package arrow-avro --doc

---

📝 Additional Observations

1. Breaking change handled correctly: This is a breaking API change, but it's the right kind—users will get clear compile errors pointing them to the new name. The fix is trivial (s/AvroBinaryFormat/AvroSoeFormat/g).

2. Documentation clarity: The updated mod.rs:37-48 documentation is particularly good—it clearly explains the three different wire formats (SOE/Confluent/Apicurio) and which FingerprintStrategy to use for each.

3. Removed example: The removed Confluent wire-format example from lib.rs:67-150 seems appropriate since:

   • The functionality is already documented elsewhere
   • The reader module has comprehensive examples
   • Reducing duplication improves maintainability

---

🎯 Summary

Recommendation: ✅ Approve

This is a high-quality refactoring that:

• Improves code readability and maintainability
• Enhances documentation accuracy
• Maintains backward compatibility through compile-time errors
• Preserves all existing functionality and test coverage

Suggested next steps:

1. Verify all doctests pass
2. Consider updating CHANGELOG.md to note the breaking rename
3. Merge once CI passes

Great work on making the codebase more understandable! 🚀

[augmentcode]

Review completed. 1 suggestion posted.

[augmentcode]

Decoder also supports Confluent and Apicurio framed messages; consider saying “consumes Avro‑framed bytes (SOE/Confluent/Apicurio)” instead of only “SOE framed” to avoid misleading readers.

🤖 React with 👍 or 👎 to let us know if the comment was useful.

[martin-augment]

value:good-but-wont-fix; category:documentation; feedback:The AI reviewer is correct that the decoder supports several formats but all of them are Single Object Encoded with custom headers, so there is no need to make this part of the documentation more specific.

[github-actions]

Findings

• Major – arrow-json/src/writer/encoder.rs:412-431: make_encoder no longer matches DataType::BinaryView, so BinaryViewArray now falls through to the new fallback that returns ArrowError::JsonError("Unsupported data type for JSON encoding: BinaryView"). That means any JSON writer call that previously worked with BinaryViewArray (e.g. via BinaryViewBuilder) will now fail at runtime. The original BinaryViewEncoder arm should be restored.
• Major – arrow-json/src/writer/encoder.rs:787-792: MapEncoder::try_new now rejects Utf8View keys. Before this PR, the JSON writer accepted Utf8View maps; after the change it returns ArrowError::JsonError("Only UTF8 keys supported…"). This regresses support for view-based string maps—please keep accepting Utf8View.
• Major – parquet-variant-compute/src/variant_to_arrow.rs:196-281: make_primitive_variant_to_arrow_row_builder dropped the explicit arms for DataType::Null and DataType::Time64(TimeUnit::Microsecond). Both types now hit the error branches at lines 272-281 (NotYetImplemented / InvalidArgumentError), whereas previously they built valid arrays via VariantToNullArrowRowBuilder and the Time builder. This breaks typed-value projections that request those output types; the dedicated builders need to be reinstated.

Summary

• JSON writers lost support for BinaryViewArray and Utf8View map keys, and variant→arrow conversion no longer handles Null or Time64 outputs—these were working before and should be restored.

Next Steps

1. Reintroduce the BinaryView match arm (and associated encoder/tests) in the JSON writer.
2. Allow Utf8View keys again in MapEncoder::try_new.
3. Restore the Null and Time64 builders in variant_to_arrow.