Skip to content

Commit

Permalink
rust: write attachments in parts (#1285)
Browse files Browse the repository at this point in the history
### Changelog

<!-- Write a one-sentence summary of the user-impacting change (API,
UI/UX, performance, etc) that could appear in a changelog. Write "None"
if there is no user-facing change -->

rust: support writing attachments in multiple chunks

### Docs

<!-- Link to a Docs PR, tracking ticket in Linear, OR write "None" if no
documentation changes are needed. -->

None

### Description

This change supports writing large attachments to MCAP files without the
need to load them into memory.

Currently, in order to write an attachment, you need to load the entire
file into memory, create an `Attachment` record, then use that with the
`attach` method on the writer.
This means that it is not possible to attach files when the file exceeds
the available memory on the computer.

This change adds The methods `start_attachment`, `put_attachment_bytes`
and `finish_attachment` for interactively writing an attachment without
needing to buffer the file in memory. It looks like this:

```rust
let writer = Writer::new(..);
let attachment_length = ..;
let header = AttachmentHeader { .. };

writer.start_attachment(attachment_length, header)?;
writer.put_attachment_bytes(..)?;
writer.put_attachment_bytes(..)?;
writer.put_attachment_bytes(..)?;
writer.finish_attachment()?;
```

I've updated the current `attach` method to use this approach and the
roundtrip tests pass. I've also added some tests for using this API
directly.

<!-- Describe the problem, what has changed, and motivation behind those
changes. Pretend you are advocating for this change and the reader is
skeptical. -->

<!-- In addition to unit tests, describe any manual testing you did to
validate this change. -->

<table><tr><th>Before</th><th>After</th></tr><tr><td>

attachments must be loaded into memory to be written

<!--before content goes here-->

</td><td>

attachments can be written in parts

<!--after content goes here-->

</td></tr></table>

<!-- If necessary, link relevant Linear or Github issues. Use `Fixes:
foxglove/repo#1234` to auto-close the Github issue or Fixes: FG-### for
Linear isses. -->

---------

Co-authored-by: james-rms <james@foxglove.dev>
  • Loading branch information
bennetthardwick and james-rms authored Dec 6, 2024
1 parent 44de986 commit d42ed24
Show file tree
Hide file tree
Showing 4 changed files with 313 additions and 55 deletions.
2 changes: 1 addition & 1 deletion rust/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ categories = [ "science::robotics", "compression" ]
repository = "https://github.com/foxglove/mcap"
documentation = "https://docs.rs/mcap"
readme = "README.md"
version = "0.12.1"
version = "0.13.0"
edition = "2021"
license = "MIT"

Expand Down
8 changes: 8 additions & 0 deletions rust/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -89,6 +89,14 @@ use thiserror::Error;

#[derive(Debug, Error)]
pub enum McapError {
#[error("tried to write to output while attachment is in progress")]
AttachmentInProgress,
#[error("tried to write bytes to an attachment but no attachment was in progress")]
AttachmentNotInProgress,
#[error("tried to write {excess} more bytes to attachment than the requested attachment length {attachment_length}")]
AttachmentTooLarge { excess: u64, attachment_length: u64 },
#[error("tried to finish writing attachment but current length {current} was not expected length {expected}")]
AttachmentIncomplete { current: u64, expected: u64 },
#[error("Bad magic number")]
BadMagic,
#[error("Footer record couldn't be found at the end of the file, before the magic bytes")]
Expand Down
Loading

0 comments on commit d42ed24

Please sign in to comment.