Rust: Port anduril/mcap-rs #611
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
|
mrkline
force-pushed
the
anduril-rust
branch
3 times, most recently
from
9f4265f to
247221d
Compare
james-rms
reviewed
Sep 26, 2022
james-rms
reviewed
Sep 26, 2022
- Make field names match. - Extract chunks. - Flatten record headers. - Skip data_len header fields
Duplicated data invites inconsistency - the slices we return for schemas, messages, and attachments know their length. Inside the codec, just serialize the length fields separately.
mrkline
force-pushed
the
anduril-rust
branch
from
247221d to
aef7062
Compare
Discuss tradeoffs of memory-mapped vs streaming reads.
james-rms
reviewed
Sep 28, 2022
rust/README.md
Outdated
| @@ -1,31 +1,18 @@ | |||
| # Rust MCAP reading library | |||
| # mcap-rs | |||
There was a problem hiding this comment.
Yep! Thanks, fixed in ae22506
james-rms
reviewed
Sep 28, 2022
| } | ||
|
|
||
| // Hard mode: randomly access every message in the MCAP. | ||
| // Yes, this is dumb and O(n^2). |
There was a problem hiding this comment.
7 minute CI isn't a terrible situation - with this test rust is the long pole, taking the crown from C++ (5 mins). I think we can speed this up a little by writing a custom MCAP with only 10 messages as part of this test instead of using demo.mcap. Happy to do that as a separate PR.
mrkline
force-pushed
the
anduril-rust
branch
from
14c345c to
ae22506
Compare
james-rms
approved these changes
Sep 28, 2022
There was a problem hiding this comment.
Re: serde as an optional dependency, I think this is impossible without duplicating a large amount of record struct declaration. Probably the right path there is to just write a manual transformation from serde_json::Value objects to Records.
Closed
|
👏 🥳 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Public-Facing Changes
Description
Adds a more fully-featured Rust library that supports indexed reading, CRC validation, record writing, chunking, and automatic summary writing.
TODOs
serdean optional dependency that we only pull in for the conformance reader. (I can't imagine users wanting to serialize random chunks of an MCAP file to JSON/CBOR/Bincode/etc.)tests/round_trip.rs) checks the message indexes by individually seeking each message from its respective chunk. This is... slow. Should we axe that part of the test to avoid bogging down CI?