Onda Format v0.5.0

[jrevels]

• implements explore usage of arrow files instead of msgpack.zst #25 (switched from MessagePack to tabular metadata representation w/ Arrow files)
• implements allow sample data and non-Onda-specified metadata artifacts to be addressed by URI rather than relative location #22 (provided explicit file_path column for signals table)
• implements change name of "recordings.msgpack.zst" to "manifest.onda" #24 "in spirit" (*.onda.annotations.arrow and *.onda.signals.arrow files can have arbitrary file names)
• implements support float32/float64 sample_type fields #26 (float32/float64 sample type support)
• replaces wonky file_extension/file_options fields with new file_format column to go along with the new file_path column
• minor touch ups to a couple of sections to improve clarity/provide detail.

Summarizing some of the high-level impact of these changes:

• Users may now separately construct/manipulate/transfer/analyze annotations (in *.onda.annotations.arrow files) and signals (in *.onda.signals.arrow files).
• Users may work with multiple files of the same type referencing the same set of recordings (e.g. I could splitting all of a dataset's annotations across multiple *.onda.annotations.arrow files, or write new signals to an existing recording w/o modifying existing files).
• Users may now leverage rich tooling ecosystems for tabular data manipulation widely available across a variety of languages to manipulate/analyze signal/annotation metadata.
• (De)serializing/reading Onda metadata can now be made quite a bit faster/mmap-able.
• Users may now store sample data in decoded floating point format.
• Users are no longer required to maintain a relationship between metadata and sample data locations; arbitrary URIs (e.g. S3 URIs) can be used to directly associate signals with sample data files. Relative paths may still be used when desirable.
• IMHO, on top of all of the above, the new format is actually clearer/cleaner/easier to understand 😎

I'm currently preparing a corresponding PR to Onda.jl 🙂

[palday]

A few small ambiguities and some formulation tweaks, but two bigger comments

1. a lot of my comments on "uh, this could be confusing for people of a certain background" doesn't need any changes in this, but might be a great candidate for something akin to an FAQ or documentation elsewhere.
2. what you call "interleaved" reminds me a lot of what is called "multiplexed" in EEG formats. I haven't thought hard enough about it to know if they are actually the same or whether we're abusing DSP terms (like when non math people use "group" and "set" interchangeably), but maybe @hannahilea knows better?

But yeah, I think Arrow makes everything better.

[palday]

even better.

[jrevels]

what you call "interleaved" reminds me a lot of what is called "multiplexed" in EEG formats. I haven't thought hard enough about it to know if they are actually the same or whether we're abusing DSP terms (like when non math people use "group" and "set" interchangeably), but maybe @hannahilea knows better?

I think it's basically the same IIUC. FWIW I tend to use multiplex more as a verb and interleave more as a noun (e.g. "this service multiplexes incoming streams into a single interleaved output stream"). Interleaved (e.g. vs. planar) is what's commonly used in codec terminology.

One subtle terminology difference might arise if you e.g. multiplexed incoming streams into a chunked planar format, rather than a fully interleaved format.

[palday]

One subtle terminology difference might arise if you e.g. multiplexed incoming streams into a chunked planar format, rather than a fully interleaved format.

And I think some of the formats do this. 🤷