.bag2 files? #819
Comments
|
Since no one else is jumping in, I'll add my 2 cents. Since we're still mostly on ROS 1, we typically fragment the bags according to file size while recording them, so during a field experiment we usually limit a bag to a couple of gigs. Our rationale is:
I think ROS2's switch to sqlite on the backend makes point 3 less important, but the first two points still stand. I do think the single container helps when postprocessing, because it's a lot more convenient to pass in a single large file to an analysis or visualization script. It's also a lot more convenient for sharing, as you pointed out. |
|
I'm not opposed to this idea in principle - though I'd want to make sure we get the user experience right without complicating the options too much, and still support the other format. The sqlite3 databases already encode the contents of the |
|
Agreed. I think I'd want at least these two options.
The rationale for the first one is I can see users wanting a simple command to get this behavior at recording time. The second case would be for something like my usage pattern, where I have my field data fragmented into a bunch of files, but want an easy way to upload it to a colleague or simplify post-processing and analysis of it. Is it possible to get this kind of behavior with our current set of command line arguments and utilities? |
@emersonknapp Does this mean that |
|
@chaoflow As far as I am understanding concept behind It's comfy when you deal with multiple recordings and want to know what inside of these recordings without loading full stack, - you can simply open |
|
I see the argument of sharing (uploading/downloading) a bag file and thus have a single file containing all the info. I would say no to making it a single file by default for reasons being mentioned in this thread here already - i.e. composing various individual bag files via the |
|
I only just came across this thread, so cross referencing it with #855 since some similar discussions are happening over there. |
|
Quite some time since the last comment, but I think that the new way of having a folder is quite painful to use. It is not that bad, but another minor inconvenience in ROS2. Many file formats (e.g. epub, apk, ...) consist of an internal folder structure that is hidden by a tar or zip archive. Accessing the metadata is quite easy as one just needs to grab an archive tool of his choice and view the metadata files. In rosbag2 it is the other way around. Everybody needs to grab such a tool if the bag should be transferred/downloaded/accessed. All of this just for the sake of easier access to the metadata. I hope nobody takes that personally, overall I think that e.g. sqlite is a great choice for the backend and appreciate the work that went into this. This is just some feedback from somebody who uses ros bags a lot and is not that involved in the development. |
|
The MCAP storage plugin (https://github.com/ros-tooling/rosbag2_storage_mcap) is ready for testing. Once it has a few users that can confirm it's working for their day-to-day usage, we'll publish to rolling then humble as an apt package and close this issue out. |
|
We took the ideas from this thread, #782, #855, and several others and incorporated them into the design of MCAP, a highly efficient, self-contained recording format for ROS 2 and more. A ROS 2 MCAP storage plugin is available for Foxy, Galactic, Humble and Rolling, and using it is as simple as Therefore, I think we should close this issue now. |
|
Sorry if this is described somewhere, but I couldn't find it in the
documentation or blog posts about this. Does MCAP rely on having the
message definitions locally for embedding them in the file? In other words,
if I attach my computer to a ROS2 system and I don't have the message
definitions available on my computer, can I still record using MCAP and end
up with a file with those definitions embedded in it?
…On Tue, Sep 13, 2022, 1:01 AM Adrian Macneil ***@***.***> wrote:
We took the ideas from this thread, #855
<#855>, and several others and
incorporated them into the design of MCAP <https://mcap.dev/>, a highly
efficient, self-contained recording format for ROS 2 and more.
A ROS 2 MCAP storage plugin
<https://github.com/ros-tooling/rosbag2_storage_mcap> is available for
Foxy, Galactic, Humble and Rolling, and using it is as simple as ros2 bag
record -s mcap.
Therefore, I think we should close this issue now.
—
Reply to this email directly, view it on GitHub
<#819 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABBZWHFE6O3PMWYNQS72G6DV6AKE7ANCNFSM5ADXXCPA>
.
You are receiving this because you commented.Message ID:
***@***.***>
|
|
@danthony06 ROS 2 in general does not support what you are describing. In order to connect and communicate with a remote ROS 2 node, you must have the message definitions locally. This is a regression from ROS 1 and discussed here. Assuming you do have a copy of the message definitions available locally, MCAP will find them and store in the file for future use. |
|
I am going to close this as implemented - we now allow for playing self-contained ROS 2 individual bag files. There may be further related features, like the discussion about message types above, but that is beyond the scope of this ticket and should be tracked independently. |
|
Understood that there's nothing to do on your end. I'm a little frustrated
losing this capability in general, but that's not a shortcoming of MCAP.
…On Tue, Sep 13, 2022, 9:50 AM Adrian Macneil ***@***.***> wrote:
@danthony06 <https://github.com/danthony06> ROS 2 in general does not
support what you are describing. In order to connect and communicate with a
remote ROS 2 node, you must have the message definitions locally. This is a
regression from ROS 1 and discussed here
<ros2/ros2#1159>.
Assuming you do have a copy of the message definitions available, MCAP
will find them and store in the file for future use.
—
Reply to this email directly, view it on GitHub
<#819 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABBZWHEQAASLWB43V5EHQUDV6CIDLANCNFSM5ADXXCPA>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
@danthony06 FYI there is some other work being done by open robotics which will hopefully give us runtime message definitions for the next major version of ROS 2 (Iron). |
|
Closing this issue was not meant to imply #782 is also closed, it is still open - we can maybe move any discussion on that particular feature to that issue, so as not to lose any extra context! My understanding of the scope of this issues is "rosbag2 can use standalone files", which is now a capability. Best not to pack too much into an individual issue |
Description
What do people think about introducing the option of a single file container for ROS2 bags? I'm thinking something like a tar file with the extension
.bag2since compression is handled at a lower level, but I mostly just want to start the discussion and get a temperature read.The practical use cases are around storage and sharing. Uploading a single file for cloud storage can be an atomic operation (it's either uploaded or not, downstream processing actions can be triggered off the upload successfully completing) and verifying the upload succeeded is a more straightforward hash check of a single file where uploading a folder gets more into rsync territory. When sharing files with coworkers or on the Internet, it's convenient to post a single file for download instead of figuring out how to share a folder full of files (and confirm that you're received all of the files on the receiving end).
The text was updated successfully, but these errors were encountered: