Seanaye/persistence trait

[seanaye]

Description

This MR introduces a Persistence trait which abstracts away the persistence layer for Completed bao files. This means that you can BYO persistence impl to the current fs store (e.g. S3).

Breaking Changes

The external facing breaking changes are mostly around the "asyncification" of certain trait methods. I will update this section in more detail later.

Notes & open questions

This is still a draft MR I am just looking for feedback on the general approach before I clean things up a bit more. e.g. There are some places that can panic right now

Change checklist

• Self-review.
• Documentation updates following the style guide, if relevant.
• Tests if relevant.
• All breaking changes documented.

[rklaehn]

Thanks for the PR. I will have to check if the storage abstraction adds performance costs for the very important local/sync case.

It might be that the best way to deal with this use case is to have a separate store implementation that allows plugging in an async storage backend, since the local store is very performance critical.

The way I usually test this is by using sendme and then syncing a directory structure that contains a lot of tiny files, like the linux kernel.

Warning: I got some important deadlines the next weeks and won't be able to look into this until after may 18.

[seanaye]

Thanks for taking a look, I will test with sendme. Is there a specific command you use to benchmark the impl? I still have a few commits to push