feat: sharable reference to storage backend

[roeap]

Description

While working on #632 i saw that a shareable storage backend is also required for #690 - as such i decided to move this to a separate PR, to get things stared for #689.

Related Issue(s)

Documentation

[wjones127]

Thank you for pulling this out into a separate PR. 🙌

This looks great to me!

[houqp]

In the long run, I wonder if it would be better to add a Clone trait constraint to the StorageBackend trait instead.

[wjones127]

In the long run, I wonder if it would be better to add a Clone trait constraint to the StorageBackend trait instead.

Is the expectation there that internally they might use an Arc so that cloning is cheap? Not sure which way is more idiomatic in Rust.

[houqp]

Is the expectation there that internally they might use an Arc so that cloning is cheap? Not sure which way is more idiomatic in Rust.

I think using Arc internally would be the last resort if the upstream sdk client doesn't implement Clone because it incurs extra runtime overheads. All of our current storage backends should be able to derive Clone without using Arcs from a quick glance. Most of the derived clone implementation should be very cheap and if it is not it shouldn't be a problem assuming storage backend clone is a very infrequent operation. On the other hand, accessing the backend itself for read/write is a frequent runtime operation, so it's better to not force the atomic reference counter overhead everywhere only because we need infrequent cheap clones.

If clone is a frequent runtime operation, then wrapping all backend with an Arc is likely the right design choice.

We even have a very old task that talked about getting rid of the trait object constraint for storage backend entirely in #257 :)

[wjones127]

On the other hand, accessing the backend itself for read/write is a frequent runtime operation, so it's better to not force the atomic reference counter overhead everywhere only because we need infrequent cheap clones.

I was under the impression the increment / decrement overhead just happened in Arc::clone() and Arc::drop(), and thus we shouldn't have any overhead within objects, just a little bit when we are creating new structs where we need to pass a reference to the filesystem. Is it called also when calling methods of T on an Arc<T>?

[houqp]

I was under the impression the increment / decrement overhead just happened in Arc::clone() and Arc::drop(), and thus we shouldn't have any overhead within objects,

I was wrong and you are right in this case @wjones127 :) Accessing the heap object should have no overhead, Arc's overhead only comes from clone and drop. So it won't make a difference for us compared to Rc because clone is a very infrequent operation.