Allow access to underlying storage backend

[wjones127]

Description

We currently don't allow external access to the underlying storage of a DeltaTable. But for the Python bindings we will need access to be able to pass the storage backend to other functions that need to do IO (see #689).

The type is currently Box<dyn StorageBackend>. We could instead making it Arc<dyn StorageBackend>, and then can allow users to retrieve the backend by cloning the Arc pointer. StorageBackend is already Send + Sync, so I don't think there should be any problem allowing shared ownership.

There are plenty of places where we create a new storage backend from a URI and storage options, when we really should just be taking an Arc<StorageBackend>. Here's a good example, IMO:

delta-rs/rust/src/writer/json.rs

Lines 182 to 190 in 58b2629

	/// Create a new JsonWriter instance
	pub fn try_new(
	table_uri: String,
	schema: ArrowSchemaRef,
	partition_columns: Option<Vec<String>>,
	storage_options: Option<HashMap<String, String>>,
	) -> Result<Self, DeltaWriterError> {
	let storage =
	get_backend_for_uri_with_options(&table_uri, storage_options.unwrap_or_default())?;

Use Case

In Python, we need to extract the storage backend so we can pass the filesystem to PyArrow functions such as the dataset scanner.

Related Issue(s)

• This blocks fixing Support get_backend_for_uri_with_options in Read Path #689