refactor: load table state lazily

[wjones127]

Right now, when we instantiate a Delta table, we load the entire table state into memory. For many workloads, we often don't need to have all of it, especially if we are only querying certain partitions at a time. Instead, we should ignore the add and remove actions when instantiating, and only load them as needed during scans.

When we read the log files, we should cache them on disk so we can quickly scan them again later for add and remove actions.

Instantiate table

1. Verify _delta_log exists
2. Either identify most recent version, or verify version requested exists
3. Download relevant log files
4. Scan for the table-level actions, such as metadata and protocol.

This gives you an instance of DeltaTable you can inspect and get table-level metadata. When asked for files, will run scan process below.

Scan table

1. Using the partition filter, scan the log files to get the add and remove actions that are relevant.
2. Use the actions to generate the file set
3. Prune the files based on stats
4. Scan all the files.

This kind of operation will need to run any time asked for a files.

In-memory caching

Below a certain threshold (which we could make configurable), it's probably fine to keep all the table state in-memory. So we might find some data structure that we can use as a memory-limited in-memory cache for table state. 🤔