[Prototype][SPARK-53751][SDP] Explicit Checkpoint Location

[JiaqiWang18]

What changes were proposed in this pull request?

Prototype PR for explicit checkpoint location that follows below format:

checkpoints-root/
      ├── myst/                            # Table "myst"
      │    ├── flow1/                      # Flow to myst
      │    │    ├── 0/                    # Versioned checkpoint (0)
      │    │    │    ├── commits/
      │    │    │    ├── offsets/
      │    │    │    └── sources/
      │    │    └── 1/                    # Versioned checkpoint (1)
      │    │
      │    └── flow2/                     # Another flow to myst
      │         ├── 0/                    # Versioned checkpoint (0)
      │         │    ├── commits/
      │         │    ├── offsets/
      │         │    └── sources/
      │         └── 1/                    # Versioned checkpoint (1)
      │
      └── mysink/                         # Sink "mysink"
            └── flowA/                     # Flow to mysink
                 ├── 0/                    # Versioned checkpoint (0)
                 │    ├── commits/
                 │    ├── offsets/
                 │    └── sources/
                 └── 1/                    # Versioned checkpoint (1)

Backend changes should be mostly finalized, all tests should pass.

For user-facing API, use a spark.sql.pipelines.storageRoot for now, as it is easier to revert.

Why are the changes needed?

To unblock development of sinks
Path toward supporting multi-flow

Does this PR introduce any user-facing change?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

[JiaqiWang18]

with explicit storage location for checkpoint, we shouldn't need to create the tables and obtain its path beforehand. resolvedGraph should suffice.

[JiaqiWang18]

this match is to avoid calling TRUNCATE twice for full refresh MVs