CatalogPlugin, TableCatalog and loading a table

Author: jaceklaskowski

docs/connector/catalog/CatalogPlugin.md

@@ -1,10 +1,13 @@
 # CatalogPlugin
 
-`CatalogPlugin` is an [abstraction](#contract) of [table catalogs](#implementations).
+`CatalogPlugin` is an [abstraction](#contract) of [catalogs](#implementations) (of assets like [functions](FunctionCatalog.md), [tables](TableCatalog.md) and [views](ViewCatalog.md)).
 
 !!! note "CatalogHelper"
     [CatalogHelper](CatalogHelper.md) is a Scala implicit class of `CatalogPlugin` with extensions methods.
 
+!!! note "Demo"
+    Learn more in [Demo: Developing CatalogPlugin](../../demo/developing-catalogplugin.md).
+
 ## Contract
 
 ### Default Namespace { #defaultNamespace }
@@ -53,7 +56,3 @@ String name()
 * [SupportsNamespaces](SupportsNamespaces.md)
 * [TableCatalog](TableCatalog.md)
 * [ViewCatalog](ViewCatalog.md)
-
-## Demo
-
-Learn more in [Demo: Developing CatalogPlugin](../../demo/developing-catalogplugin.md).

docs/connector/catalog/CatalogV2Util.md

@@ -1,18 +1,19 @@
 # CatalogV2Util
 
-## Loading Table { #loadTable }
+## Load Table { #loadTable }
 
 ```scala
 loadTable(
   catalog: CatalogPlugin,
   ident: Identifier,
-  timeTravelSpec: Option[TimeTravelSpec] = None): Option[Table]
+  timeTravelSpec: Option[TimeTravelSpec] = None,
+  writePrivilegesString: Option[String] = None): Option[Table]
 ```
 
-`loadTable` [loads the table](#getTable) (by the given identifier and the optional [TimeTravelSpec](../../time-travel/TimeTravelSpec.md) in the given [CatalogPlugin](CatalogPlugin.md)).
+`loadTable` [loads a table](#getTable) (by the given `Identifier` and the optional [TimeTravelSpec](../../time-travel/TimeTravelSpec.md) using the given [CatalogPlugin](CatalogPlugin.md)).
 
 !!! note
-    `loadTable` is a Scala `Option`-aware wrapper around [CatalogV2Util.getTable](#getTable) that may not only return `null` but also throw an exception.
+    `loadTable` is a Scala `Option`-aware wrapper around [CatalogV2Util.getTable](#getTable).
 
 ---
 
@@ -22,28 +23,36 @@ loadTable(
 * `CatalogV2Util` is requested to [loadRelation](#loadRelation)
 * `CatalogImpl` is requested to [load a table](../../CatalogImpl.md#loadTable)
 
-## getTable { #getTable }
+## Load Table { #getTable }
 
 ```scala
 getTable(
   catalog: CatalogPlugin,
   ident: Identifier,
-  timeTravelSpec: Option[TimeTravelSpec] = None): Table
+  timeTravelSpec: Option[TimeTravelSpec] = None,
+  writePrivilegesString: Option[String] = None): Table
 ```
 
+`getTable` assumes the given [CatalogPlugin](CatalogPlugin.md) to be a [TableCatalog](CatalogHelper.md#asTableCatalog) to [load a table](TableCatalog.md#loadTable) (by the given `Identifier`).
+
+---
+
 `getTable` requests the given [CatalogPlugin](CatalogPlugin.md) for the [TableCatalog](CatalogHelper.md#asTableCatalog) to [load a table](TableCatalog.md#loadTable) (possibly versioned based on the [TimeTravelSpec](../../time-travel/TimeTravelSpec.md)).
 
+!!! note
+    It is not allowed for `getTable` to be called with both `timeTravelSpec` and `writePrivilegesString` defined.
+
 !!! note "NoSuchTableException for versioned tables"
     [TableCatalog](TableCatalog.md) throws a `NoSuchTableException` exception for versioned tables by default (and leaves other behaviour to custom [TableCatalog](TableCatalog.md#implementations)s, e.g. [Delta Lake]({{ book.delta }}/DeltaCatalog)).
 
 ---
 
 `getTable` is used when:
 
-* `CatalogV2Util` is requested to [loadTable](#loadTable)
+* `CatalogV2Util` is requested to [load a table](#loadTable)
 * `DataSourceV2Utils` is requested to [loadV2Source](../../connectors/DataSourceV2Utils.md#loadV2Source)
 
-## <span id="getTableProviderCatalog"> getTableProviderCatalog
+## getTableProviderCatalog { #getTableProviderCatalog }
 
 ```scala
 getTableProviderCatalog(

docs/connector/catalog/TableCatalog.md

@@ -61,10 +61,10 @@ Identifier[] listTables(
   String[] namespace)
 ```
 
-Used when the following commands are executed:
+Used when:
 
-* [DropNamespaceExec](../../physical-operators/DropNamespaceExec.md)
-* [ShowTablesExec](../../physical-operators/ShowTablesExec.md)
+* `DelegatingCatalogExtension` is requested to [listTables](DelegatingCatalogExtension.md#listTables)
+* [ShowTablesExec](../../physical-operators/ShowTablesExec.md) is executed
 
 ### Load Table { #loadTable }
 
@@ -74,19 +74,30 @@ Table loadTable(
 Table loadTable(
   Identifier ident,
   long timestamp)
+Table loadTable(
+  Identifier ident,
+  Set<TableWritePrivilege> writePrivileges) // (1)!
 Table loadTable(
   Identifier ident,
   String version)
 ```
 
+1. Added in 3.5.3
+
 Used when:
 
 * `CatalogV2Util` is requested to [load a table](CatalogV2Util.md#getTable)
-* `DataFrameReader` is requested to [load](../../DataFrameReader.md#load) (for [SupportsCatalogOptions](SupportsCatalogOptions.md) providers)
-* `DataFrameWriter` is requested to [save](../../DataFrameWriter.md#save), [insertInto](../../DataFrameWriter.md#insertInto) and [saveAsTable](../../DataFrameWriter.md#saveAsTable)
-* `DelegatingCatalogExtension` is requested to [loadTable](DelegatingCatalogExtension.md#loadTable)
-* `TableCatalog` is requested to [tableExists](#tableExists)
-* `V2SessionCatalog` is requested to [createTable](../../V2SessionCatalog.md#createTable), [alterTable](../../V2SessionCatalog.md#alterTable), [dropTable](../../V2SessionCatalog.md#dropTable), [renameTable](../../V2SessionCatalog.md#renameTable)
+* `DataFrameWriter` is requested to [insertInto](../../DataFrameWriter.md#insertInto), [saveInternal](../../DataFrameWriter.md#saveInternal), [saveAsTable](../../DataFrameWriter.md#saveAsTable)
+* `AtomicCreateTableAsSelectExec` physical operator is executed
+* `AtomicReplaceTableAsSelectExec` physical operator is executed
+* [CreateTableAsSelectExec](../../physical-operators/CreateTableAsSelectExec.md) physical operator is executed
+* `ReplaceTableAsSelectExec` physical operator is executed
+* `DelegatingCatalogExtension` is requested to [load a table](DelegatingCatalogExtension.md#loadTable)
+* `RenameTableExec` physical operator is executed
+* `AtomicReplaceTableExec` physical operator is executed
+* `ReplaceTableExec` physical operator is executed
+* `V2SessionCatalog` is requested to [dropTableInternal](../../V2SessionCatalog.md#dropTableInternal), [loadTable](../../V2SessionCatalog.md#loadTable), [dropTable](../../V2SessionCatalog.md#dropTable), [renameTable](../../V2SessionCatalog.md#renameTable)
+* `DataStreamWriter` is requested to `toTable`
 
 ### Rename Table { #renameTable }
 

docs/connectors/DataSourceV2Utils.md

@@ -1,8 +1,12 @@
+---
+title: DataSourceV2Utils
+---
+
 # DataSourceV2Utils Utility
 
 `DataSourceV2Utils` is an utility to [extractSessionConfigs](#extractSessionConfigs) and [getTableFromProvider](#getTableFromProvider) for batch and streaming reads and writes.
 
-## <span id="extractSessionConfigs"> extractSessionConfigs
+## extractSessionConfigs { #extractSessionConfigs }
 
 ```scala
 extractSessionConfigs(
@@ -24,7 +28,7 @@ extractSessionConfigs(
 * (Spark Structured Streaming) `DataStreamReader` is requested to load data from a streaming data source
 * (Spark Structured Streaming) `DataStreamWriter` is requested to start a streaming query
 
-## <span id="getTableFromProvider"> Creating Table (using TableProvider)
+## Creating Table (using TableProvider) { #getTableFromProvider }
 
 ```scala
 getTableFromProvider(
@@ -44,7 +48,7 @@ getTableFromProvider(
 * `DataStreamReader` ([Spark Structured Streaming]({{ book.structured_streaming }}/DataStreamReader)) is requested to load data from a streaming data source
 * `DataStreamWriter` ([Spark Structured Streaming]({{ book.structured_streaming }}/DataStreamWriter)) is requested to start a streaming query
 
-## <span id="loadV2Source"> loadV2Source
+## Load V2 Source { #loadV2Source }
 
 ```scala
 loadV2Source(
@@ -56,11 +60,15 @@ loadV2Source(
   paths: String*): Option[DataFrame]
 ```
 
-`loadV2Source` creates a [DataFrame](../DataFrame.md).
+`loadV2Source` [extractSessionConfigs](DataSourceV2Utils.md#extractSessionConfigs) and [adds the given paths](#getOptionsWithPaths) if specified.
+
+For the given [TableProvider](../connector/TableProvider.md) being a [SupportsCatalogOptions](../connector/catalog/SupportsCatalogOptions.md), `loadV2Source`...FIXME
+
+In the end, for a [SupportsRead](../connector/SupportsRead.md) table with [BATCH_READ](../connector/TableCapability.md#BATCH_READ) capability, `loadV2Source` creates a `DataFrame` with [DataSourceV2Relation](../logical-operators/DataSourceV2Relation.md#create) logical operator. Otherwise, `loadV2Source` gives no `DataFrame` (`None`).
 
 ---
 
 `loadV2Source` is used when:
 
-* `DataFrameReader` is requested to [load data](../DataFrameReader.md#load)
+* [DataFrameReader.load](../DataFrameReader.md#load) operator is used
 * [CreateTempViewUsing](../logical-operators/CreateTempViewUsing.md) logical operator is executed