diff --git a/website/www/site/content/en/documentation/io/managed-io.md b/website/www/site/content/en/documentation/io/managed-io.md index 48cf2a28addb..ced0443c6954 100644 --- a/website/www/site/content/en/documentation/io/managed-io.md +++ b/website/www/site/content/en/documentation/io/managed-io.md @@ -102,6 +102,7 @@ and Beam SQL is invoked via the Managed API under the hood. catalog_name (str)
catalog_properties (map[str, str])
config_properties (map[str, str])
+ direct_write_byte_limit (int32)
drop (list[str])
keep (list[str])
only (str)
@@ -185,7 +186,7 @@ and Beam SQL is invoked via the Managed API under the hood. - MYSQL + SQLSERVER jdbc_url (str)
connection_init_sql (list[str])
@@ -219,7 +220,7 @@ and Beam SQL is invoked via the Managed API under the hood. - SQLSERVER + MYSQL jdbc_url (str)
connection_init_sql (list[str])
@@ -654,6 +655,17 @@ and Beam SQL is invoked via the Managed API under the hood. Properties passed to the Hadoop Configuration. + + + direct_write_byte_limit + + + int32 + + + For a streaming pipeline, sets the limit for lifting bundles into the direct write path. + + drop @@ -1046,7 +1058,7 @@ For more information on table properties, please visit https://iceberg.apache.or -### `POSTGRES` Read +### `POSTGRES` Write
@@ -1063,73 +1075,73 @@ For more information on table properties, please visit https://iceberg.apache.or str @@ -1151,184 +1163,184 @@ For more information on table properties, please visit https://iceberg.apache.or str +
- Connection URL for the JDBC source. + Connection URL for the JDBC sink.
- connection_init_sql + autosharding - list[str] + boolean - Sets the connection init sql statements used by the Driver. Only MySQL and MariaDB support this. + If true, enables using a dynamically determined number of shards to write.
- connection_properties + batch_size - str + int64 - Used to set connection properties passed to the JDBC driver not already defined as standalone parameter (e.g. username and password can be set using parameters above accordingly). Format of the string must be "key1=value1;key2=value2;". + n/a
- disable_auto_commit + connection_init_sql - boolean + list[str] - Whether to disable auto commit on read. Defaults to true if not provided. The need for this config varies depending on the database platform. Informix requires this to be set to false while Postgres requires this to be set to true. + Sets the connection init sql statements used by the Driver. Only MySQL and MariaDB support this.
- driver_class_name + connection_properties str - Name of a Java Driver class to use to connect to the JDBC source. For example, "com.mysql.jdbc.Driver". + Used to set connection properties passed to the JDBC driver not already defined as standalone parameter (e.g. username and password can be set using parameters above accordingly). Format of the string must be "key1=value1;key2=value2;".
- driver_jars + driver_class_name str - Comma separated path(s) for the JDBC driver jar(s). This can be a local path or GCS (gs://) path. + Name of a Java Driver class to use to connect to the JDBC source. For example, "com.mysql.jdbc.Driver".
- fetch_size + driver_jars - int32 + str - This method is used to override the size of the data that is going to be fetched and loaded in memory per every database call. It should ONLY be used if the default value throws memory errors. + Comma separated path(s) for the JDBC driver jar(s). This can be a local path or GCS (gs://) path.
- Name of the table to read from. + Name of the table to write to.
- num_partitions + password - int32 + str - The number of partitions + Password for the JDBC source.
- output_parallelization + username - boolean + str - Whether to reshuffle the resulting PCollection so results are distributed to all workers. + Username for the JDBC source.
- partition_column + write_statement str - Name of a column of numeric type that will be used for partitioning. + SQL query used to insert records into the JDBC sink.
+
+ +### `POSTGRES` Read + +
+ + + + + + -
ConfigurationTypeDescription
- password + jdbc_url str - Password for the JDBC source. + Connection URL for the JDBC source.
- read_query + connection_init_sql - str + list[str] - SQL query used to query the JDBC source. + Sets the connection init sql statements used by the Driver. Only MySQL and MariaDB support this.
- username + connection_properties str - Username for the JDBC source. + Used to set connection properties passed to the JDBC driver not already defined as standalone parameter (e.g. username and password can be set using parameters above accordingly). Format of the string must be "key1=value1;key2=value2;".
-
- -### `POSTGRES` Write - -
- - - - - - @@ -1344,30 +1356,30 @@ For more information on table properties, please visit https://iceberg.apache.or
ConfigurationTypeDescription
- jdbc_url + disable_auto_commit - str + boolean - Connection URL for the JDBC sink. + Whether to disable auto commit on read. Defaults to true if not provided. The need for this config varies depending on the database platform. Informix requires this to be set to false while Postgres requires this to be set to true.
- autosharding + driver_class_name - boolean + str - If true, enables using a dynamically determined number of shards to write. + Name of a Java Driver class to use to connect to the JDBC source. For example, "com.mysql.jdbc.Driver".
- batch_size + driver_jars - int64 + str - n/a + Comma separated path(s) for the JDBC driver jar(s). This can be a local path or GCS (gs://) path.
- connection_init_sql + fetch_size - list[str] + int32 - Sets the connection init sql statements used by the Driver. Only MySQL and MariaDB support this. + This method is used to override the size of the data that is going to be fetched and loaded in memory per every database call. It should ONLY be used if the default value throws memory errors.
- connection_properties + jdbc_type str - Used to set connection properties passed to the JDBC driver not already defined as standalone parameter (e.g. username and password can be set using parameters above accordingly). Format of the string must be "key1=value1;key2=value2;". + Type of JDBC source. When specified, an appropriate default Driver will be packaged with the transform. One of mysql, postgres, oracle, or mssql.
- driver_class_name + location str - Name of a Java Driver class to use to connect to the JDBC source. For example, "com.mysql.jdbc.Driver". + Name of the table to read from.
- driver_jars + num_partitions - str + int32 - Comma separated path(s) for the JDBC driver jar(s). This can be a local path or GCS (gs://) path. + The number of partitions
- jdbc_type + output_parallelization - str + boolean - Type of JDBC source. When specified, an appropriate default Driver will be packaged with the transform. One of mysql, postgres, oracle, or mssql. + Whether to reshuffle the resulting PCollection so results are distributed to all workers.
- location + partition_column str - Name of the table to write to. + Name of a column of numeric type that will be used for partitioning.
- username + read_query str - Username for the JDBC source. + SQL query used to query the JDBC source.
- write_statement + username str - SQL query used to insert records into the JDBC sink. + Username for the JDBC source.
-### `MYSQL` Read +### `SQLSERVER` Read
@@ -1544,7 +1556,7 @@ For more information on table properties, please visit https://iceberg.apache.or
-### `MYSQL` Write +### `SQLSERVER` Write
@@ -1688,7 +1700,7 @@ For more information on table properties, please visit https://iceberg.apache.or
-### `SQLSERVER` Read +### `MYSQL` Read
@@ -1865,7 +1877,7 @@ For more information on table properties, please visit https://iceberg.apache.or
-### `SQLSERVER` Write +### `MYSQL` Write