Skip to content

Commit

Permalink
enhance rate limit doc
Browse files Browse the repository at this point in the history 
Signed-off-by: xxchan <xxchan22f@gmail.com>
  • Loading branch information
@xxchan
xxchan committed Nov 19, 2024
1 parent d704083 commit d29346a
Show file tree
Hide file tree
Showing 4 changed files with 22 additions and 11 deletions.
4 changes: 2 additions & 2 deletions operate/view-configure-runtime-parameters.mdx
View file
Original file line number Diff line number Diff line change
Expand Up @@ -67,8 +67,8 @@ Below is the detailed information about the parameters you may see after using t
| cdc\_source\_wait\_streaming\_start\_timeout | 30 | For limiting the startup time of a shareable CDC streaming source when the source is being created. Unit: seconds. |
| row\_security | true/false | See [here](https://www.postgresql.org/docs/current/runtime-config-client.html#GUC-ROW-SECURITY) for details. Unused in RisingWave, support for compatibility. |
| standard\_conforming\_strings | on | See [here](https://www.postgresql.org/docs/current/runtime-config-client.html#GUC-STANDARD-CONFORMING-STRINGS) for details. |
| source\_rate\_limit | default/ A positive integer / 0 | Set the maximum number of records per second per source, for each parallelism. The source here refers to an upstream source. This parameter is applied to tables and tables with sources. The value can be default, 0, or a positive integer. SET SOURCE\_RATE\_LIMIT TO 0 will pause the source read for sources. SET SOURCE\_RATE\_LIMIT TO DEFAULT will disable the rate limit within the session, but it will not change the rate limits of existing DDLs.Note that the total throughput of a streaming job is determined by multiplying the parallelism with the throttle rate. To obtain the parallelism value for a streaming job, you can refer to the streaming\_parallelism runtime parameter in this table. Additionally, we support altering rate limits in [sources](/docs/current/sql-alter-source/#set-source-rate-limit) and [tables that have source](/docs/current/sql-alter-table/#set-source-rate-limit). |
| backfill\_rate\_limit | default/ A positive integer / 0 | Set the maximum number of records per second per parallelism for the backfill process of materialized views, sinks, and indexes. This parameter throttles the snapshot read stream for backfill. The value can be default, 0, or a positive integer. SET BACKFILL\_RATE\_LIMIT TO 0 will pause the snapshot read stream for backfill. SET BACKFILL\_RATE\_LIMIT TO default will disable the backfill rate limit within the session, but it will not change the backfill rate limit of existing DDLs. To obtain the parallelism value for a streaming job, you can refer to the streaming\_parallelism runtime parameter in this table. Additionally, we support altering backfill rate limits in [materialized views](/docs/current/sql-alter-materialized-view/#set-backfill%5Frate%5Flimit) and [CDC tables](/docs/current/sql-alter-table/#set-backfill%5Frate%5Flimit). |
| source\_rate\_limit | default/positive integer/0 | Set the maximum number of records per second per source, for each parallelism. This parameter is applied when creating new sources and tables with sources. <br/><br/> The value can be default, 0, or a positive integer. <br/> SET SOURCE\_RATE\_LIMIT TO 0 will pause the source read for sources. <br/> SET SOURCE\_RATE\_LIMIT TO DEFAULT will remove the rate limit. <br/><br/> Setting this variable will only affect new DDLs within the session, but not change the rate limits of existing jobs. Use `ALTER` to change the rate limits in existing [sources](/sql/commands/sql-alter-source/#set-source-rate-limit) and [tables that have source](/sql/commands/sql-alter-table/#set-source-rate-limit). <br/><br/> Note that the total throughput of a streaming job is determined by multiplying the parallelism with the throttle rate. To obtain the parallelism value for a streaming job, you can refer to the streaming\_parallelism runtime parameter in this table. |
| backfill\_rate\_limit | default/positive integer/0 | Set the maximum number of records per second per parallelism for the backfill process of materialized views, sinks, and indexes. This parameter is applied when creating new jobs, and throttles the backfill from upstream materialized views and sources. <br/><br/> The value can be default, 0, or a positive integer. <br/> SET BACKFILL\_RATE\_LIMIT TO 0 will pause the backfill. <br/> SET BACKFILL\_RATE\_LIMIT TO default will remove the backfill rate limit. <br/><br/> Setting this variable will only affect new DDLs within the session, but not change the rate limits of existing jobs. Use `ALTER` to change the backfill rate limits in existing [materialized views](//sql-alter-materialized-view/#set-backfill%5Frate%5Flimit) and [CDC tables](/sql/commands/sql-alter-table/#set-backfill%5Frate%5Flimit). <br/><br/> Note that the total throughput of a streaming job is determined by multiplying the parallelism with the throttle rate. To obtain the parallelism value for a streaming job, you can refer to the streaming\_parallelism runtime parameter in this table. |
| rw\_streaming\_over\_window\_cache\_policy | full | Cache policy for partition cache in streaming over window. Can be "full", "recent", "recent\_first\_n" or "recent\_last\_n". |
| background\_ddl | true/false | Run DDL statements in background. |
| server\_encoding | UTF8 | Show the server-side character set encoding. At present, this parameter can be shown but not set, because the encoding is determined at database creation time. |
Expand Down
9 changes: 6 additions & 3 deletions sql/commands/sql-alter-materialized-view.mdx
View file
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,6 @@ ALTER MATERIALIZED VIEW materialized_view_name
```
-- Change the owner of the materialized view named "materialized_view1" to user "user1"
ALTER MATERIALIZED VIEW materialized_view1 OWNER TO user1;
```

### `SET SCHEMA`
Expand Down Expand Up @@ -96,7 +95,8 @@ ALTER MATERIALIZED VIEW mv_name
SET BACKFILL_RATE_LIMIT { TO | = } { default | rate_limit_number };
```

Use this statement to modify the backfill rate limit of a materialized view being created. For the specific value of `BACKFILL_RATE_LIMIT`, refer to [How to view runtime parameters](/docs/current/view-configure-runtime-parameters/#how-to-view-runtime-parameters).
This statement controls the rate limit of a newly created materialized view's backfilling process from upstream materialized views and sources.
For the specific value of `BACKFILL_RATE_LIMIT`, refer to [How to view runtime parameters](/operate/view-configure-runtime-parameters/#how-to-view-runtime-parameters).

```sql Examples
-- Pause the backfill
Expand All @@ -107,5 +107,8 @@ ALTER MATERIALIZED VIEW mv1 SET BACKFILL_RATE_LIMIT=1;

-- Disable the backfill
ALTER MATERIALIZED VIEW mv1 SET BACKFILL_RATE_LIMIT=DEFAULT;

```

<Note>
To modify the rate limit of the sources used in the materialized view, please refer to [SET SOURCE_RATE_LIMIT](/sql/commands/sql-alter-source#set-source-rate-limit).
</Note>
10 changes: 9 additions & 1 deletion sql/commands/sql-alter-source.mdx
View file
Original file line number Diff line number Diff line change
Expand Up @@ -173,7 +173,15 @@ ALTER SOURCE source_name
SET SOURCE_RATE_LIMIT { TO | = } { default | rate_limit_number };
```

Use this statement to modify the rate limit of a source. For the specific value of `SOURCE_RATE_LIMIT`, refer to [How to view runtime parameters](/docs/current/view-configure-runtime-parameters/#how-to-view-runtime-parameters).
Use this statement to modify the rate limit of a source. For the specific value of `SOURCE_RATE_LIMIT`, refer to [How to view runtime parameters](/operate/view-configure-runtime-parameters/#how-to-view-runtime-parameters).

<Note>
For a newly created materialized view on a shared source, it will backfill from
the source. The backfilling process will not be affected by the `SOURCE_RATE_LIMIT`
of the source.

To modify the rate limit of the backfilling process, please refer to [SET BACKFILL_RATE_LIMIT](/sql/commands/sql-alter-materialized-view#set-backfill-rate-limit).
</Note>

```sql Example
-- Alter the rate limit of a source to default
Expand Down
10 changes: 5 additions & 5 deletions sql/commands/sql-alter-table.mdx
View file
Original file line number Diff line number Diff line change
Expand Up @@ -203,8 +203,6 @@ ALTER TABLE t_user REFRESH SCHEMA;
```

<Info>
**NOTE**

If a downstream fragment references a column that is either missing or has undergone a type change in the updated schema, the command will be declined.
</Info>

Expand All @@ -215,7 +213,8 @@ ALTER TABLE table_name
SET SOURCE_RATE_LIMIT { TO | = } { default | rate_limit_number };
```

Use this statement to modify the rate limit of tables that have a source. For the specific value of `SOURCE_RATE_LIMIT`, refer to [How to view runtime parameters](/docs/current/view-configure-runtime-parameters/#how-to-view-runtime-parameters).
For tables with connector, this statement controls the rate limit of the associated source.
For the specific value of `SOURCE_RATE_LIMIT`, refer to [How to view runtime parameters](/docs/current/view-configure-runtime-parameters/#how-to-view-runtime-parameters).

```sql Example
-- Create a table with source
Expand All @@ -238,13 +237,14 @@ ALTER TABLE kafka_source SET source_rate_limit TO default;
ALTER TABLE kafka_source SET source_rate_limit TO 1000;
```

### `SET BACKFILL_RATE_LIMIT`[]
### `SET BACKFILL_RATE_LIMIT`
```sql
ALTER TABLE table_name
SET BACKFILL_RATE_LIMIT { TO | = } { default | rate_limit_number };
```

Use this statement to modify the backfill rate limit of a CDC table being created from a CDC source. For the specific value of `BACKFILL_RATE_LIMIT`, refer to [How to view runtime parameters](/docs/current/view-configure-runtime-parameters/#how-to-view-runtime-parameters).
For CDC table created from a CDC source, this statement controls the rate limit of backfilling from the CDC database.
For the specific value of `BACKFILL_RATE_LIMIT`, refer to [How to view runtime parameters](/operate/view-configure-runtime-parameters/#how-to-view-runtime-parameters).

```sql Examples
-- Pause the backfill
Expand Down

0 comments on commit d29346a

Please sign in to comment.