Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

enhance rate limit doc #62

Merged
merged 2 commits into from
Nov 20, 2024
Merged

enhance rate limit doc #62

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
d29346a
enhance rate limit doc
xxchan Nov 19, 2024
7529f9a
Update sql/commands/sql-alter-source.mdx
xxchan Nov 20, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions operate/view-configure-runtime-parameters.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -67,8 +67,8 @@ Below is the detailed information about the parameters you may see after using t
| cdc\_source\_wait\_streaming\_start\_timeout | 30 | For limiting the startup time of a shareable CDC streaming source when the source is being created. Unit: seconds. |
| row\_security | true/false | See [here](https://www.postgresql.org/docs/current/runtime-config-client.html#GUC-ROW-SECURITY) for details. Unused in RisingWave, support for compatibility. |
| standard\_conforming\_strings | on | See [here](https://www.postgresql.org/docs/current/runtime-config-client.html#GUC-STANDARD-CONFORMING-STRINGS) for details. |
| source\_rate\_limit | default/ A positive integer / 0 | Set the maximum number of records per second per source, for each parallelism. The source here refers to an upstream source. This parameter is applied to tables and tables with sources. The value can be default, 0, or a positive integer. SET SOURCE\_RATE\_LIMIT TO 0 will pause the source read for sources. SET SOURCE\_RATE\_LIMIT TO DEFAULT will disable the rate limit within the session, but it will not change the rate limits of existing DDLs.Note that the total throughput of a streaming job is determined by multiplying the parallelism with the throttle rate. To obtain the parallelism value for a streaming job, you can refer to the streaming\_parallelism runtime parameter in this table. Additionally, we support altering rate limits in [sources](/docs/current/sql-alter-source/#set-source-rate-limit) and [tables that have source](/docs/current/sql-alter-table/#set-source-rate-limit). |
| backfill\_rate\_limit | default/ A positive integer / 0 | Set the maximum number of records per second per parallelism for the backfill process of materialized views, sinks, and indexes. This parameter throttles the snapshot read stream for backfill. The value can be default, 0, or a positive integer. SET BACKFILL\_RATE\_LIMIT TO 0 will pause the snapshot read stream for backfill. SET BACKFILL\_RATE\_LIMIT TO default will disable the backfill rate limit within the session, but it will not change the backfill rate limit of existing DDLs. To obtain the parallelism value for a streaming job, you can refer to the streaming\_parallelism runtime parameter in this table. Additionally, we support altering backfill rate limits in [materialized views](/docs/current/sql-alter-materialized-view/#set-backfill%5Frate%5Flimit) and [CDC tables](/docs/current/sql-alter-table/#set-backfill%5Frate%5Flimit). |
| source\_rate\_limit | default/positive integer/0 | Set the maximum number of records per second per source, for each parallelism. This parameter is applied when creating new sources and tables with sources. <br/><br/> The value can be default, 0, or a positive integer. <br/> SET SOURCE\_RATE\_LIMIT TO 0 will pause the source read for sources. <br/> SET SOURCE\_RATE\_LIMIT TO DEFAULT will remove the rate limit. <br/><br/> Setting this variable will only affect new DDLs within the session, but not change the rate limits of existing jobs. Use `ALTER` to change the rate limits in existing [sources](/sql/commands/sql-alter-source/#set-source-rate-limit) and [tables that have source](/sql/commands/sql-alter-table/#set-source-rate-limit). <br/><br/> Note that the total throughput of a streaming job is determined by multiplying the parallelism with the throttle rate. To obtain the parallelism value for a streaming job, you can refer to the streaming\_parallelism runtime parameter in this table. |
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

More of a formatting suggestion. We can treat parameter names and values as code (code_snippet).

The terms used here (I know they were carried over from previous versions) are confusing: "tables with sources" and "tables that have sources." I think it's worth having a discussion whether we want to have a dedicated name for it, such as "streaming tables."

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can treat parameter names and values as code (code_snippet).

Didn't change this to keep consistency with others in the table. (Maybe we should change them all?)

| backfill\_rate\_limit | default/positive integer/0 | Set the maximum number of records per second per parallelism for the backfill process of materialized views, sinks, and indexes. This parameter is applied when creating new jobs, and throttles the backfill from upstream materialized views and sources. <br/><br/> The value can be default, 0, or a positive integer. <br/> SET BACKFILL\_RATE\_LIMIT TO 0 will pause the backfill. <br/> SET BACKFILL\_RATE\_LIMIT TO default will remove the backfill rate limit. <br/><br/> Setting this variable will only affect new DDLs within the session, but not change the rate limits of existing jobs. Use `ALTER` to change the backfill rate limits in existing [materialized views](//sql-alter-materialized-view/#set-backfill%5Frate%5Flimit) and [CDC tables](/sql/commands/sql-alter-table/#set-backfill%5Frate%5Flimit). <br/><br/> Note that the total throughput of a streaming job is determined by multiplying the parallelism with the throttle rate. To obtain the parallelism value for a streaming job, you can refer to the streaming\_parallelism runtime parameter in this table. |
| rw\_streaming\_over\_window\_cache\_policy | full | Cache policy for partition cache in streaming over window. Can be "full", "recent", "recent\_first\_n" or "recent\_last\_n". |
| background\_ddl | true/false | Run DDL statements in background. |
| server\_encoding | UTF8 | Show the server-side character set encoding. At present, this parameter can be shown but not set, because the encoding is determined at database creation time. |
Expand Down
9 changes: 6 additions & 3 deletions sql/commands/sql-alter-materialized-view.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,6 @@ ALTER MATERIALIZED VIEW materialized_view_name
```
-- Change the owner of the materialized view named "materialized_view1" to user "user1"
ALTER MATERIALIZED VIEW materialized_view1 OWNER TO user1;
```

### `SET SCHEMA`
Expand Down Expand Up @@ -96,7 +95,8 @@ ALTER MATERIALIZED VIEW mv_name
SET BACKFILL_RATE_LIMIT { TO | = } { default | rate_limit_number };
```

Use this statement to modify the backfill rate limit of a materialized view being created. For the specific value of `BACKFILL_RATE_LIMIT`, refer to [How to view runtime parameters](/docs/current/view-configure-runtime-parameters/#how-to-view-runtime-parameters).
This statement controls the rate limit of a newly created materialized view's backfilling process from upstream materialized views and sources.
For the specific value of `BACKFILL_RATE_LIMIT`, refer to [How to view runtime parameters](/operate/view-configure-runtime-parameters/#how-to-view-runtime-parameters).

```sql Examples
-- Pause the backfill
Expand All @@ -107,5 +107,8 @@ ALTER MATERIALIZED VIEW mv1 SET BACKFILL_RATE_LIMIT=1;

-- Disable the backfill
ALTER MATERIALIZED VIEW mv1 SET BACKFILL_RATE_LIMIT=DEFAULT;

```

<Note>
To modify the rate limit of the sources used in the materialized view, please refer to [SET SOURCE_RATE_LIMIT](/sql/commands/sql-alter-source#set-source-rate-limit).
</Note>
10 changes: 9 additions & 1 deletion sql/commands/sql-alter-source.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -173,7 +173,15 @@ ALTER SOURCE source_name
SET SOURCE_RATE_LIMIT { TO | = } { default | rate_limit_number };
```

Use this statement to modify the rate limit of a source. For the specific value of `SOURCE_RATE_LIMIT`, refer to [How to view runtime parameters](/docs/current/view-configure-runtime-parameters/#how-to-view-runtime-parameters).
Use this statement to modify the rate limit of a source. For the specific value of `SOURCE_RATE_LIMIT`, refer to [How to view runtime parameters](/operate/view-configure-runtime-parameters/#how-to-view-runtime-parameters).

<Note>
For a newly created materialized view on a source with historical data e.g. Kafka source, it will backfill from
the source. The backfilling process will not be affected by the `SOURCE_RATE_LIMIT`
of the source.

To modify the rate limit of the backfilling process, please refer to [SET BACKFILL_RATE_LIMIT](/sql/commands/sql-alter-materialized-view#set-backfill-rate-limit).
</Note>

```sql Example
-- Alter the rate limit of a source to default
Expand Down
10 changes: 5 additions & 5 deletions sql/commands/sql-alter-table.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -203,8 +203,6 @@ ALTER TABLE t_user REFRESH SCHEMA;
```

<Info>
**NOTE**

If a downstream fragment references a column that is either missing or has undergone a type change in the updated schema, the command will be declined.
</Info>

Expand All @@ -215,7 +213,8 @@ ALTER TABLE table_name
SET SOURCE_RATE_LIMIT { TO | = } { default | rate_limit_number };
```

Use this statement to modify the rate limit of tables that have a source. For the specific value of `SOURCE_RATE_LIMIT`, refer to [How to view runtime parameters](/docs/current/view-configure-runtime-parameters/#how-to-view-runtime-parameters).
For tables with connector, this statement controls the rate limit of the associated source.
For the specific value of `SOURCE_RATE_LIMIT`, refer to [How to view runtime parameters](/docs/current/view-configure-runtime-parameters/#how-to-view-runtime-parameters).
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how about giving an example to demostrate the two rate limit here?


```sql Example
-- Create a table with source
Expand All @@ -238,13 +237,14 @@ ALTER TABLE kafka_source SET source_rate_limit TO default;
ALTER TABLE kafka_source SET source_rate_limit TO 1000;
```

### `SET BACKFILL_RATE_LIMIT`[]
### `SET BACKFILL_RATE_LIMIT`
```sql
ALTER TABLE table_name
SET BACKFILL_RATE_LIMIT { TO | = } { default | rate_limit_number };
```

Use this statement to modify the backfill rate limit of a CDC table being created from a CDC source. For the specific value of `BACKFILL_RATE_LIMIT`, refer to [How to view runtime parameters](/docs/current/view-configure-runtime-parameters/#how-to-view-runtime-parameters).
For CDC table created from a CDC source, this statement controls the rate limit of backfilling from the CDC database.
For the specific value of `BACKFILL_RATE_LIMIT`, refer to [How to view runtime parameters](/operate/view-configure-runtime-parameters/#how-to-view-runtime-parameters).

```sql Examples
-- Pause the backfill
Expand Down