Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add --match flag to work queues documentation #7768

Merged
merged 7 commits into from
Jan 6, 2023
Merged

Add --match flag to work queues documentation #7768

Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
e781d0c
added match documentation
Dec 5, 2022
0485363
Merge branch 'PrefectHQ:main' into work_queues_docs
riquelmev Dec 5, 2022
2e71277
added back divs
Dec 5, 2022
88170a2
added punctuation
Dec 5, 2022
47bf9eb
addressed comments
Dec 10, 2022
c183387
fixed spacing
Dec 10, 2022
59327e6
Apply suggestions from code review
zanieb Jan 6, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 24 additions & 8 deletions docs/concepts/work-queues.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,13 +42,14 @@ Agents are configured to pull work from one or more work queues. If the agent re

Configuration parameters you can specify when starting an agent include:

| Option | Description |
| --- | --- |
| -q, --work-queue | One or more work queues that the agent will poll for work. |
| --api TEXT | The API URL for the Prefect Orion server. Default is the value of `PREFECT_API_URL`. |
| --run-once | Only run agent polling once. By default, the agent runs forever. |
| --prefetch-seconds | The amount of time before a flow run's scheduled start time to begin submission. Default is the value of `PREFECT_AGENT_PREFETCH_SECONDS`. |
| --hide-welcome | Do not display the startup ASCII art for the agent process. |
| Option | Description |
|---|---|
| -q, --work-queue | One or more work queues that the agent will poll for work. |
| --api TEXT | The API URL for the Prefect Orion server. Default is the value of `PREFECT_API_URL`. |
| --run-once | Only run agent polling once. By default, the agent runs forever. |
| --prefetch-seconds | The amount of time before a flow run's scheduled start time to begin submission. Default is the value of `PREFECT_AGENT_PREFETCH_SECONDS`. |
| --hide-welcome | Do not display the startup ASCII art for the agent process. |
| --match QUEUE_PREFIX | Polls for queues matching the given QUEUE_PREFIX. Note: this option overrides anything passed with `-q` or `--work-queue`. |

You must start an agent within an environment that can access or create the infrastructure needed to execute flow runs. Your agent will deploy flow runs to the infrastructure specified by the deployment.

Expand All @@ -62,12 +63,14 @@ You must start an agent within an environment that can access or create the infr

### Starting an agent

Use the `prefect agent start` CLI command to start an agent. You must pass at least one work queue name that the agent will poll for work. If the work queue does not exist, it will be created.
Use the `prefect agent start` CLI command to start an agent. You must pass at least one work queue name or match string that the agent will poll for work. If the work queue does not exist, it will be created.

<div class="terminal">

```bash
$ prefect agent start -q [work queue name]
```

</div>

For example:
Expand All @@ -94,6 +97,19 @@ In this case, Prefect automatically created a new `my-queue` work queue.

By default, the agent polls the API specified by the `PREFECT_API_URL` environment variable. To configure the agent to poll from a different server location, use the `--api` flag, specifying the URL of the server.

In addition, agents can match multiple work queues by providing a `--match` string instead of specifying all of the work queues. The agent will poll every work queue with a name that starts with the given string. New queues matching this prefix will be found by the agent without needing to restart it.

For example:
<div class="terminal">

```bash
$ prefect agent start --match "foo-"
```

</div>

This example will poll every work queue that starts with "foo-".

### Configuring prefetch

By default, the agent begins submission of flow runs a short time (10 seconds) before they are scheduled to run. This allows time for the infrastructure to be created, so the flow run can start on time. In some cases, infrastructure will take longer than this to actually start the flow run. In these cases, the prefetch can be increased using the `--prefetch-seconds` option or the `PREFECT_AGENT_PREFETCH_SECONDS` setting. Submission can begin an arbitrary amount of time before the flow run is scheduled to start. If this value is _larger_ than the amount of time it takes for the infrastructure to start, the flow run will _wait_ until its scheduled start time. This allows flow runs to start exactly on time.
Expand Down