Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update schedules.md #5826

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
+0 −66
Open

Update schedules.md #5826

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
66 changes: 0 additions & 66 deletions docs/user_guide/productionizing/schedules.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -85,70 +85,4 @@ You can [archive/deactivate the launch plan](https://docs.flyte.org/projects/fly
flytectl update launchplan -p flyteexamples -d development {{ name_of_lp }} --version <foo> --archive
```

## Platform configuration changes for AWS scheduler

The Scheduling feature can be run using the Flyte native scheduler which comes with Flyte. If you intend to use the AWS scheduler then it requires additional infrastructure to run, so these will have to be created and configured. The following sections are only required if you use the AWS scheme for the scheduler. You can still run the Flyte native scheduler on AWS.

### Setting up scheduled workflows

To run workflow executions based on user-specified schedules, you'll need to fill out the top-level `scheduler` portion of the flyteadmin application configuration.

In particular, you'll need to configure the two components responsible for scheduling workflows and processing schedule event triggers.

:::{note}
This functionality is currently only supported for AWS installs.
:::

#### Event scheduler

To schedule workflow executions, you'll need to set up an [AWS SQS](https://aws.amazon.com/sqs/) queue. A standard-type queue should suffice. The flyteadmin event scheduler creates [AWS CloudWatch](https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/Create-CloudWatch-Events-Scheduled-Rule.html) event rules that invoke your SQS queue as a target.

With that in mind, let's take a look at an example `eventScheduler` config section and dive into what each value represents:

```bash
scheduler:
eventScheduler:
scheme: "aws"
region: "us-east-1"
scheduleRole: "arn:aws:iam::{{ YOUR ACCOUNT ID }}:role/{{ ROLE }}"
targetName: "arn:aws:sqs:us-east-1:{{ YOUR ACCOUNT ID }}:{{ YOUR QUEUE NAME }}"
scheduleNamePrefix: "flyte"
```

- **scheme**: in this case because AWS is the only cloud back-end supported for scheduling workflows, only `"aws"` is a valid value. By default, the no-op scheduler is used.
- **region**: this specifies which region initialized AWS clients should use when creating CloudWatch rules.
- **scheduleRole** This is the IAM role ARN with permissions set to `Allow`
: - `events:PutRule`
- `events:PutTargets`
- `events:DeleteRule`
- `events:RemoveTargets`
- **targetName** this is the ARN for the SQS Queue you've allocated to scheduling workflows.
- **scheduleNamePrefix** this is an entirely optional prefix used when creating schedule rules. Because of AWS naming length restrictions, scheduled rules are a random hash and having a shared prefix makes these names more readable and indicates who generated the rules.

#### Workflow executor

Scheduled events which trigger need to be handled by the workflow executor, which subscribes to triggered events from the SQS queue configured above.

:::{NOTE}
Failure to configure a workflow executor will result in all your scheduled events piling up silently without ever kicking off workflow executions.
:::

Again, let's break down a sample config:

```bash
scheduler:
eventScheduler:
...
workflowExecutor:
scheme: "aws"
region: "us-east-1"
scheduleQueueName: "{{ YOUR QUEUE NAME }}"
accountId: "{{ YOUR ACCOUNT ID }}"
```

- **scheme**: in this case because AWS is the only cloud back-end supported for executing scheduled workflows, only `"aws"` is a valid value. By default, the no-op executor is used and in case of sandbox we use `"local"` scheme which uses the Flyte native scheduler.
- **region**: this specifies which region AWS clients should use when creating an SQS subscriber client.
- **scheduleQueueName**: this is the name of the SQS Queue you've allocated to scheduling workflows.
- **accountId**: Your AWS [account id](https://docs.aws.amazon.com/IAM/latest/UserGuide/console_account-alias.html#FindingYourAWSId).

[flytesnacks]: https://github.com/flyteorg/flytesnacks/tree/master/examples/productionizing/
Loading