Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update bandit docs #385

Merged
merged 2 commits into from
Nov 29, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 6 additions & 9 deletions bandit/README.md
Original file line number Diff line number Diff line change
@@ -1,23 +1,20 @@
## support-bandit

Defines three lambdas `get-bandit-tests`, `query-lambda` and `calculate-lambda` that form a state machine (using Step Functions) to record view and acquisition data for [multi-armed bandit](https://vwo.com/blog/multi-armed-bandit-algorithm/) experiments. RRCP channel tests (only Epics for now) that are marked as bandit experiments will be picked up and their performance data written to a DynamoDB table. The state machine is scheduled to run once an hour.
Defines two lambdas, `get-bandit-tests` and `query-lambda`, that form a state machine (using Step Functions) to record view and acquisition data for [multi-armed bandit](https://vwo.com/blog/multi-armed-bandit-algorithm/) experiments. RRCP channel tests (Epics + Banners) that are marked as bandit experiments will be picked up and their performance data written to a DynamoDB table. The state machine is scheduled to run once an hour.

![Overall architecture](docs/overall-architecture.png)
Diagram showing how this state machine fits into the existing architecture.
Diagram showing how this state machine (shown here as "bandit sampling") fits into the existing architecture.
The multi-armed bandit decision-making happens in SDC. (Original [here](https://docs.google.com/drawings/d/18FQDS9bT2ciTur-OHoaWfw6vMFxP3Zf-pz8_GCwM8FE))

### get-bandit-tests

The first lambda in the state machine is `get-bandit-tests`. It queries the DynamoDB table `support-admin-console-channel-tests-` for Epic tests with the property `isBanditTest: true`. It returns a list of these tests as its output.
The first lambda in the state machine is `get-bandit-tests`. It queries the DynamoDB table `support-admin-console-channel-tests-` for Epic and Banner tests that have a multi-armed bandit methodology in their `methodologies` field. It returns a list of these tests as its output.

### query-lambda

The next lambda, `query-lambda`, takes the tests from `get-bandit-tests` as its input. It queries the Athena tables `acquisition.epic_views_` and `acquisition.acquisition_events_` to get the views and acquisitions in GBP per variant per test. It uses the hour prior to the lambda execution as its date range. As its output it returns an array of query execution IDs.
The next lambda, `query-lambda`, takes the tests from `get-bandit-tests` as its input. It queries the BigQuery tables `fact_page_view_anonymised` and `fact_acquisition_event` to get the views and acquisitions in GBP per variant per test. It uses the hour prior to the lambda execution as its date range.

This lambda has a dependency on the `super-mode-calculator` to create the partitions necessary to query the Athena data. To avoid the partition and data not being ready, the hourly schedule is offset by 15 minutes past the hour.

### calculate-lambda

The final lambda in the state machine, `calculate-lambda`, takes the array of query execution IDs as its input. It polls Athena to check if the queries have finished executing, and the state machine retries this step if not. Once the data is returned it is written to a DynamoDB table `support-bandit-`. The data recorded consists of a test name, timestamp and an array of performance data per variant.
It writes this data to a DynamoDB table `support-bandit-`. The data recorded consists of a test name (prefixed with the channel), timestamp and an array of performance data per variant.

![Step functions architecture](docs/step-function-architecture.png)
Diagram showing the architecture of this state machine.
Binary file modified bandit/docs/overall-architecture.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified bandit/docs/step-function-architecture.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading