Skip to content

Commit

Permalink
[backend-comparison] Refresh access token and display authenticated u…
Browse files Browse the repository at this point in the history
…ser name (#1483)

* [backend-comparison] Serialize both auth tokens to cache file

We need to refresh token to be able to renew an expired access token.

* [backend-comparison] Refresh access token

* [backend-comparison] Display user name with auth command

* [backend-comparison] Update README

* [backend-comparison] Fix PR comments

* [backend-comparison] Fix hyphen consistency in benchmark names

* [backend-comparison] Fix release build error when refreshing token

The reqwest must have an explicit empty body otherwise the release
build returns a 411 when refreshing the tokens without even calling
the benchmark server endpoint.
  • Loading branch information
syl20bnr authored Mar 20, 2024
1 parent 47a84cc commit e8863da
Show file tree
Hide file tree
Showing 7 changed files with 274 additions and 180 deletions.
1 change: 1 addition & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

7 changes: 5 additions & 2 deletions backend-comparison/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,7 @@ github-device-flow = { workspace = true }
rand = { workspace = true }
ratatui = { workspace = true, optional = true }
reqwest = {workspace = true, features = ["blocking", "json"]}
serde = { workspace = true }
serde_json = { workspace = true }
strum = { workspace = true }
strum_macros = { workspace = true }
Expand All @@ -54,7 +55,8 @@ name = "binary"
harness = false

[[bench]]
name = "max_pool2d"
name = "max-pool2d"
path = "benches/max_pool2d.rs"
harness = false

[[bench]]
Expand All @@ -66,7 +68,8 @@ name = "data"
harness = false

[[bench]]
name = "custom_gelu"
name = "custom-gelu"
path = "benches/custom_gelu.rs"
harness = false

[[bin]]
Expand Down
30 changes: 22 additions & 8 deletions backend-comparison/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,10 @@ within the corresponding backend crate.
## burnbench CLI

This crate comes with a CLI binary called `burnbench` which can be executed via
`cargo run --bin burnbench`.
`cargo run --release --bin burnbench`.

Note that you need to run the `release` target of `burnbench` otherwise you won't
be able to share your benchmark results.

The end of options argument `--` is used to pass arguments to the `burnbench`
application. For instance `cargo run --bin burnbench -- list` passes the `list`
Expand All @@ -23,7 +26,7 @@ argument to `burnbench` effectively calling `burnbench list`.
To list all the available benches and backends use the `list` command:

```sh
> cargo run --bin burnbench -- list
> cargo run --release --bin burnbench -- list
Finished dev [unoptimized] target(s) in 0.10s
Running `target/debug/burnbench list`
Available Backends:
Expand Down Expand Up @@ -54,22 +57,21 @@ with the arguments `--benches` and `--backends` respectively. In the following
example we execute the `unary` benchmark against the `wgpu-fusion` backend:

```sh
> cargo run --bin burnbench -- run --benches unary --backends wgpu-fusion
> cargo run --release --bin burnbench -- run --benches unary --backends wgpu-fusion
```

Shorthands can be used, the following command line is the same:

```sh
> cargo run --bin burnbench -- run -b unary -B wgpu-fusion
> cargo run --release --bin burnbench -- run -b unary -B wgpu-fusion
```

Multiple benchmarks and backends can be passed on the same command line. In this
case, all the combinations of benchmarks with backends will be executed.

```sh
> cargo run --bin burnbench -- run --benches unary binary --backends wgpu-fusion tch-gpu
Finished dev [unoptimized] target(s) in 0.09s
Running `target/debug/burnbench run --benches unary binary --backends wgpu-fusion wgpu`
Running `target/release/burnbench run --benches unary binary --backends wgpu-fusion wgpu`
Executing the following benchmark and backend combinations (Total: 4):
- Benchmark: unary, Backend: wgpu-fusion
- Benchmark: binary, Backend: wgpu-fusion
Expand All @@ -88,22 +90,34 @@ Sharing results is opt-in and it is enabled with the `--share` arguments passed
to the `run` command:

```sh
> cargo run --bin burnbench -- run --share --benches unary --backends wgpu-fusion
> cargo run --release --bin burnbench -- run --share --benches unary --backends wgpu-fusion
```

To be able to upload results you must be authenticated. We only support GitHub
authentication. To authenticate run the `auth` command, then follow the URL
to enter your device code and authorize the Burnbench application:

```sh
> cargo run --bin burnbench -- run auth
> cargo run --release --bin burnbench -- auth
```

If everything is fine you should get a confirmation in the terminal that your
token has been saved to the burn cache directory.

We don't store any of your personal information. An anonymized user name will
be attributed to you and displayed in the terminal once you are authenticated.
For instance:

```
🔑 Your username is: CuteFlame
```

You can now use the `--share` argument to upload and share your benchmarks!

Note that your access token will be refreshed automatically so you should not
need to reauthorize the application again except if your refresh token itself
becomes invalid.

### Terminal UI

This is a work in progress and is not usable for now.
Expand Down
Loading

0 comments on commit e8863da

Please sign in to comment.