Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Reference Only] Shsanyal devpa 308 opt #1

Draft
wants to merge 10 commits into
base: llama_fp8_12062024
Choose a base branch
from
Draft

[Reference Only] Shsanyal devpa 308 opt #1

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
a126452
opts v1
sanyalington Nov 10, 2024
98270da
opt logits and expsum calc
sanyalington Nov 18, 2024
73a1edb
mfma16x16 load patterns only
sanyalington Nov 20, 2024
50208aa
checkpoint
sanyalington Nov 20, 2024
0977a1c
checkpoint working mfma16 kernel
sanyalington Nov 29, 2024
eb63b1f
functional 16 bit dtype kernel with ctx len handling
sanyalington Nov 29, 2024
ae66314
mfma16 PA kernel clean up; add tests for supported configs only
sanyalington Nov 29, 2024
cf863a7
optimize output writes
sanyalington Nov 29, 2024
402fdde
fetch q in shared mem for better address patterns
sanyalington Dec 1, 2024
8f7cbba
fp8 conversion optimizations; fast f32 -> bf16 rne conversion without…
sanyalington Dec 9, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 6 additions & 5 deletions benchmarks/kernels/benchmark_paged_attention.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,9 +9,8 @@
from vllm.utils import (STR_DTYPE_TO_TORCH_DTYPE, FlexibleArgumentParser,
create_kv_caches_with_random)

NUM_BLOCKS = 1024 * 1024
PARTITION_SIZE = 512

NUM_BLOCKS = 256 * 1024
PARTITION_SIZE = 256

@torch.inference_mode()
def main(
Expand Down Expand Up @@ -101,7 +100,7 @@ def run_cuda_benchmark(num_iters: int, profile: bool = False) -> float:
start_time = time.perf_counter()

# Using default kv_scale
k_scale = v_scale = 1.0
k_scale = v_scale = 0.1

for _ in range(num_iters):
if version == "v1":
Expand Down Expand Up @@ -161,6 +160,8 @@ def run_cuda_benchmark(num_iters: int, profile: bool = False) -> float:
kv_cache_dtype,
k_scale,
v_scale,
None,
PARTITION_SIZE
)
else:
raise ValueError(f"Invalid version: {version}")
Expand All @@ -180,7 +181,7 @@ def run_cuda_benchmark(num_iters: int, profile: bool = False) -> float:
if do_profile:
latency = run_benchmark(num_iters=1, profile=True)
else:
latency = run_benchmark(num_iters=1000, profile=False)
latency = run_benchmark(num_iters=10000, profile=False)
print(f"Kernel running time: {latency * 1000000:.3f} us")


Expand Down
Loading
Loading