Fix KV chunking for POD. #1054
Fix KV chunking for POD. #1054
Conversation
…PerMultiprocessor returns 0, so manually calculate the value instead.
|
Confirmed that this combined with setting prefill bs to 1 does make the kernel faster. (H100) |
Which is the GPU architecture you are testing on? |
| cudaDeviceGetAttribute(&num_sm, cudaDevAttrMultiProcessorCount, dev_id)); | ||
| FLASHINFER_CUDA_CALL(cudaOccupancyMaxActiveBlocksPerMultiprocessor( | ||
| &num_blocks_per_sm, kernel, num_threads_p, smem_size_p)); | ||
| // FLASHINFER_CUDA_CALL(cudaOccupancyMaxActiveBlocksPerMultiprocessor( |
There was a problem hiding this comment.
It's interesting to me, and likely a bug of cudaOccupancyMaxActiveBlocksPerMultiprocessor.
Let's merge this first, thanks for the contribution!
There was a problem hiding this comment.
cudaOccupancyMaxActiveBlocksPerMultiprocessor is buggy on both A100 40G and H100 for me
The one above was H100, this one is A100 40G |
|
@Edenzzzz there might be some problem with bandwidth measure because they exceed hardware limit (for H100, the maximum bandwidth is 3352 GB/s). |
|
Yeah, it's not obvious to me why. Will try NCU |
|
@yzh119 The kernel timing is correct according to nsys |
|
Huh, this is leading me to believe there's still some bug in POD. Looking into it. |
|
There was a bug: #1059 |
3 participants
For some reason cudaOccupancyMaxActiveBlocksPerMultiprocessor returns 0, so manually calculate the value instead.