[V1] Remove pre-allocation for KV cache

[WoosukKwon]

This PR removes the pre-allocation logic in the KV cache manager, which seems to be unnecessary in current vLLM.

The pre-allocation logic was introduced among the first few PRs for V1 re-architecture.
Originally, it had two purposes:

1. As written in the comment, we wanted to reduce the overheads of memory allocation (more exactly, allocating new blocks). With pre-allocation, we were able to amortize the overheads by allocating new blocks for a request every 64 steps instead of every single step.
2. While not written as a comment, its another purpose was to avoid frequent preemptions, where new requests are scheduled but preempted in the next few steps. We thought pre-allocation could reduce the chance of this happening.

However, it seems these two ideas are no longer valid:

1. Since @comaniac introduced doubly linked list for free blocks, the overhead of block allocation has become marginal. Prior to the PR, we used inefficient list operations to manage the free blocks, leading to high overheads whenever block allocation happened. With DLL, the overhead is small, so is the saving from pre-allocation. *Still, we have substantial overheads from allocate_slots as a whole. However, pre-allocation does not save much of them.
2. It turns out that the pre-allocation does not meaningfully mitigate the chance of preemption. We need to 1) investigate how much preemption affects the overall performance, and 2) find a better way to mitigate it.

Moreover, pre-allocation does not go very well with the new hybrid memory allocator and KV cache connector. As it only complicates the logic without performance advantages, I delete it in this PR.

Performance

• python benchmarks/benchmark_throughput.py --model meta-llama/Llama-3.1-8B --dataset <sharegpt> --num-prompts 5000
  main: 53.42 reqs/s (0 preemption), this PR: 53.41 reqs/s (0 preemption)
• python benchmarks/benchmark_throughput.py --model meta-llama/Llama-3.1-8B --input-len 1000 --output-len 10000 --num-prompts 100
  main: 0.22 reqs/s (96 preemptions), this PR: 0.22 reqs/s (95 preemptions)

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

[comaniac]

What's the reason of this change? I thought preallocation was a feature that you heavily promoted when introducing v1?

[yinghai]

what is num_preallocate_tokens ?

[WoosukKwon]

@yinghai I've updated the PR description.

The idea is, when allocating KV cache blocks for a request, rather than allocating just the right amount of blocks, we allocated a few more extra blocks ahead. Originally, this amortized the overheads of block allocation.
However, I recently found that this is no longer the case (for the reason I described in the PR), and pre-allocation only makes the logic complicated without performance advantages. So I'm deleting it in this PR.

[WoosukKwon]

What's the reason of this change? I thought preallocation was a feature that you heavily promoted when introducing v1?

@comaniac Good question. I was writing the PR description actually 😂 PTAL.
I will also add benchmark results.

[comaniac]

LGTM