Skip to content

Conversation

@zeroorhero
Copy link
Owner

FILL IN THE PR DESCRIPTION HERE

FIX #xxxx (link existing issues this PR will resolve)

BEFORE SUBMITTING, PLEASE READ THE CHECKLIST BELOW AND FILL IN THE DESCRIPTION ABOVE


PR Checklist (Click to Expand)

Thank you for your contribution to vLLM! Before submitting the pull request, please ensure the PR meets the following criteria. This helps vLLM maintain the code quality and improve the efficiency of the review process.

PR Title and Classification

Only specific types of PRs will be reviewed. The PR title is prefixed appropriately to indicate the type of change. Please use one of the following:

  • [Bugfix] for bug fixes.
  • [CI/Build] for build or continuous integration improvements.
  • [Doc] for documentation fixes and improvements.
  • [Model] for adding a new model or improving an existing model. Model name should appear in the title.
  • [Frontend] For changes on the vLLM frontend (e.g., OpenAI API server, LLM class, etc.)
  • [Kernel] for changes affecting CUDA kernels or other compute kernels.
  • [Core] for changes in the core vLLM logic (e.g., LLMEngine, AsyncLLMEngine, Scheduler, etc.)
  • [Hardware][Vendor] for hardware-specific changes. Vendor name should appear in the prefix (e.g., [Hardware][AMD]).
  • [Misc] for PRs that do not fit the above categories. Please use this sparingly.

Note: If the PR spans more than one category, please include all relevant prefixes.

Code Quality

The PR need to meet the following code quality standards:

  • We adhere to Google Python style guide and Google C++ style guide.
  • Pass all linter checks. Please use format.sh to format your code.
  • The code need to be well-documented to ensure future contributors can easily understand the code.
  • Include sufficient tests to ensure the project to stay correct and robust. This includes both unit tests and integration tests.
  • Please add documentation to docs/source/ if the PR modifies the user-facing behaviors of vLLM. It helps vLLM user understand and utilize the new features or changes.

Notes for Large Changes

Please keep the changes as concise as possible. For major architectural changes (>500 LOC excluding kernel/data/config/test), we would expect a GitHub issue (RFC) discussing the technical design and justification. Otherwise, we will tag it with rfc-required and might not go through the PR.

What to Expect for the Reviews

The goal of the vLLM team is to be a transparent reviewing machine. We would like to make the review process transparent and efficient and make sure no contributor feel confused or frustrated. However, the vLLM team is small, so we need to prioritize some PRs over others. Here is what you can expect from the review process:

  • After the PR is submitted, the PR will be assigned to a reviewer. Every reviewer will pick up the PRs based on their expertise and availability.
  • After the PR is assigned, the reviewer will provide status update every 2-3 days. If the PR is not reviewed within 7 days, please feel free to ping the reviewer or the vLLM team.
  • After the review, the reviewer will put an action-required label on the PR if there are changes required. The contributor should address the comments and ping the reviewer to re-review the PR.
  • Please respond to all comments within a reasonable time frame. If a comment isn't clear or you disagree with a suggestion, feel free to ask for clarification or discuss the suggestion.

Thank You

Finally, thank you for taking the time to read these guidelines and for your interest in contributing to vLLM. Your contributions make vLLM a great tool for everyone!

@zeroorhero zeroorhero force-pushed the add-external-swapper branch 9 times, most recently from 28acb63 to 9009e50 Compare September 3, 2024 02:42
When we create the llm engine, we support
specifying the type of external swapper
and the size of the media type. Currently,
only local files are supported. Developers
can customize the format type of the external
swapper. For example, the format of the file
is "file://path/to/directory".
And supports users to add their own external swapper.

Change-Id: I7989e7aba32ad218629c067fc984c8744c25ab64
Signed-off-by: Changqi Lu <luchangqi.123@bytedance.com>
@zeroorhero zeroorhero force-pushed the add-external-swapper branch 2 times, most recently from ff06ee6 to 2d264a2 Compare September 3, 2024 07:44
Firstly, two methods determine_num_external_available_blocks and
initialize_external_cache are added to determine the number of
external blocks and perform initialization operations when
vllm-engine is started.

Then, according to the scheduling results of shceduler, the blocks
that need to be swapped are divided into two types: CPU and External,
and then the block swap operation is performed.

Change-Id: I155439e6a8af21ae2241c3eba892f82b7c03fcb2
Signed-off-by: Changqi Lu <luchangqi.123@bytedance.com>
Changqi Lu added 3 commits September 3, 2024 19:16
In the previous version of schedule, the blocks_to_swap_out field
in the return result only has the mapping of GPU to CPU block id.
However, after adding devices other than CPU, it is necessary to
distinguish different devices, so device related information is
added to this field.

At the same time, external swapper related methods are added to
scheduler and BlockManager, and corresponding implementations are made.

Change-Id: Iaec7fd7df17fa99c06bfc94d6cb314f44bc04522
Signed-off-by: Changqi Lu <luchangqi.123@bytedance.com>
The ExternalSwapperBase interface is extracted,
and users can implement their own ExternalSwapper.
ExternalSwapperBase corresponds one-to-one with cache_engine
and The interaction between cache_engine and external swapper
can be called into the specific implementation class of the
corresponding external swapper.

Change-Id: I585520bda1298bcb32e501d5f090299e4a21e1ad
Signed-off-by: Changqi Lu <luchangqi.123@bytedance.com>
Implementing external swapper for local file.

Change-Id: Ia57e8b62c68ea9f32cb03d29047131194fa245a8
Signed-off-by: Changqi Lu <luchangqi.123@bytedance.com>
luchangqi.123 added 3 commits September 3, 2024 21:19
Add swap_out_to_local_file and swap_in_from_local_file
kernels their and corresponding tests and benchmarks.

Change-Id: I222a819ee57a5604d5b1e9a61c5614d3208ef251
Signed-off-by: Changqi Lu <luchangqi.123@bytedance.com>
Added num_cumulative_preemption and external_cache_usage_sys
indicators to metrics

Change-Id: Id7f9c811cb3874e989b596ef9dbaa8b91442f875
Signed-off-by: Changqi Lu <luchangqi.123@bytedance.com>
benchmark_throughput support external swapper configuration.

Change-Id: Ib139ab8fb2c63aecf80a5251f855cdbb14b3da41
Signed-off-by: Changqi Lu <luchangqi.123@bytedance.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants