Enable SDG batching with vLLM #1797
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
markmc
requested changes
Jul 19, 2024
gabe-l-hart
force-pushed
the
SDGBatchArguments
branch
from
e9d0d02 to
bda25e9
Compare
mergify
bot
added
ci-failure
gabe-l-hart
force-pushed
the
SDGBatchArguments
branch
from
bda25e9 to
f052f4b
Compare
mergify
bot
added
ci-failure
4 tasks
|
This pull request has merge conflicts that must be resolved before it can be |
This was referenced Jul 27, 2024
Signed-off-by: Mark McLoughlin <markmc@redhat.com>
markmc
force-pushed
the
SDGBatchArguments
branch
from
f052f4b to
973a178
Compare
mergify
bot
added
dependencies
markmc
force-pushed
the
SDGBatchArguments
branch
from
973a178 to
8929260
Compare
mergify
bot
added
ci-failure
markmc
force-pushed
the
SDGBatchArguments
branch
from
8929260 to
7b620b4
Compare
mergify
bot
added
CI/CD
mergify
bot
added
documentation
danmcp
reviewed
Jul 27, 2024
src/instructlab/data/generate.py
Outdated
| if isinstance(backend_instance, llama_cpp_server): | ||
| batch_size = 0 | ||
| logger.warning( | ||
| "Disabling SDG batching - unsupported with llama.cpp serving" |
There was a problem hiding this comment.
Should we only log this message when the user requested a particular batch-size?
gabe-l-hart
force-pushed
the
SDGBatchArguments
branch
from
7954527 to
fcb84b3
Compare
danmcp
reviewed
Jul 27, 2024
Relates to instructlab/sdg#135 Since instructlab-sdg-0.1.3, data generation in batches is supported and controlled by an parameter to the `generate_data()` function. This is not supported with llama-cpp, and so we disable it in that case. Co-authored-by: Mark McLoughlin <markmc@redhat.com> Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> Signed-off-by: Mark McLoughlin <markmc@redhat.com>
Now that we require a new enough version of instructlab-sdg, we can expose this. Signed-off-by: Mark McLoughlin <markmc@redhat.com>
markmc
force-pushed
the
SDGBatchArguments
branch
from
fcb84b3 to
fafa8c0
Compare
@gabe-l-hart was that intentional? Sorry I didn't coordinate - just looking to get this merged quickly |
Also add a troubleshooting note to reference instructlab#1892 which tracks a todo item to add some way to automatically disable in this case. Signed-off-by: Mark McLoughlin <markmc@redhat.com>
markmc
force-pushed
the
SDGBatchArguments
branch
from
fafa8c0 to
3c03122
Compare
|
@markmc Sorry if that wasn't wanted! I got the mergebot conflict notification and wanted to clear the notification before diving into kid time and becoming unavailable for the rest of the weekend. Feel free to force push back if needed. (yikes, typing on a phone is hard!) |
n1hility
approved these changes
Jul 27, 2024
danmcp
approved these changes
Jul 27, 2024
markmc
approved these changes
Jul 28, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Epic: instructlab/sdg#135
Requires: instructlab/sdg#157
When using a serving backend that supports batches of requests (i.e. vLLM), the current SDG pipeline sends the entire dataset as a single large batch of requests to the OpenAI server. This may lead to some requests waiting too long for the response, resulting in timeout errors and potentially overloading the backend server with extremely large batches.
SDG now supports launching parallel tasks to send smaller batches in parallel to the OpenAPI server. This is controlled by the
batch_sizeargument togenerate_data():batch_size=None- use the library's built-in default batch size (currently 0, but will change to 8 in a future release)batch_size=0- disable batching, just use one batchbatch_size=<int>- use batches of the specified sizeIf the user sets the batch size to 8, the system will run a concurrent thread on each CPU on the system, each sending a batch of 8 prompts. So, on an 8 CPU system, this would result in a total of 64 requests processed simultaneously by the backend server.
This PR adds
--batch-sizeto ilab data generate to allow overriding the default behaviour of a batch size of 8 with vLLM and zero with llama-cpp.Supports instructlab/sdg#135