Docs fix about EAGLE and streaming output #3166
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
zhaochenyang20
requested changes
Jan 27, 2025
Comment on lines
88
to
121
| "def remove_overlap(existing_text, new_chunk):\n", | ||
| " \"\"\"\n", | ||
| " Finds the largest suffix of 'existing_text' that is a prefix of 'new_chunk'\n", | ||
| " and removes that overlap from the start of 'new_chunk'.\n", | ||
| " \"\"\"\n", | ||
| " max_overlap = 0\n", | ||
| " max_possible = min(len(existing_text), len(new_chunk))\n", | ||
| "\n", | ||
| " for i in range(max_possible, 0, -1):\n", | ||
| " if existing_text.endswith(new_chunk[:i]):\n", | ||
| " max_overlap = i\n", | ||
| " break\n", | ||
| "\n", | ||
| " return new_chunk[max_overlap:]\n", | ||
| "\n", | ||
| "\n", | ||
| "def generate_text_no_repeats(llm, prompt, sampling_params):\n", | ||
| " \"\"\"\n", | ||
| " Example function that:\n", | ||
| " 1) Streams the text,\n", | ||
| " 2) Removes chunk overlaps,\n", | ||
| " 3) Returns the merged text.\n", | ||
| " \"\"\"\n", | ||
| " final_text = \"\"\n", | ||
| " for chunk in llm.generate(prompt, sampling_params, stream=True):\n", | ||
| " chunk_text = chunk[\"text\"]\n", | ||
| "\n", | ||
| " cleaned_chunk = remove_overlap(final_text, chunk_text)\n", | ||
| "\n", | ||
| " final_text += cleaned_chunk\n", | ||
| "\n", | ||
| " return final_text\n", | ||
| "\n", | ||
| "\n", |
There was a problem hiding this comment.
Move this to python/sglang/utils.py and import it at the beginning of docs. Remember also to change other streaming part.
There was a problem hiding this comment.
Imported and solve for streaming part also in the Streaming Asynchronous Generation.
Will check cahnge other streaming part tmrw
Comment on lines
11
to
17
| "> **Note**: To run the following tests or benchmarks, you also need to install the [**cutex**](https://pypi.org/project/cutex/) module. \n", | ||
| "> **Requirement**: Python 3.6+ on a Unix-like OS with **fcntl** support. \n", | ||
| "> **Installation**: \n", | ||
| "> ```bash\n", | ||
| "> pip install cutex\n", | ||
| "> ```\n", | ||
| "\n", |
There was a problem hiding this comment.
To run the following tests or benchmarks, you also need to install cutex:
pip install cutex
zhaochenyang20
requested changes
Jan 27, 2025
…into docs/streaming-fix
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
To improve docs rendering for speculative_decoding and to solve issue #3164(#3164)
Modifications
Solve necessary rendering issues in speculative_decoding.ipynb and also help solve streaming output issue mentioned in issue 3164
Checklist