fix stop detections

[mingfang]

The current stop detection works but only after the partial stop sequence was already streamed.
This causes ReAct to break.

This change adds partial stop detection and avoids streaming it.

[suquark]

@merrymercy any suggestions of testing this

[mingfang]

One way to test is to use this curl that simulates the LangChain calculator agent example.
The example is to ask what is 1+1.
It should respond by asking the calculator tool for 1+1, with the fix.

curl -d '{"model":"vicuna-13b-v1.1","temperature":0,"max_tokens":256,"top_p":1,"frequency_penalty":0,"presence_penalty":0,"n":1,"best_of":1,"stop":["\nObservation: "],"stream":true,"prompt":["Answer the following questions as best you can. You have access to the following tools:\n\ncalculator: Useful for getting the result of a math expression. The input to this tool should be a valid mathematical expression that could be executed by a simple calculator.\n\nUse the following format in your response:\n\nQuestion: the input question you must answer\nThought: you should always think about what to do\nAction: the action to take, should be one of [calculator]\nAction Input: the input to the action\nObservation: the result of the action\n... (this Thought/Action/Action Input/Observation can repeat N times)\nThought: I now know the final answer\nFinal Answer: the final answer to the original input question\n\nBegin!\n\nQuestion: what is 1+1\nThought:"]}' localhost:8000/v1/completions -H 'content-type: application/json'

The output should end like this

...more before this...
data: {"id": "cmpl-3ADpLjweTitvHHpddeV55R", "object": "text_completion", "model": "vicuna-13b-v1.1", "choices": [{"index": 0, "text": "+1", "logprobs": null, "finish_reason": null}]}

data: {"id": "cmpl-3ADpLjweTitvHHpddeV55R", "object": "text_completion", "model": "vicuna-13b-v1.1", "choices": [{"index": 0, "text": "", "logprobs": null, "finish_reason": "stop"}]}

data: [DONE]

[merrymercy]

LGTM

[suquark]

LGTM! Thanks!

[plancktree]

but wen

One way to test is to use this curl that simulates the LangChain calculator agent example. The example is to ask what is 1+1. It should respond by asking the calculator tool for 1+1, with the fix.

curl -d '{"model":"vicuna-13b-v1.1","temperature":0,"max_tokens":256,"top_p":1,"frequency_penalty":0,"presence_penalty":0,"n":1,"best_of":1,"stop":["\nObservation: "],"stream":true,"prompt":["Answer the following questions as best you can. You have access to the following tools:\n\ncalculator: Useful for getting the result of a math expression. The input to this tool should be a valid mathematical expression that could be executed by a simple calculator.\n\nUse the following format in your response:\n\nQuestion: the input question you must answer\nThought: you should always think about what to do\nAction: the action to take, should be one of [calculator]\nAction Input: the input to the action\nObservation: the result of the action\n... (this Thought/Action/Action Input/Observation can repeat N times)\nThought: I now know the final answer\nFinal Answer: the final answer to the original input question\n\nBegin!\n\nQuestion: what is 1+1\nThought:"]}' localhost:8000/v1/completions -H 'content-type: application/json'

The output should end like this

...more before this...
data: {"id": "cmpl-3ADpLjweTitvHHpddeV55R", "object": "text_completion", "model": "vicuna-13b-v1.1", "choices": [{"index": 0, "text": "+1", "logprobs": null, "finish_reason": null}]}

data: {"id": "cmpl-3ADpLjweTitvHHpddeV55R", "object": "text_completion", "model": "vicuna-13b-v1.1", "choices": [{"index": 0, "text": "", "logprobs": null, "finish_reason": "stop"}]}

data: [DONE]

but when I use vllm in 4 v100 GPUS the problem occurs
there is not "stop"
when I use vllm in 4 v100 GPUS the problem occurs
there is "stop" reason.I wander why