[Bugfix] Update Florence-2 tokenizer to make grounding tasks work

[Isotr0py]

FIX #16164 (comment)

• The Florence-2 hf_processor will add some special tokens to tokenizer for tasks like object detection and OCR with Region. (https://huggingface.co/microsoft/Florence-2-large/blob/main/processing_florence2.py#L87-L94)
• This PR updates the tokenizer in Florence-2 examples and tests to make these special tokens can be handled properly.

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

[DarkLight1337]

Thanks for fixing!

[PedroMiolaSilva]

@Isotr0py hey, its me again :).

We are trying to validate the tasks here, but the model doesnt seem to be ok in some tasks.

Tasks in which the response seems fine :CAPTION, DETAILED_CAPTION, MORE_DETAILED_CAPTION,
OCR,OCR_WITH_REGION ,DENSE_REGION_CAPTION ,PHRASE_GROUNDING ,OPEN_VOCABULARY_DETECTION,REFERRING_EXPRESSION_SEGMENTATION, REGION_TO_SEGMENTATION, REGION_TO_DESCRIPTION

There are some tasks that most of the times return empty content and rarely responds (usually with wrong stuff, as in the examples below): REGION_PROPOSAL , REGION_TO_CATEGORY and OD

• Response examples:

For REGION_PROPOSAL:

{"id":"chatcmpl-dd541c649a614816a56345bf2b53ed59","object":"chat.completion","created":1745510659,"model":"microsoft/Florence-2-large","choices":[{"index":0,"message":{"role":"assistant","reasoning_content":null,"content":"ualoks<loc_52><loc_333><loc_918><loc_774><loc_706><loc_198><loc_909><loc_546><loc_38><loc_140><loc_210><loc_366><loc_697><loc_574><loc_865><loc_772><loc_144><loc_581><loc_310><loc_772><loc_353><loc_184><loc_521><loc_342><loc_776><loc_383><loc_812><loc_421>","tool_calls":[]},"logprobs":null,"finish_reason":"stop","stop_reason":null}],"usage":{"prompt_tokens":588,"total_tokens":619,"completion_tokens":31,"prompt_tokens_details":null},"prompt_logprobs":null}

For OD :

{"id":"chatcmpl-c350d37e5f52471fb1f3891f6c1217a8","object":"chat.completion","created":1745510770,"model":"microsoft/Florence-2-large","choices":[{"index":0,"message":{"role":"assistant","reasoning_content":null,"content":" letterstamp<loc_10><loc_0><loc_994><loc_85><loc_0><loc_0><loc_163><loc_49><loc_61><loc_0><loc_278><loc_91><loc_74><loc_95><loc_75><loc_120><loc_76><loc_158><loc_82><loc_120><loc_125><loc_276><loc_128><loc_35><loc_779><loc_820><loc_283><loc_40><loc_100><loc_720><loc_375><loc_104><loc_290><loc_190><loc_533><loc_278><loc_290><loc_371><loc_373><loc_810><loc_190><loc_641><loc_295><loc_260><loc_276><loc_372><loc_304><loc_286><loc_627><loc_366><loc_708><loc_847><loc_583><loc_913><loc_649><loc_121><loc_447><loc_142><loc_534><loc_103><loc_293><loc_122><loc_367><loc_134><loc_378><loc_163><loc_443><loc_46><loc_298><loc_79><loc_324><loc_793><loc_194><loc_859><loc_206>taxi<loc_53><loc_331><loc_914><loc_775><loc_37><loc_106><loc_213><loc_368><loc_531><loc_97><loc_912><loc_436><loc_195><loc_198><loc_508><loc_363><loc_710><loc_178><loc_795><loc_237><loc_872><loc_126><loc_929><loc_185>","tool_calls":[]},"logprobs":null,"finish_reason":"stop","stop_reason":null}],"usage":{"prompt_tokens":590,"total_tokens":695,"completion_tokens":105,"prompt_tokens_details":null},"prompt_logprobs":null}

For REGION_TO_CATEGORY :
{INPUT} -> "<REGION_TO_CATEGORY> <loc_37><loc_136><loc_212><loc_370>"

{"id":"chatcmpl-12672daad94a4e8bbbd297c2d482a1b0","object":"chat.completion","created":1745509158,"model":"microsoft/Florence-2-large","choices":[{"index":0,"message":{"role":"assistant","reasoning_content":null,"content":"","tool_calls":[]},"logprobs":null,"finish_reason":"stop","stop_reason":null}],"usage":{"prompt_tokens":590,"total_tokens":591,"completion_tokens":1,"prompt_tokens_details":null},"prompt_logprobs":null}

This is what we are using:

source ../.env

docker run \
--runtime nvidia \
-e VLLM_USE_V1=0 \
--gpus 0 \
--ipc=host \
-p "8000:8000" \
--env "HUGGING_FACE_HUB_TOKEN=${HUGGING_FACE_HUB_TOKEN}" \
-v "${HF_HOME}:/root/.cache/huggingface" \
-v "$(pwd):/app" \
vllm/vllm-openai:latest \
--tensor-parallel-size 1 \
--model microsoft/Florence-2-large \
--tokenizer Isotr0py/Florence-2-tokenizer \
--gpu-memory-utilization 0.1 \
--trust-remote-code \
--chat-template /app/template_florence2.jinja \
--max-model-len 1024 \
--max-num-seqs 8 \
--dtype float16

An example of request:

 curl -X POST http://localhost:8000/v1/chat/completions \
   -d '{
     "model": "microsoft/Florence-2-large",
     "messages":  [
          {
              "role": "user",
              "content": [
                  {
                      "type": "image_url",
                      "image_url": {
                          "url": "https://raw.githubusercontent.com/AssemblyAI-Community/florence-2/refs/heads/master/car.png"
                      }
                  },
                  {"type": "text", "text": "{INPUT}"}
              ]
          }
      ],
	 "skip_special_tokens": false
 }'

@Isotr0py could you take a look in this?

[Isotr0py]

@PedroMiolaSilva Hmmm, I tried the below prompts and the outputs look reasonable to me:

    prompts = [
        {   # implicit prompt with task token
            "prompt": "<OD>",
            "multi_modal_data": {
                "image": ImageAsset("stop_sign").pil_image
            },
        },
        {   # implicit prompt with task token
            "prompt": "<REGION_PROPOSAL>",
            "multi_modal_data": {
                "image": ImageAsset("stop_sign").pil_image
            },
        },
        {   # implicit prompt with task token
            "prompt": "<REGION_TO_CATEGORY><loc_0><loc_0><loc_999><loc_675>",
            "multi_modal_data": {
                "image": ImageAsset("stop_sign").pil_image
            },
        },
    ]

    sampling_params = SamplingParams(
        temperature=0,
        top_p=1.0,
        max_tokens=64,
        skip_special_tokens=False,
        repetition_penalty=1.2,
    )

Outputs:

Processed prompts: 100%|██████████████████████████████████████████████████| 3/3 [00:00<00:00,  4.74it/s, est. speed input: 9.48 toks/s, output: 96.34 toks/s]
Decoder prompt: '</s>', Generated text: 'building<loc_0><loc_0><loc_999><loc_675>car<loc_524><loc_544><loc_920><loc_761>stop sign<loc_163><loc_132><loc_363><loc_422>wheel<loc_802><loc_674><loc_890><loc_758><loc_566><loc_677><loc_645><loc_760>'
Decoder prompt: '</s>', Generated text: '<loc_0><loc_1><loc_999><loc_678><loc_763><loc_0><loc_998><loc_663><loc_524><loc_544><loc_920><loc_761><loc_163><loc_132><loc_363><loc_422><loc_802><loc_674><loc_890><loc_758><loc_566><loc_677><loc_645><loc_760><loc_493><loc_489><loc_508><loc_507>'
Decoder prompt: '</s>', Generated text: 'building<loc_0><loc_0><loc_999><loc_675>'

Corresponding HF outputs:

Prompt: <OD>; Outputs: </s><s><s><s>building<loc_0><loc_0><loc_999><loc_675>car<loc_524><loc_544><loc_921><loc_762>stop sign<loc_163><loc_132><loc_363><loc_422>wheel<loc_802><loc_674><loc_890><loc_758><loc_566><loc_677><loc_645><loc_760></s>
Prompt: <REGION_PROPOSAL>; Outputs: </s><s><s><s><loc_0><loc_0><loc_999><loc_675><loc_0><loc_0><loc_310><loc_671><loc_763><loc_0><loc_999><loc_663><loc_524><loc_544><loc_921><loc_762><loc_163><loc_132><loc_363><loc_422><loc_802><loc_674><loc_890><loc_758><loc_566><loc_677><loc_646><loc_760></s>
Prompt: <REGION_TO_CATEGORY><loc_0><loc_0><loc_999><loc_675>; Outputs: </s><s>building<loc_0><loc_0><loc_999><loc_675></s>

Anyway, let me test on your image as well...

[Isotr0py]

Hmm, seems that offline inference can also generate reasonable outputs on car.png:

Decoder prompt: '</s>', Generated text: 'car<loc_52><loc_333><loc_932><loc_774>stop sign<loc_39><loc_140><loc_209><loc_366>wheel<loc_704><loc_576><loc_864><loc_772><loc_148><loc_584><loc_309><loc_773>'
Decoder prompt: '</s>', Generated text: '<loc_52><loc_333><loc_932><loc_774><loc_711><loc_202><loc_905><loc_545><loc_39><loc_140><loc_209><loc_366><loc_704><loc_576><loc_864><loc_772><loc_148><loc_584><loc_309><loc_773><loc_103><loc_555><loc_135><loc_616><loc_424><loc_503><loc_472><loc_514><loc_637><loc_642><loc_646><loc_668><loc_795><loc_358><loc_804><loc_367><loc_120><loc_339><loc_128><loc_348><loc_219><loc_631><loc_231><loc_636>'
Decoder prompt: '</s>', Generated text: 'stop sign<loc_37><loc_136><loc_212><loc_370>'

Perhaps the issue is about online serving?

[Isotr0py]

@PedroMiolaSilva You can try this:

 curl -X POST http://localhost:8000/v1/chat/completions \
   -H 'Content-Type: application/json' \
   -d '{
     "model": "microsoft/Florence-2-large",
     "messages":  [
          {
              "role": "user",
              "content": [
                  {
                      "type": "image_url",
                      "image_url": {
                          "url": "https://raw.githubusercontent.com/AssemblyAI-Community/florence-2/refs/heads/master/car.png"
                      }
                  },
                  {"type": "text", "text": "<OD>"}
              ]
          }
      ],
     "temperature": 0.0,
     "repetition_penalty": 1.2,
     "min_tokens": 2,
     "skip_special_tokens": false
 }'

Seems that the first <s>(bos token) will stop the generation in online serving, so we have to add "min_tokens". And the default temperature is 1.0, so it could be too high to get accurate results.

Outputs:

{"id":"chatcmpl-5bcefd0192c5400aa4201f7e29d126df","object":"chat.completion","created":1745686827,"model":"microsoft/Florence-2-large","choices":[{"index":0,"message":{"role":"assistant","reasoning_content":null,"content":"nel<loc_52><loc_334><loc_932><loc_774>stop sign<loc_39><loc_140><loc_209><loc_366>wheel<loc_704><loc_576><loc_864><loc_772><loc_148><loc_584><loc_309><loc_773>","tool_calls":[]},"logprobs":null,"finish_reason":"stop","stop_reason":null}],"usage":{"prompt_tokens":590,"total_tokens":611,"completion_tokens":21,"prompt_tokens_details":null},"prompt_logprobs":null}

[PedroMiolaSilva]

@Isotr0py it worked here. Thanks again!