Tmp fix illegal sample

[hnyls2002]

No description provided.

[ispobock]

@hnyls2002 I also faced this illegal sample result issue. In my cases (run llama2-13b-chat model on A100-80G) it can be reproduced by this request from ShareGPT dataset:

curl -X POST "http://127.0.0.1:30000/generate" \
     -H "Content-Type: application/json" \
     -d '{
           "text": "| Article Topic | Target Persona | Target Search Phrase |\n| --- | --- | --- |\n| Understanding Defects Liability Periods in Building Contracts | Developers | Defects Liability Period in Building Contracts |\n| What Homeowners Need to Know About Defects Liability Periods | Homeowners | Defects Liability Period for Homeowners |\n| How Builders Can Manage Defects Liability Periods | Builders | Defects Liability Period Management for Builders |\n| Contractors'\'' Obligations During the Defects Liability Period | Contractors | Contractor Obligations in Defects Liability Period |\n| Benefits of a Defects Liability Period for Developers | Developers | Benefits of Defects Liability Period for Developers |\n| Homeowners'\'' Rights During the Defects Liability Period | Homeowners | Homeowner Rights in Defects Liability Period |\n| Importance of Documenting Defects During the Liability Period | Builders | Documenting Defects in Defects Liability Period |\n| Defects Liability Period Extensions and Their Implications | Contractors | Extensions of Defects Liability Period |\n| How to Ensure Compliance with the Defects Liability Period | Developers | Ensuring Compliance in Defects Liability Period |\n| Common Misconceptions About Defects Liability Periods | Homeowners | Misconceptions About Defects Liability Periods |\n| Legal Implications of Defects Liability Periods for Builders | Builders | Legal Implications of Defects Liability Periods |\n| Remedies Available for Homeowners During Defects Liability Periods | Homeowners | Remedies for Homeowners in Defects Liability Period |\n| How to Prepare for the End of Defects Liability Period | Developers | Preparing for End of Defects Liability Period |\n| Understanding Defects Liability Periods in Subcontracting | Contractors | Defects Liability Period in Subcontracting |\n| Best Practices for Managing Defects Liability Periods | Builders | Best Practices for Defects Liability Period Management |",
           "sampling_params": {
               "temperature": 1,
               "ignore_eos": true,
               "max_new_tokens": 200
           }
         }' 

If I don't set the top_k, the first decoded token_id will be illegal (1057103872)
If I set top_k=1, the 78th decoded token_id will be illegal (-1062346752)

It seems like an issue for top_k_top_p_sampling_from_probs kernel. (ref: flashinfer-ai/flashinfer#384)
Hope we can find the root cause soon.