ENH: Removed the max tokens limitation and boost performance by avoid unnecessary repeated cuda device detection.

[mikeshi80]

Revert codeqwen1.5 to 64k context length.

Also optimized code to avoid query the whether cuda device exists and count the cuda device to boost performance.

[qinxuye]

Can you show that without modified the max_token_field, if max_tokens is passed(e.g. 64k), there is issue with chat?

[mikeshi80]

Because max_tokens field has the limitation that the value cannot be greater than 32768, when the max_tokens exceeds it, the server will report the error as below:

xinference-local: raise validation_error
xinference-local: pydantic.v1.error_wrappers.ValidationError: 1 validation error for CreateChatCompletion
xinference-local: max_tokens
xinference-local: ensure this value is less than or equal to 32768 (type=value_error.number.not_le; limit_value=32768)

And I printed the CreateChatCompletion's schema with schema_json, NOTICE: some properties have been reduced in case of too long text.

{
  "title": "CreateChatCompletion",
  "type": "object",
  "properties": {
    "frequency_penalty": {
      "title": "Frequency Penalty",
      "default": 0.0,
      "type": "number"
    },
    "logit_bias": {
      "title": "Logit Bias",
      "type": "object",
      "additionalProperties": {
        "type": "number"
      }
    },
    "logprobs": {
      "title": "Logprobs",
      "type": "integer"
    },
    "max_tokens": {
      "title": "Max Tokens",
      "description": "The maximum number of tokens to generate.",
      "default": 1024,
      "minimum": 1,
      "maximum": 32768,
      "type": "integer"
    },
    "stream_interval": {
      "title": "Stream Interval",
      "default": 2,
      "type": "integer"
    }
  },
  "required": [
    "model"
  ],
  "additionalProperties": false
}

The limitation for max_tokens is defined in schema, so the CreateChatCompletion.parse_obj will raise errors if max_tokens exceeded 32768.

[qinxuye]

Could you rebase the main branch?

[mikeshi80]

synchronized

[qinxuye]

LGTM