The output from the WizardCoder model has leading whitespace.

[MinjeJeon]

Describe the bug
A clear and concise description of what the bug is.

After launching a self-hosted server with the WizardCoder-3B model and using auto-completion in VSCode, there is an issue where additional whitespace is added to the generated string at the beginning.

Information about your GPU
Please provide output of nvidia-smi

Additional context
Add any other context about the problem here.

I am using the tabby docker image 0.1.2.
Tabby extension version: 0.5.0
VSCode version: 1.82.2

[wsxiaoys]

Thank you for reporting the bug. This seems to be a case that can be addressed through post-processing.

Additionally, enhancing the prompt template of WizardCoder might also lead to improved output.

[wsxiaoys]

With the decoding fix released in v0.2.1, this situation should have improved to some extent.

curl -X 'POST' \
  'http://localhost:8080/v1/completions' \
  -H 'Accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "language": "python",
  "segments": {
    "prefix": "import requests\nimport "
  }
}'

{
  "id": "cmpl-f063d751-036d-459b-b708-e1e5f33614ad",
  "choices": [
    {
      "index": 0,
      "text": "icalendar\nimport datetime\nimport pytz\nimport os\nimport sys\nimport json\nimport logging\nimport time\nimport re\nimport traceback\nimport threading\nimport subprocess\nimport shutil\nimport hashlib\nimport xml.etree.ElementTree as ETree\n"
    }
  ]
}

Hi @MinjeJeon, could you please verify if you have observed an improvement in your experience?

[MinjeJeon]

I tested it with the curl you told me and it works well.

❯❯❯ curl -X 'POST' \
  'http://localhost:8080/v1/completions' \
  -H 'Accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "language": "python",
  "segments": {
    "prefix": "import requests\nimport "
  }
}'
{"id":"cmpl-aba64349-6c25-46f2-80e0-9663536ccc5d","choices":[{"index":0,"text":"icalendar\nimport datetime\nimport pytz\nimport os\nimport sys\nimport time\nimport json\nimport re\nimport logging\nimport logging.config\n"}]}

But when I ran another auto-completion, I noticed that there was leading whitespace as shown below. I wonder if the --chat-model parameter would help.

curl -X 'POST' \
  'http://localhost:8080/v1/completions' \
  -H 'Accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "language": "python",
  "segments": {
    "prefix": "# calculate fibnacci series up to nth term\ndef "
  }
}'
{"id":"cmpl-ecf24bb4-5f1e-4292-b3a1-a26876ef6745","choices":[{"index":0,"text":" fibonacci(n):\n    if n == 0:\n        return 0\n    elif n == 1:\n        return 1\n    else:\n        return fibonacci(n-1) + fibonacci(n-2)\n"}]}

Thank you for your reply.

[MinjeJeon]

I tested it with the 1.0.0 vscode extension. I confirmed that the problem was solved.