fix: #3558 wrong model metadata import or download from HuggingFace

[louis-jan]

Describe Your Changes

When downloading the HF GGUF model via URL import from the model hub, the model is using default settings instead of the correct settings. The fixed context length is set to 2048, the prompt template is incorrect, and there is an issue with the stop word. However, importing GGUF directly seems to resolve these issues.

Steps to reproduce

Download the HF GGUF model from the model hub using URL import.
Check the settings for context length, prompt template, and stop word.
Compare the settings with the correct settings for the GGUF model.

Expected behavior

The HF GGUF model downloaded via URL import should have the correct settings for context length, prompt template, and stop word, matching the settings when GGUF is imported directly.

Additional context

The issue seems to be specific to downloading the HF GGUF model via URL import from the model hub. Importing GGUF directly does not exhibit the same issue with default settings.

Fixes Issues

• Fixes bug: Dropping GGUF metadata when creating model.yaml for Model Downloaded via URL #3558
• Fixes bug: Wrong maximum context length for qwen2.5-coder #3714
• Fixes bug: Model downloaded from Hugging Face can only run on CPU because it doesn't retrieve the correct GGUF model's metadata #3708

Screenshots

Qwen 2.5 download from HF

Code Changes

1. extensions/model-extension/src/index.test.ts
2. extensions/model-extension/src/index.ts
3. extensions/model-extension/src/node/index.ts
4. extensions/model-extension/src/node/node.test.ts

Here's a summary of the major changes across these files:

index.test.ts

• Added mock functions for new dependencies (downloadMock, mkdirMock, writeFileSyncMock, copyFileMock).
• Added setup for a global fetch mock.
• Imported additional functions for testing (renderJinjaTemplate, Template).
• Added a new test case for downloadModel with invalid gguf metadata.

index.ts

• Added support for downloading HF model metadata by fetching and retrieving GGUF metadata.
• Improved the creation and update of model settings and parameters using fetched metadata.
• Added a new method retrieveGGUFMetadata that returns a partially updated model based on metadata.
• Handled conflicts and updated parameters and settings using GGUF metadata.
• Refactored the eos token handling and merging model configurations.

node/index.ts

• Removed the original implementation of renderedTemplate from the GGUF metadata parsing process.
• Added a new function renderJinjaTemplate for parsing Jinja templates based on the metadata.

node/node.test.ts

• Added tests for the renderJinjaTemplate function.
• Includes tests handling cases where token IDs are missing or the template is empty.

Detailed Examples

• GGUF Metadata Handling: Added code to check for and update model settings based on metadata attributes like eos_token_id, context_length, block_count, etc.

  await fs.writeFileSync(modelJsonPath, JSON.stringify(model, null, 2))
  const updatedModel = await this.retrieveGGUFMetadata(metadata)
  model.settings = { ...model.settings, ...updatedModel.settings };

• Infrastructural Mocks in Tests: Created mocks and their respective test setups to emulate filesystem and network behaviors for more deterministic tests.

  global.fetch = jest.fn(() => Promise.resolve(...));
  expect(sut.downloadModel(...)).rejects.toBeTruthy();

These changes all seem aimed to improve the robustness and flexibility of model handling within the codebase, especially concerning dynamically adjusting to metadata from fetched models.

[github-actions]

Barecheck - Code coverage report

Total: 62.02%

Your code coverage diff: 0.00% ▴

✅ All code changes are covered

[dan-homebrew]

Thank you @louis-jan!