Skip to content

feat: Cortex pull from HF can select quantization #1006

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
Tracked by #1077
louis-menlo opened this issue Aug 11, 2024 · 3 comments
Closed
Tracked by #1077

feat: Cortex pull from HF can select quantization #1006

louis-menlo opened this issue Aug 11, 2024 · 3 comments
Assignees
Labels
category: model management Model pull, yaml, model state P1: important Important feature / fix
Milestone

Comments

@louis-menlo
Copy link
Contributor

louis-menlo commented Aug 11, 2024

Describe the bug
When pulling a model from HF, I could not select the model quantization, it downloads the first one by default (lowest quality)

To Reproduce
Steps to reproduce the behavior:

  1. run cortex
  2. pull any models on HF E.g. TheBlock/Tiny
  3. it pulls Q2 by default
  4. see error

Expected behavior
It should prompt users to select the quantization they want to pull

@louis-menlo louis-menlo added the type: bug Something isn't working label Aug 11, 2024
@imtuyethan imtuyethan moved this to Need Investigation in Menlo Sep 2, 2024
@Van-QA
Copy link
Contributor

Van-QA commented Sep 4, 2024

The issue is reproducible and needs ‌‌investigation from dev team, but since we are focusing on the cortex.cpp, I think we can wait until this feature is implemented in the new version of cortex.cpp

Image

@freelerobot freelerobot changed the title bug: model quantization selection is gone bug: can't select model quantizations Sep 6, 2024
@freelerobot freelerobot added P1: important Important feature / fix category: model management Model pull, yaml, model state labels Sep 6, 2024
@dan-menlo dan-menlo assigned namchuai and unassigned vansangpfiev and Van-QA Sep 6, 2024
@dan-menlo dan-menlo changed the title bug: can't select model quantizations feat: Cortex pull from HF can select quantization Sep 6, 2024
@dan-menlo dan-menlo removed the type: bug Something isn't working label Sep 6, 2024
@dan-menlo
Copy link
Contributor

dan-menlo commented Sep 6, 2024

@namchuai I am converting this into a feature, as it's a part of cortex pull and cortex run functionality.

We can close ticket once you have verified that cortex.cpp can trigger quantization selection when downloading HF GGUF download

@freelerobot
Copy link
Contributor

works

@github-project-automation github-project-automation bot moved this from In Review to Completed in Menlo Sep 23, 2024
@gabrielle-ong gabrielle-ong added this to the v1.0.0 milestone Oct 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
category: model management Model pull, yaml, model state P1: important Important feature / fix
Projects
Archived in project
Development

No branches or pull requests

7 participants