Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Cooperation/Announcement] Discover and use llamafile on ModelScope #665

Open
4 tasks done
yingdachen opened this issue Dec 27, 2024 · 0 comments
Open
4 tasks done

Comments

@yingdachen
Copy link

yingdachen commented Dec 27, 2024

Prerequisites

  • I am running the latest code. Mention the version if possible as well.
  • I carefully followed the README.md.
  • I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
  • I reviewed the Discussions, and have a new and useful enhancement to share.

Feature Description

ModelScope is the one of the leading open-model communities, and has recently incorporated native support for llamafile through it command line tools.

Simply install ModelScope with pip install modelscope -U, and you will have access to a wealth of llamafile models on ModelScope. Get it up and running with ModelScope command modelscope llamafile --model {model_id} , for example:
modelscope llamafile --model Qwen-Llamafile/Qwen2.5-3B-Instruct-llamafile.

There are also a few other running options such as to specify quantiazation level:
modelscope llamafile --model Qwen-Llamafile/Qwen2.5-3B-Instruct-llamafile --accuracy Q2_K

More details can be found in documentation here and here (a bit of google translation needed since the doc is in Chinease).

Currently there are 350+ llamafile repos on ModelScope, covering almost all major LLMs. It is definitely the largest collection out there. All of which you can access directly via link.

ModelScope is also open sourced at https://github.com/modelscope/modelscope

Motivation

We are glad to share this new feature with the llamafile comunities, and we welcome any new request or contribution of new llamafiles to ModelScope!

Cheers.

Possible Implementation

No response

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant