[Contributor] Proposal - adding node bindings, similar to whisper, a la examples/addon.node

[yacineMTB]

Prerequisites

• I am running the latest code. Development is very rapid so there are no tagged versions as of now.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new bug or useful enhancement to share.

Proposal

My proposal is to add node bindings, under examples/addon.node similar to whisper.cpp's node addon

Why

• There already has been work on it, and I think it might be useful for other people to have!

Why you should stop me

• More maintenance
• More things for people to create issues regarding
• More code - diffs to main might necessitate a diff to addon
• If I could write it myself, maybe other people should too!

If a contributor says yes, I'll clean up my diffy, and attempt to mainline it.

A no is completely reasonable!

cc @cto-junior - helping me with the cpp

[bakkot]

One of the roadmap items is a general refactor, and another is to clean up state in general, for which there's now a PR. You might want to wait until those are done (at least the open PR) before adding new bindings which depend on the current design.

[ghost]

@bakkot Thanks for the quick response! THe PR looks backward compatible right now but you're right it might go through some changes in the process.

I also wanted to know if making lora adapters hot-swappable in the pipeline or not?

I did find #905

[ggerganov]

Adding examples/addon.node is welcome

In general, the purpose of the examples is to demonstrate usage of llama.cpp. We rely heavily on contributions to maintain those and we are fine if they break at some point and there is no longer interest in supporting them. Having a dedicated CI that runs upon local changes in the example is optional, but recommended to ensure longer term support.
If the example comes with no or little changes to llama.cpp and ggml, then it is always easy to get merged

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.