Librarification

[philpax]

This is a very rough split of the CLI application into a library and application. It includes #8 because that was on my branch when I did this - we can figure out what to do with it when it's time.

The main differences are

• conversion of printlns to logs
• removal of prints
• adding a callback function for arrival of new tokens

Turns out this is basically entirely sufficient for first-pass use as a library. Here's proof.

[mwbryant]

Everything LGTM with some very minor nits. I've tested it and with a fixed seed it gives the same results and performance as the original. Needing to set the flags (which also required Rust 1.68) for the build script is still a pain but I'm sure there's a solution to that.

Seems to be resilient to all the crazy things I tried, inference can be called multiple times without issue.

In the future it would be nice to have inference with a continuous state between calls (i.e. an interactive mode)

[setzer22]

This was quick! 😄 Thanks a lot for the changes.

Not much to add on top of @mwbryant's review, but I left a couple additional comments.

[setzer22]

Amazing work on llamacord btw! 😄 I'm gonna try this out

[philpax]

Thanks for the feedback, everyone! Will action within the next few hours so we can get this merged 👍

[philpax]

The llama-rs crate now has no logging of its own; all progress is reported through the callback. Pretty happy with how that turned out!

I've also switched over to thiserror. I've tried to remain consistent with the existing names, but I'd like to agree on #17 before we have any major users.

As an aside: should the CLI be llama-cli or llama-rs-cli?

[setzer22]

As an aside: should the CLI be llama-cli or llama-rs-cli

I was originally thinking llama-rs-cli. But after seeing the name, I really like llama-cli, especially if it ends up in someone's path via cargo install, it's easier to type. So if you prefer that one I'm on board as well!

[setzer22]

Amazing work, thanks a lot again for this!

Left a few minor comments, should be quick enough to address 👍

There's also a merge conflict since we merged #10. I'd suggest just solving it in merge, and not rebase (it's cleaner IMO, and some of us already pulled that branch).

[setzer22]

A very clean solution! 😄 Having this will be especially important if we end up implementing parallel loading as described on #15. Because relying on the order of print messages for the little dots breaks the moment you start multithreading.