Embeding LLamaSharp inside .NET MAUI Android App

[Karimiprogramer]

Description

I am a .NET MAUI develpoer and I am interested in embeding a LLM inside my application with LLamaSharp. After build llama.cpp with ndk , How can I embed libraries on my application??

[andersonvieiragomeslopes]

Description

I am a .NET MAUI develpoer and I am interested in embeding a LLM inside my application with LLamaSharp. After build llama.cpp with ndk , How can I embed libraries on my application??

I'm also interested, did you manage to use it?

[Karimiprogramer]

After solving some problems with embeding model inside app , I will publish it into my github.

[Darlanio]

Looking forward to your solution to this problem. I am not sure if llamasharp would work out of the box on android/ios, but maybe it is possible to compile llama.cpp for the specified platform?

At least llama.cpp runs under Termux on Android...

I will keep an eye on your repository:
https://github.com/Karimiprogramer/Sigma

[Darlanio]

Tested on 0.19 without compiling the binaries myself and got the exception System.PlatformNotSupportedException: 'Operation is not supported on this platform.' on the ModelParams constructor.

[Darlanio]

Looking foward to Android supported nuget libraries.

[homerokzam]

Is there a solution for iOS?

[andersonvieiragomeslopes]

"Hey @AmSmart, could you join this discussion here? I saw that you had success with this Maui Android Backend Support."

[AmSmart]

Yeah, I have a PR that brings Android support for this library. Just that I haven't had enough time to write the tests and all. I'll open my PR this weekend and see if it's good enough to get merged. Also planning an IOS update in the future.

[amirvenus]

Yeah, I have a PR that brings Android support for this library. Just that I haven't had enough time to write the tests and all. I'll open my PR this weekend and see if it's good enough to get merged. Also planning an IOS update in the future.

Any updates on this, @AmSmart ?

This will be a really cool feature!

[Karimiprogramer]

I have successfully built CPU-based versions of libllama and libgguf for Android and imported them into my .NET MAUI application. However, I am facing some challenges in utilizing these libraries effectively within my app.

I would greatly appreciate any guidance or support you can provide to help me overcome these issues. Thank you!

[martindevans]

One thing to check - did you build exactly the llama.cpp version listed in the table at the bottom of the readme? Each version of LLamaSharp is only compatible with one specific version of llama.cpp!

[Darlanio]

I wish I had time to check this now, but will have to wait until I have time in April.

[AmSmart]

Update: The extra demand here gave me the motivation to complete my PR #1130 introducing Android support. Will still require a bit of back and forth but I hope it can get merged soon enough.

I've built this nupkgs off my PR so anyone blocked can use them to experiment temporarily. Reiterating that these are not official packages from the repo so it is definitely worth waiting for the next release with official Android support.