Extensibility

[reluctantfuturist]

🚀 Describe the new functionality needed

Being able to reconfigure/reassemble LLS at deploy time (extending the distribution concept): i.e. between build time (aka lls build and runtime (reconfiguration via API endpoints)

💡 Why is this needed? What if we don't build it?

Needed for enterprise deployment reliability

Other thoughts

No response

[cdoern]

could I be assigned to this as well? This will probably be taken care of in conjunction with #2386 to ensure both of these things can be supported usage patterns. thanks!

[rhuss]

It's important to clarify what we mean by "dynamic reconfiguration." Rather than trying to hot-reload providers while the LlamaStack server is actively running (which would be complex and error-prone), I think we should focus on deployment-time composition. In other words, new providers and their libraries are added before the LLS server starts (in a Kubernetes setup, during the Pod initialization), not injected on-the-fly via a running API. This deploy-time approach covers the vast majority of use cases and keeps the runtime stable, avoiding the tricky corner cases of live hot-reloading.

By adopting deploy-time modularity, we align with the Bring Your Own Provider (BYOP) model in LlamaStack introduced in #1672, which encourages out-of-tree providers (external plugins) similar to how OpenStack and Kubernetes handle extensions, discussed in #1717

This means the core LLS distribution can stay lean, and users can plug in additional providers as needed at deploy time.

Settling on the BYOP introduced by @leseb we can easily construct the out-of-tree providers directory in provider.de during initialisation time.

However, it's important that all need Python packages need to be available within this container so that the whole system stays self contained and works also in disconnected environment (i.e. when the K8s cluster does not has access to the internet). This probably only works nicely with pure Python packages (for the same major Python version) as native code might not be compatible with the LLS base server.

For providers that require native libs, one could either split that up in a pure-python part that implements a remote provider where the external deployment holds the native part. For an ease of deployment one could consider this to become a sidecar in the LLS instance deployment, so communication would go over localhost (or IPC if absolutely needed).

[cdoern]

@rhuss so this means basically during the llama stack run process someone can inject new deps and providers that weren't included in the build process? I can open a different PR to enable this functionality!

[rhuss]

My intention was that during startup the distribution might be assembled but that is already provided IMO by #1672. It's more for the non-local case, when LLS instance is running from within a container that does not has the dependencies installed already. Those should not be dynamically reinstalled during startup but picked up through an pre-process like init-containers.

Let me try to create a more detailed spec on this to clarify what I mean. Running pip install in llama stack run is IMO not the solution.

For the local use case, you can do this manually. But for the in-cluster use-case this is more tricky to avoid pulling down python package when starting up the server.

[github-actions]

This issue has been automatically marked as stale because it has not had activity within 60 days. It will be automatically closed if no further activity occurs within 30 days.

[rhuss]

In the meantime we opened #3751 to track extensibility on Kubernetes. Ideally this can be fully implemented in the operator.

[reluctantfuturist]

In the meantime we opened #3751 to track extensibility on Kubernetes. Ideally this can be fully implemented in the operator.

Let's close this one then in favor of the new k8s issue?

[rhuss]

yeah, good idea, but then please add #3751 to the "Roadmap to V1" umbrella issue, replacing this issue.