Skip to content

Conversation

dineshyv
Copy link
Contributor

@dineshyv dineshyv commented Nov 19, 2024

This PR adds a method in stack to return the stackrunconfig object based on the template name. This will be used to instantiate a direct client without the need for an explicit run.yaml
Test plan:

  1. pip install -e . on both llama stack and llama stack client with add ability to instantiate direct client with template llama-stack-client-python#37
  2. Bring up ollama
  3. Create a direct client:
import asyncio
from llama_stack_client.lib.direct.direct import LlamaStackDirectClient
from llama_stack.distribution.stack import (
    get_stack_run_config_from_template,
)


async def main():
    config = get_stack_run_config_from_template("ollama")
    print(f"using the following config: {config}")
    client = LlamaStackDirectClient(config)
    await client.initialize()

    response = await client.models.list()
    print(response)


if __name__ == "__main__":
    asyncio.run(main())

INFERENCE_MODEL=llama3.2:1b-instruct-fp16 python app.py

@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Meta Open Source bot. label Nov 19, 2024
@dineshyv dineshyv force-pushed the dineshyv/stackrun-template branch from 42e92d0 to c8351b9 Compare November 19, 2024 00:22
@ashwinb
Copy link
Contributor

ashwinb commented Nov 19, 2024

There should be a way to print out this run.yaml since the user won't know exactly what config they are starting from. DirectClient probably should dump it out when starting.

@dineshyv dineshyv merged commit fe19076 into main Nov 19, 2024
2 checks passed
@dineshyv dineshyv deleted the dineshyv/stackrun-template branch November 19, 2024 02:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Meta Open Source bot.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants