Skip to content

LCORE-792: Lightspeed core needs to fully support VertexAI LLM provider #924

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
tisnik merged 1 commit into from
Dec 16, 2025
Merged

LCORE-792: Lightspeed core needs to fully support VertexAI LLM provider #924

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
67 changes: 60 additions & 7 deletions .github/workflows/e2e_tests.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ jobs:
fail-fast: false
matrix:
mode: ["server", "library"]
environment: ["ci", "azure"]
environment: ["ci", "azure", "vertexai"]

name: "E2E: ${{ matrix.mode }} mode / ${{ matrix.environment }}"

Expand Down Expand Up @@ -52,8 +52,7 @@ jobs:
- name: Load lightspeed-stack.yaml configuration
run: |
MODE="${{ matrix.mode }}"
CONFIG_FILE="tests/e2e/configuration/lightspeed-stack-${MODE}-mode.yaml"

CONFIG_FILE="tests/e2e/configuration/${MODE}-mode/lightspeed-stack.yaml"
echo "Loading configuration for ${MODE} mode"
echo "Source: ${CONFIG_FILE}"

Expand Down Expand Up @@ -91,6 +90,45 @@ jobs:
echo "✅ Successfully obtained Azure access token."
echo "AZURE_API_KEY=$ACCESS_TOKEN" >> $GITHUB_ENV

- name: Save VertexAI service account key to file
if: matrix.environment == 'vertexai'
env:
GOOGLE_SA_KEY: ${{ secrets.GOOGLE_SA_KEY }}
run: |
echo "Setting up Google Cloud service account credentials..."

if [ -z "$GOOGLE_SA_KEY" ]; then
echo "❌ GOOGLE_SA_KEY is not set. Please configure the secret in GitHub repository settings."
exit 1
fi

GCP_KEYS_PATH=./tmp/.gcp-keys
echo "GCP_KEYS_PATH=$GCP_KEYS_PATH" >> $GITHUB_ENV

mkdir -p $GCP_KEYS_PATH

echo "Writing service account key to file..."

# Decode from base64, needed because GH changes the key if using the raw key
printf '%s' "$GOOGLE_SA_KEY" | base64 -d > $GCP_KEYS_PATH/gcp-key.json

# Verify the file was created and is valid JSON
if [ ! -f "$GCP_KEYS_PATH/gcp-key.json" ]; then
echo "❌ Failed to create gcp-key.json file"
exit 1
fi

if ! jq empty "$GCP_KEYS_PATH/gcp-key.json" 2>/dev/null; then
echo "❌ gcp-key.json is not valid JSON"
exit 1
fi
echo "✅ gcp-key.json is valid JSON"

# Set proper permissions (readable by all, needed for container user 1001)
chmod 644 $GCP_KEYS_PATH/gcp-key.json

echo "GOOGLE_APPLICATION_CREDENTIALS=/opt/app-root/.gcp-keys/gcp-key.json" >> $GITHUB_ENV

- name: Select and configure run.yaml
env:
CONFIG_ENVIRONMENT: ${{ matrix.environment || 'ci' }}
Expand Down Expand Up @@ -146,19 +184,30 @@ jobs:
run: |
echo $QUAY_ROBOT_TOKEN | docker login quay.io -u=$QUAY_ROBOT_USERNAME --password-stdin

- name: Create dummy GCP keys directory
if: matrix.environment != 'vertexai'
run: |
echo "Creating dummy GCP keys directory for non-VertexAI environment..."
mkdir -p ./tmp/.gcp-keys-dummy
echo "✅ Dummy directory created."
Comment on lines +187 to +192
Copy link
Contributor

@coderabbitai coderabbitai bot Dec 16, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical

Critical: GCP_KEYS_PATH not set for non-VertexAI environments.

The dummy directory is created, but the GCP_KEYS_PATH environment variable is never set for non-VertexAI environments (ci, azure). This causes GCP_KEYS_PATH to be undefined when passed to docker-compose at lines 202 and 228, which may lead to volume mount failures.

Apply this fix:

 - name: Create dummy GCP keys directory
   if: matrix.environment != 'vertexai'
   run: |
     echo "Creating dummy GCP keys directory for non-VertexAI environment..."
     mkdir -p ./tmp/.gcp-keys-dummy
+    echo "GCP_KEYS_PATH=./tmp/.gcp-keys-dummy" >> $GITHUB_ENV
     echo "✅ Dummy directory created."
🤖 Prompt for AI Agents 
.github/workflows/e2e_tests.yaml around lines 187 to 192: the dummy GCP keys
directory is created for non-VertexAI environments but GCP_KEYS_PATH is not set,
so downstream steps (docker-compose) receive an undefined path; update the
workflow to export GCP_KEYS_PATH pointing to the created dummy directory so
subsequent steps see it (for example, append
"GCP_KEYS_PATH=./tmp/.gcp-keys-dummy" to $GITHUB_ENV or set it in the step's
env) ensuring the variable is available to later steps that use it for
docker-compose volume mounts.

Copy link
Contributor Author

@are-ces are-ces Dec 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is addressed by setting a fallback GCP_KEYS_PATH in the docker-compose.yaml

Copy link
Contributor

@coderabbitai coderabbitai bot Dec 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Verify the GCP_KEYS_PATH fallback in docker-compose files
echo "=== Checking docker-compose.yaml ==="
grep -n "GCP_KEYS_PATH" docker-compose.yaml

echo -e "\n=== Checking docker-compose-library.yaml ==="
grep -n "GCP_KEYS_PATH" docker-compose-library.yaml

Length of output: 238

@are-ces You're absolutely right! I can see both docker-compose.yaml and docker-compose-library.yaml use the syntax ${GCP_KEYS_PATH:-./tmp/.gcp-keys-dummy}, which provides a fallback value when the environment variable is unset. This means the volume mount will work correctly for non-VertexAI environments even without explicitly setting GCP_KEYS_PATH in the workflow.

My original concern was incorrect—the code already handles this case properly. Thank you for the clarification! 👍


- name: Run services (Server Mode)
if: matrix.mode == 'server'
env:
env:
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
AZURE_API_KEY: ${{ env.AZURE_API_KEY }}
VERTEX_AI_LOCATION: ${{ secrets.VERTEX_AI_LOCATION }}
VERTEX_AI_PROJECT: ${{ secrets.VERTEX_AI_PROJECT }}
GOOGLE_APPLICATION_CREDENTIALS: ${{ env.GOOGLE_APPLICATION_CREDENTIALS }}
GCP_KEYS_PATH: ${{ env.GCP_KEYS_PATH }}
run: |
# Debug: Check if environment variable is available for docker-compose
echo "OPENAI_API_KEY is set: $([ -n "$OPENAI_API_KEY" ] && echo 'YES' || echo 'NO')"
echo "OPENAI_API_KEY length: ${#OPENAI_API_KEY}"

docker compose version
docker compose up -d

# Check for errors and show logs if any services failed
if docker compose ps | grep -E 'Exit|exited|stopped'; then
echo "Some services failed to start - showing logs:"
Expand All @@ -173,10 +222,14 @@ jobs:
env:
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
AZURE_API_KEY: ${{ env.AZURE_API_KEY }}
VERTEX_AI_LOCATION: ${{ secrets.VERTEX_AI_LOCATION }}
VERTEX_AI_PROJECT: ${{ secrets.VERTEX_AI_PROJECT }}
GOOGLE_APPLICATION_CREDENTIALS: ${{ env.GOOGLE_APPLICATION_CREDENTIALS }}
GCP_KEYS_PATH: ${{ env.GCP_KEYS_PATH }}
run: |
echo "Starting service in library mode (1 container)"
docker compose -f docker-compose-library.yaml up -d

if docker compose -f docker-compose-library.yaml ps | grep -E 'Exit|exited|stopped'; then
echo "Service failed to start - showing logs:"
docker compose -f docker-compose-library.yaml logs
Expand Down
4 changes: 4 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -121,6 +121,7 @@ Lightspeed Core Stack is based on the FastAPI framework (Uvicorn). The service i
|----------------|-----------------------------------------------------------------------|
| OpenAI | https://platform.openai.com |
| Azure OpenAI | https://azure.microsoft.com/en-us/products/ai-services/openai-service |
| Google VertexAI| https://cloud.google.com/vertex-ai |
| RHOAI (vLLM) | See tests/e2e-prow/rhoai/configs/run.yaml |
| RHEL AI (vLLM) | See tests/e2e/configs/run-rhelai.yaml |

Expand Down Expand Up @@ -175,6 +176,9 @@ __Note__: Support for individual models is dependent on the specific inference p
| RHEL AI (vLLM)| meta-llama/Llama-3.1-8B-Instruct | Yes | remote::vllm | [1](tests/e2e/configs/run-rhelai.yaml) |
| Azure | gpt-5, gpt-5-mini, gpt-5-nano, gpt-5-chat, gpt-4.1, gpt-4.1-mini, gpt-4.1-nano, o3-mini, o4-mini | Yes | remote::azure | [1](examples/azure-run.yaml) |
| Azure | o1, o1-mini | No | remote::azure | |
| VertexAI | google/gemini-2.0-flash, google/gemini-2.5-flash, google/gemini-2.5-pro [^1] | Yes | remote::vertexai | [1](examples/vertexai-run.yaml) |

[^1]: List of models is limited by design in llama-stack, future versions will probably allow to use more models (see [here](https://github.com/llamastack/llama-stack/blob/release-0.3.x/llama_stack/providers/remote/inference/vertexai/vertexai.py#L54))

The "provider_type" is used in the llama stack configuration file when refering to the provider.

Expand Down
14 changes: 11 additions & 3 deletions docker-compose-library.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,20 +12,28 @@ services:
# Mount both config files - lightspeed-stack.yaml should have library mode enabled
- ./lightspeed-stack.yaml:/app-root/lightspeed-stack.yaml:Z
- ./run.yaml:/app-root/run.yaml:Z
- ${GCP_KEYS_PATH:-./tmp/.gcp-keys-dummy}:/opt/app-root/.gcp-keys:ro
environment:
# LLM Provider API Keys
- BRAVE_SEARCH_API_KEY=${BRAVE_SEARCH_API_KEY:-}
- TAVILY_SEARCH_API_KEY=${TAVILY_SEARCH_API_KEY:-}
# OpenAI
- OPENAI_API_KEY=${OPENAI_API_KEY}
- E2E_OPENAI_MODEL=${E2E_OPENAI_MODEL:-gpt-4-turbo}
# Azure
- AZURE_API_KEY=${AZURE_API_KEY:-}
- BRAVE_SEARCH_API_KEY=${BRAVE_SEARCH_API_KEY:-}
- TAVILY_SEARCH_API_KEY=${TAVILY_SEARCH_API_KEY:-}
# RHAIIS
- RHAIIS_URL=${RHAIIS_URL:-}
- RHAIIS_API_KEY=${RHAIIS_API_KEY:-}
- RHAIIS_MODEL=${RHAIIS_MODEL:-}
# RHEL AI
- RHEL_AI_URL=${RHEL_AI_URL:-}
- RHEL_AI_PORT=${RHEL_AI_PORT:-}
- RHEL_AI_API_KEY=${RHEL_AI_API_KEY:-}
- RHEL_AI_MODEL=${RHEL_AI_MODEL:-}
# VertexAI
- GOOGLE_APPLICATION_CREDENTIALS=${GOOGLE_APPLICATION_CREDENTIALS:-}
- VERTEX_AI_PROJECT=${VERTEX_AI_PROJECT:-}
- VERTEX_AI_LOCATION=${VERTEX_AI_LOCATION:-}
# Enable debug logging if needed
- LLAMA_STACK_LOGGING=${LLAMA_STACK_LOGGING:-}
healthcheck:
Expand Down
14 changes: 11 additions & 3 deletions docker-compose.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,22 +10,30 @@ services:
- "8321:8321" # Expose llama-stack on 8321 (adjust if needed)
volumes:
- ./run.yaml:/opt/app-root/run.yaml:Z
- ${GCP_KEYS_PATH:-./tmp/.gcp-keys-dummy}:/opt/app-root/.gcp-keys:ro
environment:
- BRAVE_SEARCH_API_KEY=${BRAVE_SEARCH_API_KEY:-}
- TAVILY_SEARCH_API_KEY=${TAVILY_SEARCH_API_KEY:-}
# OpenAI
- OPENAI_API_KEY=${OPENAI_API_KEY}
- E2E_OPENAI_MODEL=${E2E_OPENAI_MODEL}
# Azure
- AZURE_API_KEY=${AZURE_API_KEY}
- BRAVE_SEARCH_API_KEY=${BRAVE_SEARCH_API_KEY:-}
- TAVILY_SEARCH_API_KEY=${TAVILY_SEARCH_API_KEY:-}
# RHAIIS
- RHAIIS_URL=${RHAIIS_URL}
- RHAIIS_API_KEY=${RHAIIS_API_KEY}
- RHAIIS_MODEL=${RHAIIS_MODEL}
# RHEL AI
- RHEL_AI_URL=${RHEL_AI_URL}
- RHEL_AI_PORT=${RHEL_AI_PORT}
- RHEL_AI_API_KEY=${RHEL_AI_API_KEY}
- RHEL_AI_MODEL=${RHEL_AI_MODEL}
# VertexAI
- GOOGLE_APPLICATION_CREDENTIALS=${GOOGLE_APPLICATION_CREDENTIALS:-}
- VERTEX_AI_PROJECT=${VERTEX_AI_PROJECT:-}
- VERTEX_AI_LOCATION=${VERTEX_AI_LOCATION:-}
# Enable debug logging if needed
- LLAMA_STACK_LOGGING=${LLAMA_STACK_LOGGING:-}

networks:
- lightspeednet
healthcheck:
Expand Down
2 changes: 1 addition & 1 deletion docs/providers.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@ The tables below summarize each provider category, containing the following atri
| sambanova | remote | `litellm` | ❌ |
| tgi | remote | `huggingface_hub`, `aiohttp` | ❌ |
| together | remote | `together` | ❌ |
| vertexai | remote | `litellm`, `google-cloud-aiplatform` | ❌ |
| vertexai | remote | `google-auth` | ✅ |
| watsonx | remote | `ibm_watsonx_ai` | ❌ |

Red Hat providers:
Expand Down
Loading
Loading