Operationalising Foundry Local at the Disconnected Edge: One-Month Prototype Lessons

[Liam-dev]

How we stood up a mission‑critical prototype of Foundry Local in a fully disconnected environment, the architectural decisions we made, what worked, what hurt, and what we’re asking the community (and platform team) for—especially around embeddings and RAG.

Pace Applied Solutions – Edge AI Team

1. Why We Did This

We’re prototyping a high‑value, mission‑critical Edge AI capability that must:

• Run fully disconnected (no internet, bios disabled, no package install during runtime)
• Support rapid multi‑modal or multi-agent workflows
• Provide:
  • Basic chat & custom agent building/use
  • Lightweight RAG (retrieval‑augmented generation) over locally held documents
  • Extensible hooks for future task automations

Foundry Local emerged as the core inference layer after multiple design passes across alternative stacks.

2. Environment Constraints (The Real Battle)

Operating conditions:

• Zero connectivity: no pip installs, no dynamic dependency resolution
• Hardware parity required between staging (connected) and deployment (air‑gapped)
• Transfer pathways limited to:
  • Docker image handoffs
  • Manual file copy (USB / offline disk sync)
  • Environment cloning (filesystem replication)

Key principle that emerged:

The fewer “moving parts” (wheels, libs, orchestrators), the higher the survivability of the deployment.

3. Final Architecture (Current Prototype)

[ Staging (Connected) ]
  - Foundry Local Offline Installer
  - Model acquisition on connected PC and manual (USB) transfer to disconnected cache directory - throws some errors around no model list retrieval - but works.
  - (Attempted) external embeddings + BYO models - 
       |
       v (File transfer / snapshot)
[ Edge (Disconnected) ]
  - Foundry Local runtime
  - Local model cache (manually populated)
  - Lightweight HTML/JS wrapper (no heavy frontend framework)
  - PowerShell bootstrap scripts:
       * Launch Foundry service
       * Detect ephemeral port
       * Inject resolved port into web app config
  - Agent config + system prompt templates
  - (Experimental) RAG pipeline (currently suboptimal)

4. Foundry Local Deployment Workflow (Offline Pattern)

1. Perform installation and model acquisition on a connected twin machine.
2. Transfer the offline installer bundle (validated against hardware parity).
3. Copy model cache directories directly into the offline host’s cache path.
4. Launch Foundry Local in the disconnected environment.
5. Auto-detect ephemeral port (since static binding not exposed) → inject into UI wrapper.
6. Run local agent & chat flows.

This has been reliable for:

• Core chat models
• Agent creation / system instruction flows
• Basic multi-session concurrency

5. Lightweight UI Wrapper Strategy

Why not heavier frameworks? Faster to bootstrap + fewer build dependencies.

What it does:

• Enumerates available models
• Supports multiple structured chat sessions with per-session system instructions
• Auto-updates Foundry endpoint at runtime based on startup script discovery
• Minimal front-end state (no DB dependency; config stored as JSON)

Port Resolution Logic (simplified sketch):

# discover-foundry-port.ps1
$process = Get-NetTCPConnection | Where-Object { $_.State -eq "Listen" -and $_.OwningProcess -in (Get-Process -Name "foundry*" | Select-Object -Expand Id) }
$port = $process.LocalPort
(Get-Content .\webapp\config.template.json) -replace "{{FOUNDry_PORT}}", $port | Set-Content .\webapp\config.json

6. What Worked Well

Area	Outcome	Notes
Local model execution	Stable	Once cache copied, cold start acceptable
Agent system prompts	Flexible	Enables scenario-driven specialisation
Port auto-resolution	Eliminated manual edits	PowerShell approach sufficient
Deployment repeatability	High	Rsync / file block transfer approach works
Minimal dependency surface	Crucial	Reduced failure points in air‑gapped ops

7. The Hard Stuff

Challenge	Impact	Mitigation Attempt	Status
Ephemeral port allocation	Required dynamic discovery	Boot script port injection	Solved (workaround)
BYO embeddings (HF → ONNX via Olive)	Not recognised by Foundry service	Placed converted models into expected folder	Unresolved
RAG pipeline	Sluggish & heavy when layering torch + custom indexers	Considered pruning / ONNX-only	Currently non-ideal
Model ingestion verification	Limited introspection in disconnected mode	Manual file inspection	Partial confidence only
Deployment friction (no package manager)	Slows experimentation	Pre-bake all artefacts	Ongoing discipline required

8. RAG & Embeddings: The Major Gap

We have functional chat + agents, but RAG is where friction spikes – and this is critical at user need at edge:

• External embeddings path (HF models converted via Olive → ONNX) not registered by the Foundry service.
• Even when structuring directories to mirror existing model cache patterns, embeddings do not appear in model listings.
• Without a natively supported embeddings model in the standard catalogue, the RAG stack becomes heavier (extra runtimes + dependency bloat).

What would dramatically improve disconnected adoption:

1. A first-class embeddings model in the Foundry Local standard catalogue (even a single robust general-purpose one).
2. Clear offline ingestion spec for registering third-party embeddings:
   • Expected folder structure
   • Naming / manifest format
   • Hash / signature requirements
   • Any metadata index rebuild step
3. Ability to statically assign port(s) for Foundry services (simplifies orchestration scripts).
4. Lightweight native RAG pipeline template:
   • Embedding
   • Chunking
   • Vector store interface (pluggable backends)
   • Retrieval + response synthesis

If any of this is already possible today (especially #2), we would love a pointer.

9. Value Delivered (Already)

Even in month one:

• Accelerated prototype for mission scenarios
• Unified inference layer replacing 2–3 earlier fragmented stacks
• Reduced operational footprint for edge kit
• Agent pattern accelerating specialised task reuse
• Clearer path to scale once RAG is streamlined

10. Quick Start Tips (For Others Attempting Similar)

1. Do all experimentation connected; only “promote” known-good model/cache states.
2. Keep a manifest (hashes + sizes) of transferred model directories for integrity checks.
3. Script everything that’s repeatable (startup, port detection, model validation).
4. Avoid introducing Python virtualenv overhead on the disconnected side unless absolutely necessary.
5. Start with agent/system prompt ergonomics early—it shapes UX expectations.
6. Defer RAG until embeddings path is stable (don’t prematurely optimise a brittle pipeline).

11. Open Questions (Community / Team Ask)

We’d appreciate guidance or confirmation on:

• Is there an officially supported offline embeddings model today?
• Is there a documented process for Foundry Local to “register” externally added ONNX embedding models?
• Any roadmap hints for:
  • Simplified offline model ingestion / manifests
  • Built-in RAG scaffolding
  • BYOM embeddings support

12. What We Can Share Back

Happy to share any learnings or insights if useful.

13. Next Steps (Our Side)

Work with users on the current prototype and keep experimenting with other options, eagerly look for new releases and help test out live!

14. Final Note

Foundry Local has been genuinely impressive as a core inference backbone in constrained environments. Even with the friction points, it’s already accelerated our trajectory. Looking forward to future iterations—this is a strong foundation.

Huge thanks to the team—please keep the offline + embeddings + RAG pathway in sharp focus; it’s a force multiplier for disconnected operations.

If you have tips, corrections, or can unblock the embeddings question—reach out. We’re only a month in and eager to refine.