Proposal: Add zero-configuration networking support via zeroconf

[ericrallen]

This proposal allows the Ollama service to be made discoverable via zero configuration networking across the user's local network via Bonjour/Zeroconf/Avahi aka Multicast DNS (mDNS) using the zeroconf Go libraryso that other clients can connect to and use it without needing to know the host's IP address.

This opens up many different applications for consuming Ollama models served by other network devices.

My particular use case is to add support for Network Discoverable Ollama models to an Obsidian Plugin that I maintain so that users won't have to configure IP addresses in Obsidian or update them if/when IP addresses on their local network change (and also won't have to get into configuring a static IP for the device that is serving their local models).

Note: Network discovery is entirely opt-in via the OLLAMA_DISCOVERY environment variable flag being set to ENABLED and will automatically update the OLLAMA_HOST to 0.0.0.0 (the demo GIF was recorded with an earlier iteration of this PR that also required the user to manually set the host IP).

Demo

Note: To test this functionality, I created a simple Node.js script on another machine on my network and had it use the bonjour package to search for a service with the name OllamaProvider. It gets the IP address and port associated with that service and then makes requests to it. I only showed the IP address at the beginning of the GIF to emphasize that the requests are coming from a different machine.

It also adds a menu entry with the service name if Network Discovery has been enabled.

Instructions for Testing

1. Checkout this PR: gh pr checkout https://github.com/jmorganca/ollama/pull/751
2. Generate: go generate ./...
3. Build: go build .
4. Build and run the App:
   1. cd ./app
   2. npm install
   3. OLLAMA_DISCOVERY=ENABLED npm start
5. Search for and connect to your Network Service (I've provided an example discovery script below)

Network Discovery Script

// you'll need to `npm install bonjour` first
// it's recommended to put this in a subdirectory
// and run `npm init` before installing packages
const bonjour = require("bonjour")();

// this demo script can be run via node to find
// and connect to a Network Service with the name
//defined in OLLAMA_SERVICE_NAME
const OLLAMA_SERVICE_NAME = 'OllamaProvider'

// iterate through services
bonjour.find({}, (service) => {
  if (service.name === OLLAMA_SERVICE_NAME) {
    const address = service.addresses[0];
    const port = service.port;

    const baseUrl = new URL(address);

    baseUrl.port = port;

    const modelsUrl = new URL("/api/tags", baseUrl);

    // get available models
    fetch(modelsUrl)
      .then(async (response) => await response.json())
      .then((response) => {
        console.log(response);
      })
      .catch((error) => {
        console.error(error);
      })
  }
});

[ericrallen]

This is my first time working with Golang, so I'm honestly not sure if the changes to this file are what is expected.

Please let me know if I've unexpectedly broken anything.

[ericrallen]

This is my first time working with Golang, so I'm honestly not sure if the changes to this file are what is expected.

Please let me know if I've unexpectedly broken anything.

[ericrallen]

I wasn't sure if there was a good way to share constant strings between the Golang code in cmd.go and the TypeScript code in index.ts, so I just hardcoded the String "OllamaProvider" in both places.

We could also probably search for the service ourselves here, but I didn't want to introduce too much complexity for such a small feature that is just displaying some info to the user.

[ericrallen]

For Network Discovery to work the service needs to be running on the 0.0.0.0 instead of just the 127.0.0.1 loopback.

[ericrallen]

It's also possible that it might be an even better to approach this purely from the Electron app side and use bonjour to add an interactive menu item that toggles between enabling (restart the ollama server with the host set to 0.0.0.0 and then broadcasts the Network Service for discovery) and disabling (restart the ollama server with the host set to OLLAMA_HOST or the default and shut down the Network Service broadcast).

[technovangelist]

Why are you setting the ip at all?

[ericrallen]

Why are you setting the ip at all?

Hey there, @technovangelist!

With the default host (127.0.0.1), the ollama server is only running on the loopback address, and other devices on the network can't make requests to it.

With 0.0.0.0, it listens on all interfaces and allows other devices to send requests.

If I want to run Ollama on a device that's powerful enough to run these models but would like to consume them on a device that isn't, I have to run Ollama on 0.0.0.0 and also know the IP address of the machine that is serving them.

mDNS removes the need to have pre-existing knowledge of the Ollama host machine's IP address, and allows a client running on the low-powered device to find and connect to the service without needing to know its IP address beforehand.

[technovangelist]

Ahh. I was assuming this was for running the server on the same host and you were hitting cors issues implying you were using fetch rather than obsidians own http request methods which solve those problems.

[ericrallen]

The end goal will be to first look for localhost:11434 (or a user-specified port if they want to override the default) and check for the /api/tags endpoint and if it isn't available, then look for an OllamaProvider network service and grab the available models from /api/tags.

One thing I'm wondering now is if the Ollama app will automatically pick up the OLLAMA_DISCOVERY env variable when it starts up or if the idea of having this toggleable in the menu bar instead is the better approach.

[jmorganca]

Hi @ericrallen, this is incredibly cool, and thank you for making a PR. I'm sorry I haven't had the chance to look at this sooner.

Networking and service discovery isn't something Ollama is designed to handle right for the time being. Instead, it's designed to be paired with existing tools like Tailscale or similar, so it can stay focused on providing a reliable http service. Sorry about that!