KV Store Usage

The Problem

Manticore uses Consul's KV store to determine the desired state of all requests, similar to how job files in Nomad are a desired state of tasks and task groups. The reason for this separation is because Manticore cannot simply run docker containers as soon as a request comes in; it needs to worry about memory and size constraints. At some point, there will be not enough resources in the cluster of machines for this system, and then issues on which users should get which resources will arise. To bring the situation to order when there is no more space, a waiting list is implemented.

The Request List and the Waiting List

There are only two locations in which Manticore watches for data changes: the request list and the waiting list. Manticore uses the request list to store incoming user requests to start jobs, while the waiting list is used for keeping track of the order in which user requests should be handled, whether they have a job assigned to them, and other job-specific data. When the request list is updated, Manticore updates the waiting list accordingly; the request list is always the source of truth for which users should be handled. In other words, the request list behaves more like a desired state of Manticore, and the waiting list behaves more like the current state of Manticore. The KV values of the request and waiting list are just stringified JSON.

Synchronizing the states

The listener modules are responsible for keeping the two sets of data in sync, and for managing messy cases for when a user request is in one list but not in another. Note that by separating the two lists we know exactly when new users come in and when existing users drop out of the request list by looking at the snapshots of the two states, and can take appropriate action for when that happens.

Updating the Waiting List

The waiting list will surely update itself indefinitely because a change will cause the same watch to update the waiting list state. The waiting watcher needs this ability because the waiting list must continue to change as new jobs are running or failing for user requests despite there being no change in the request list. There could suddenly be 5 users in the waiting list and multiple updates to the store need to happen as these users become handled one at a time. In order to prevent the infinite update problem, the waiting list delegates a task to the job interface module and the job interface will inform the waiting watcher whether to update the list in the store. An update should only happen if a change in any one of the user’s jobs is found. If resources are starved and a job cannot be run then that is an indication to not take any action and wait for future updates. If a job fails or stops for a user or if a job stage is successful for a user, then that is an indication to update the waiting list.

Anatomy of the Waiting List

The waiting list is actually an object containing keys, where each key is the id of the request. The queue property for a user id indicates the position of the user in waiting list. There is no guarantee that the queue numbers will be sequential, only that the numbers are unique across all users to determine order, where lower queue numbers get handled first. The state property can either be in its starting state, waiting, in its final state, claimed, or in any possible state between the two. The state is in an intermediary value (ex. pending-1) if a request is being handled, but the resources have not been confirmed operational yet. Once it is confirmed that all resources are running and healthy then the state of that user is set to claimed. Technically, all users are in the waiting list, but users are actually waiting only if their state property is waiting. Once someone stops using their resources, their id will be removed from the request list, and consequentially the waiting list, leaving room for more users to claim their resources.

{  
   "14":{  
      "id":"14",
      "queue":1,
      "state":"claimed",
      "request":{...},
      "services":{...}
   }
   "15":{  
      "id":"15",
      "queue":2,
      "state":"waiting",
      "request":{...},
      "services":{...}
   }
}

The request object is essentially what the user submitted to the POST /job route for future reference, and the services object contains actual addresses of running resources, which will be transmitted to that user over WebSockets when ready.

HAProxy Address Data

Manticore needs to transform the data from the waiting list into something that consul-template can parse easily. The haproxy listener module's purpose is to do just that. Once all the information is digested the output of that is stored in the KV store again, but in a different location that is templateData. The information there is strictly for routing users to the locations of their resources. Note that the template HAProxy file can directly reference Consul for where the Manticore servers are, so there is no need to take additional steps in storing the server's address information in the KV store.

HAProxy Miscellaneous Data

The domain name and main port that HAProxy will listen on needs to be defined, and that is determined by environment variables passed in. This information is stored in /haproxy/domainName and /haproxy/mainPort.