Skip to content
This repository has been archived by the owner on Aug 24, 2020. It is now read-only.

Use service discovery in scheduler to discover where log-cache nodes are #52

Open
jasonkeene opened this issue Apr 30, 2018 · 5 comments

Comments

@jasonkeene
Copy link
Contributor

This would allow for log-cache instances to come and go and the cluster to adapt dynamically to scaling events, outages, etc.

@cf-gitbot
Copy link

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/157208717

The labels on this github issue will be updated when the story is started.

@poy
Copy link
Contributor

poy commented May 16, 2018

@jasonkeene How do other systems manage this? Some form of a gossip protocol?

@jasonkeene
Copy link
Contributor Author

@apoydence I was thinking of doing something similar to what the loggregator agent does.

  1. Accept a DNS name in config.
  2. At runtime query DNS to get A or AAAA records for that name.
  3. Use these IP addresses.
  4. Periodically query DNS to refresh the list of IP addresses.

This would be compatible with a LOT of service discovery systems, including kube-dns and bosh-dns.

Alternatively, you can query for SRV records which allows for port, weighting, transport and service name to be discovered. kube-dns supports this but I am not sure about bosh-dns.

@poy
Copy link
Contributor

poy commented May 22, 2018

@jasonkeene This sounds reasonable.

I would want to do more experiments with multiple schedulers after this is completed to see if there is any thrashing. I suspect each scheduler would be looking at a different subset of log cache nodes and would be instructing them differently.

@jasonkeene
Copy link
Contributor Author

That is a possibility. We can have an algo that is resistant to sudden changes to prevent nodes dropping out when they come right back. Something like a TTL for a node to stop being scheduled to. This would help with thrashing, allow for nodes that are truely gone to expire, and allow for new nodes to come online.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

3 participants