distributed suggestion service

[ddysher]

suggestion service is inherently stateful, we want to make sure a study is processed by a single suggestion service instance. For example, in current setup, if we run 2 replicas of grid service, they will both receive requests, thus essentially suggesting the same parameters twice.

The simplest solution would be to change the service affinity to 'ClientIP' in kubernetes. However, in the long run, we need to have proper handling in our suggestion services themselves as well, to handle potential failure cases.

/cc @YujiOshima @gaocegege

[YujiOshima]

@ddysher Absolutely. #33 is a similar issue.
I want to make suggestions stateless for scalability and robustness.
The solution I can come up with now is save all info of trial to DB and any suggestion instance can get info from DB.
But the DB schema will be complex.

[ddysher]

saving all trials to db sounds like a good first step, but once we can support large number of trials per study, and large number of studies, we might need to do more optimizations

The solution I can come up with now is save all info of trial to DB and any suggestion instance can get info from DB.

[jlewi]

@ddysher @YujiOshima @gaocegege Is there specific work related to this issue for 0.3? If so what is the priority?

[gaocegege]

It is similar to #33 at the beginning, thus we do not need to add it into 0.3

[YujiOshima]

I think we can solve this by solving #33 . But we need a test.
I agree we do not need to add this into 0.3. This is a little advanced.

[gaocegege]

Now we keep one suggestion for one experiment. But we still need #33 .

/close

[k8s-ci-robot]

@gaocegege: Closing this issue.

In response to this:

Now we keep one suggestion for one experiment. But we still need #33 .

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.