add GRPC/rest API to return best suggestion trial result

[hougangliu]

After a studyJob completed, we need expose API to get best suggestion trial result so that 3rd-party can use the result to train model.

// return GetTrialsReply because maybe multiple trials have same object metric value
 rpc GetBestTrials(GetStudyRequest) returns (GetTrialsReply){
        option (google.api.http) = {
            get: "/api/Manager/GetBestTrials/{study_id}"
        };
    };

[hougangliu]

@YujiOshima @richardsliu Any comment?

[hougangliu]

/assign

[johnugeorge]

Isn't this a very resource intensive process? we need get all metric logs of all workers to compute it. Is there a better way?

[hougangliu]

@johnugeorge In fact, all trial info had been stored in DB. It just need return the best trials by querying DB. No need to get all metric logs of all workers to compute it again when run this API.

[johnugeorge]

Yes. I was asking if this db query would span across all metric logs of all workers of this study. Currently, are we storing the best values of each trial in DB now?

Please correct me if I am wrong.

[hougangliu]

only get trial in DB once. we can get the best trail ID from studyJob spec (the trail's worker counterpart objectiveValue equal to the studyJob's best objectiveValue)

[hougangliu]

We can add bestTrialId in studyjob.status and user can get the trial detail by calling GetTrial gRPC api

[YujiOshima]

An API to get the best N trials and workers is so useful. But as @johnugeorge said, scanning all metrics logs is so heavy.
One solution is to add max and min value of objective value column to DB and update at every reporting metrics.
@hougangliu @johnugeorge WDYT?

[johnugeorge]

Sorry for late response because of holidays.
I agree with @YujiOshima . We can reduce unnecessary compute by adding best(max/min) value to DB and updating it during every report.

While digging deep, I saw one assumption in the current code.
https://github.com/kubeflow/katib/blob/master/pkg/controller/studyjob/studyjob_controller.go#L251
Is this assumption correct? Can we assume that last timestamp value is always the best value if we do not order by the objective value? The objective values might not always improve in every iteration.

Adding the max and min value of objective value in DB also solves this problem.