Implement validation for early stopping (#1709)

* implement validation for early stopping * fix some documents * fix error messages * implement gRPC API to verify parameters for early stopping * review: use early_stopping as gRPC API Co-authored-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> * review: fix error description Co-authored-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> * review: remove t.Run * review: remove condition to verify algorithmName for early stopping * remove description about updating gRPC API docs in kubeflow website Co-authored-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>
kubeflow · Nov 26, 2021 · 10051dc · 10051dc
1 parent 46207a3
commit 10051dc
Show file tree

Hide file tree

Showing 22 changed files with 958 additions and 180 deletions.
diff --git a/README.md b/README.md
@@ -159,10 +159,10 @@ kubectl apply -k "github.com/kubeflow/katib.git/manifests/v1beta1/installs/katib
 
 ## Release Version
 
-For the specific Katib release (for example `v0.11.1`) run this command:
+For the specific Katib release (for example `v0.12.0`) run this command:
 
 ```
-kubectl apply -k "github.com/kubeflow/katib.git/manifests/v1beta1/installs/katib-standalone?ref=v0.11.1"
+kubectl apply -k "github.com/kubeflow/katib.git/manifests/v1beta1/installs/katib-standalone?ref=v0.12.0"
 ```
 
 Make sure that all Katib components are running:

diff --git a/docs/developer-guide.md b/docs/developer-guide.md
@@ -122,10 +122,6 @@ You can find the `cert-generator` source code [here](../cmd/cert-generator/v1bet
 
 Please see [new-algorithm-service.md](./new-algorithm-service.md).
 
-## Algorithm settings documentation
-
-Please see [algorithm-settings.md](./algorithm-settings.md).
-
 ## Katib UI documentation
 
 Please see [Katib UI README](https://github.com/kubeflow/katib/tree/master/pkg/ui/v1beta1).

diff --git a/pkg/apis/manager/README.md b/pkg/apis/manager/README.md
@@ -22,12 +22,3 @@ generate the API docs from `api.proto`.
 Run `build.sh` to update every file from `api.proto` and generate the docs:
 
 - [v1beta1/build.sh](./v1beta1/build.sh) for v1beta1.
-
-After running `build.sh`, follow these steps to update the docs:
-
-1. Copy the updated content from your generated file
-   `pkg/apis/manager/<version>/gen-doc/api.md` to the doc page in the
-   `kubeflow/website` repository:
-   `kubeflow/website/blob/master/content/docs/reference/katib/<version>/katib.md`.
-1. Create a PR in the `kubeflow/website` repository.
-   (See [example PR](https://github.com/kubeflow/website/pull/1531).)
diff --git a/pkg/apis/manager/v1beta1/api.pb.go b/pkg/apis/manager/v1beta1/api.pb.go
diff --git a/pkg/apis/manager/v1beta1/api.proto b/pkg/apis/manager/v1beta1/api.proto
@@ -42,6 +42,7 @@ service Suggestion {
 service EarlyStopping {
     rpc GetEarlyStoppingRules(GetEarlyStoppingRulesRequest) returns (GetEarlyStoppingRulesReply);
     rpc SetTrialStatus(SetTrialStatusRequest) returns (SetTrialStatusReply);
+    rpc ValidateEarlyStoppingSettings(ValidateEarlyStoppingSettingsRequest) returns (ValidateEarlyStoppingSettingsReply);
 }
 
 /**
@@ -339,6 +340,16 @@ message EarlyStoppingRule {
     int32 start_step = 4; 
 }
 
+message ValidateEarlyStoppingSettingsRequest {
+    EarlyStoppingSpec early_stopping = 1;
+}
+
+/**
+ * Return INVALID_ARGUMENT Error if Early Stopping Settings are not Valid
+ */
+message ValidateEarlyStoppingSettingsReply {
+}
+
 enum ComparisonType {
     UNKNOWN_COMPARISON = 0; // Unknown comparison, not used
     EQUAL = 1; // Equal comparison, e.g. accuracy = 0.7

diff --git a/pkg/apis/manager/v1beta1/gen-doc/api.md b/pkg/apis/manager/v1beta1/gen-doc/api.md
@@ -44,6 +44,8 @@
     - [TrialStatus](#api.v1.beta1.TrialStatus)
     - [ValidateAlgorithmSettingsReply](#api.v1.beta1.ValidateAlgorithmSettingsReply)
     - [ValidateAlgorithmSettingsRequest](#api.v1.beta1.ValidateAlgorithmSettingsRequest)
+    - [ValidateEarlyStoppingSettingsReply](#api.v1.beta1.ValidateEarlyStoppingSettingsReply)
+    - [ValidateEarlyStoppingSettingsRequest](#api.v1.beta1.ValidateEarlyStoppingSettingsRequest)
 
     - [ComparisonType](#api.v1.beta1.ComparisonType)
     - [ObjectiveType](#api.v1.beta1.ObjectiveType)
@@ -696,6 +698,31 @@ Return INVALID_ARGUMENT Error if Algorithm Settings are not Valid
 
 
 
+
+<a name="api.v1.beta1.ValidateEarlyStoppingSettingsReply"></a>
+
+### ValidateEarlyStoppingSettingsReply
+Return INVALID_ARGUMENT Error if Early Stopping Settings are not Valid
+
+
+
+
+
+
+<a name="api.v1.beta1.ValidateEarlyStoppingSettingsRequest"></a>
+
+### ValidateEarlyStoppingSettingsRequest
+
+
+
+| Field | Type | Label | Description |
+| ----- | ---- | ----- | ----------- |
+| early_stopping | [EarlyStoppingSpec](#api.v1.beta1.EarlyStoppingSpec) |  |  |
+
+
+
+
+
 
 
 
@@ -784,6 +811,7 @@ EarlyStopping service defines APIs to manage Katib Early Stopping algorithms
 | ----------- | ------------ | ------------- | ------------|
 | GetEarlyStoppingRules | [GetEarlyStoppingRulesRequest](#api.v1.beta1.GetEarlyStoppingRulesRequest) | [GetEarlyStoppingRulesReply](#api.v1.beta1.GetEarlyStoppingRulesReply) |  |
 | SetTrialStatus | [SetTrialStatusRequest](#api.v1.beta1.SetTrialStatusRequest) | [SetTrialStatusReply](#api.v1.beta1.SetTrialStatusReply) |  |
+| ValidateEarlyStoppingSettings | [ValidateEarlyStoppingSettingsRequest](#api.v1.beta1.ValidateEarlyStoppingSettingsRequest) | [ValidateEarlyStoppingSettingsReply](#api.v1.beta1.ValidateEarlyStoppingSettingsReply) |  |
 
 
 <a name="api.v1.beta1.Suggestion"></a>

diff --git a/pkg/apis/manager/v1beta1/gen-doc/index.html b/pkg/apis/manager/v1beta1/gen-doc/index.html
@@ -338,6 +338,14 @@ <h2>Table of Contents</h2>
                   <a href="#api.v1.beta1.ValidateAlgorithmSettingsRequest"><span class="badge">M</span>ValidateAlgorithmSettingsRequest</a>
                 </li>
 
+                <li>
+                  <a href="#api.v1.beta1.ValidateEarlyStoppingSettingsReply"><span class="badge">M</span>ValidateEarlyStoppingSettingsReply</a>
+                </li>
+
+                <li>
+                  <a href="#api.v1.beta1.ValidateEarlyStoppingSettingsRequest"><span class="badge">M</span>ValidateEarlyStoppingSettingsRequest</a>
+                </li>
+
 
                 <li>
                   <a href="#api.v1.beta1.ComparisonType"><span class="badge">E</span>ComparisonType</a>
@@ -1630,6 +1638,37 @@ <h3 id="api.v1.beta1.ValidateAlgorithmSettingsRequest">ValidateAlgorithmSettings
 
 
 
+        <h3 id="api.v1.beta1.ValidateEarlyStoppingSettingsReply">ValidateEarlyStoppingSettingsReply</h3>
+        <p>Return INVALID_ARGUMENT Error if Early Stopping Settings are not Valid</p>
+
+
+
+
+
+        <h3 id="api.v1.beta1.ValidateEarlyStoppingSettingsRequest">ValidateEarlyStoppingSettingsRequest</h3>
+        <p></p>
+
+
+          <table class="field-table">
+            <thead>
+              <tr><td>Field</td><td>Type</td><td>Label</td><td>Description</td></tr>
+            </thead>
+            <tbody>
+
+                <tr>
+                  <td>early_stopping</td>
+                  <td><a href="#api.v1.beta1.EarlyStoppingSpec">EarlyStoppingSpec</a></td>
+                  <td></td>
+                  <td><p> </p></td>
+                </tr>
+
+            </tbody>
+          </table>
+
+
+
+
+
 
 
         <h3 id="api.v1.beta1.ComparisonType">ComparisonType</h3>
@@ -1852,6 +1891,13 @@ <h3 id="api.v1.beta1.EarlyStopping">EarlyStopping</h3>
                 <td><p></p></td>
               </tr>
 
+              <tr>
+                <td>ValidateEarlyStoppingSettings</td>
+                <td><a href="#api.v1.beta1.ValidateEarlyStoppingSettingsRequest">ValidateEarlyStoppingSettingsRequest</a></td>
+                <td><a href="#api.v1.beta1.ValidateEarlyStoppingSettingsReply">ValidateEarlyStoppingSettingsReply</a></td>
+                <td><p></p></td>
+              </tr>
+
           </tbody>
         </table>