Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add metrics collector spec and objective spec to Trial #489

Merged
merged 8 commits into from
May 9, 2019
Merged

Add metrics collector spec and objective spec to Trial #489

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
23deb5d
Add metrics collector spec to Trial spec
richardsliu May 8, 2019
5d4e704
Merge branch 'master' into trial_mc
richardsliu May 8, 2019
6f557fe
Fix e2e test
richardsliu May 8, 2019
105044c
Merge branch 'master' into trial_mc
richardsliu May 8, 2019
55a1faa
Move ObjectiveSpec definition to Trial CRD
richardsliu May 8, 2019
ed22e55
Move common types
richardsliu May 9, 2019
8ec484d
Move common types to its own package
richardsliu May 9, 2019
96494a9
Add metrics collector spec to DB
richardsliu May 9, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions pkg/api/operators/apis/trial/v1alpha2/trial_types.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,9 @@ type TrialSpec struct {
// and let the corresponding resource controller (e.g. tf-operator) handle
// the rest.
RunSpec string `json:"runSpec,omitempty"`

// Raw text for the metrics collector spec. This must be a CronJob object.
MetricsCollectorSpec string `json:"metricsCollectorSpec,omitempty"`
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add Objective name and type to spec to figure out the best values?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added ObjectiveSpec to Trial.

}

type TrialStatus struct {
Expand Down
315 changes: 162 additions & 153 deletions pkg/api/v1alpha2/api.pb.go
View file
Open in desktop

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions pkg/api/v1alpha2/api.proto
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -357,6 +357,7 @@ message TrialSpec {
string experiment_name = 1;
ParameterAssignments parameter_assignments = 2;
string run_spec = 3;
string metrics_collector_spec = 4;
}

message TrialStatus {
Expand Down
3 changes: 3 additions & 0 deletions pkg/api/v1alpha2/api.swagger.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -939,6 +939,9 @@
},
"run_spec": {
"type": "string"
},
"metrics_collector_spec": {
"type": "string"
}
}
},
Expand Down
1 change: 1 addition & 0 deletions pkg/api/v1alpha2/gen-doc/api.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -792,6 +792,7 @@ Katib will create each Hyper parameter from this config.
| experiment_name | [string](#string) | | |
| parameter_assignments | [TrialSpec.ParameterAssignments](#api.v1.alpha2.TrialSpec.ParameterAssignments) | | |
| run_spec | [string](#string) | | |
| metrics_collector_spec | [string](#string) | | |



Expand Down
7 changes: 7 additions & 0 deletions pkg/api/v1alpha2/gen-doc/index.html
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1727,6 +1727,13 @@ <h3 id="api.v1.alpha2.TrialSpec">TrialSpec</h3>
<td><p> </p></td>
</tr>

<tr>
<td>metrics_collector_spec</td>
<td><a href="#string">string</a></td>
<td></td>
<td><p> </p></td>
</tr>

</tbody>
</table>

Expand Down
179 changes: 93 additions & 86 deletions pkg/api/v1alpha2/python/api_pb2.py
View file
Open in desktop

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion pkg/db/v1alpha2/db_init.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,6 @@ func (d *dbConn) DBInit() {
parallel_trial_count INT,
max_trial_count INT,
status TINYINT,
metrics_collector_type TEXT,
gaocegege marked this conversation as resolved.
Show resolved Hide resolved
start_time DATETIME(6),
completion_time DATETIME(6),
nas_config TEXT)`)
Expand All @@ -33,6 +32,7 @@ func (d *dbConn) DBInit() {
experiment_name VARCHAR(255) NOT NULL,
parameter_assignments TEXT,
run_spec TEXT,
metrics_collector_spec TEXT,
observation TEXT,
status TINYINT,
start_time DATETIME(6),
Expand Down
6 changes: 5 additions & 1 deletion pkg/db/v1alpha2/interface.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -456,14 +456,16 @@ func (d *dbConn) RegisterTrial(trial *v1alpha2.Trial) error {
experiment_name,
parameter_assignments,
run_spec,
metrics_collector_spec,
observation,
status,
start_time,
completion_time) VALUES (?, ?, ?, ?, ?, ?, ?, ?)`,
completion_time) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)`,
trial.Name,
trial.Spec.ExperimentName,
paramAssignment,
trial.Spec.RunSpec,
trial.Spec.MetricsCollectorSpec,
observation,
trial.Status.Condition,
start_time,
Expand Down Expand Up @@ -507,6 +509,7 @@ func (d *dbConn) GetTrialList(experimentName string, filter string) ([]*v1alpha2
&trial.Spec.ExperimentName,
&paramAssignment,
&trial.Spec.RunSpec,
&trial.Spec.MetricsCollectorSpec,
&observation,
&trial.Status.Condition,
&start_time,
Expand Down Expand Up @@ -565,6 +568,7 @@ func (d *dbConn) GetTrial(trialName string) (*v1alpha2.Trial, error) {
&trial.Spec.ExperimentName,
&paramAssignment,
&trial.Spec.RunSpec,
&trial.Spec.MetricsCollectorSpec,
&observation,
&trial.Status.Condition,
&start_time,
Expand Down
6 changes: 6 additions & 0 deletions pkg/db/v1alpha2/interface_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,7 @@ var trialColumns = []string{
"experiment_name",
"parameter_assignments",
"run_spec",
"metrics_collector_spec",
"observation",
"status",
"start_time",
Expand Down Expand Up @@ -299,6 +300,7 @@ func TestRegisterTrial(t *testing.T) {
experiment_name,
parameter_assignments,
run_spec,
metrics_collector_spec,
observation,
status,
start_time,
Expand All @@ -308,6 +310,7 @@ func TestRegisterTrial(t *testing.T) {
"test1",
"{\"assignments\":[{\"name\":\"param1\",\"value\":\"0.9\"},{\"name\":\"param2\",\"value\":\"10\"}]}",
"",
"",
"{\"metrics\":[{\"name\":\"f1_score\",\"value\":\"88.95\"},{\"name\":\"loss\",\"value\":\"0.5\"},{\"name\":\"precision\",\"value\":\"88.7\"},{\"name\":\"recall\",\"value\":\"89.2\"}]}",
trial.Status.Condition,
"2016-12-31 20:02:05.123456",
Expand All @@ -328,6 +331,7 @@ func TestGetTrialList(t *testing.T) {
"test1",
"{\"assignments\":[{\"name\":\"param1\",\"value\":\"0.9\"},{\"name\":\"param2\",\"value\":\"10\"}]}",
"",
"",
"{\"metrics\":[{\"name\":\"f1_score\",\"value\":\"88.95\"},{\"name\":\"loss\",\"value\":\"0.5\"},{\"name\":\"precision\",\"value\":\"88.7\"},{\"name\":\"recall\",\"value\":\"89.2\"}]}",
api_pb.TrialStatus_RUNNING,
"2016-12-31 20:02:05.123456",
Expand All @@ -338,6 +342,7 @@ func TestGetTrialList(t *testing.T) {
"test1",
"{\"assignments\":[{\"name\":\"param1\",\"value\":\"0.8\"},{\"name\":\"param2\",\"value\":\"20\"}]}",
"",
"",
"{\"metrics\":[{\"name\":\"f1_score\",\"value\":\"88.5\"},{\"name\":\"loss\",\"value\":\"0.8\"},{\"name\":\"precision\",\"value\":\"88.2\"},{\"name\":\"recall\",\"value\":\"89.0\"}]}",
api_pb.TrialStatus_COMPLETED,
"2016-12-31 20:02:05.123456",
Expand All @@ -362,6 +367,7 @@ func TestGetTrial(t *testing.T) {
"test1",
"{\"assignments\":[{\"name\":\"param1\",\"value\":\"0.9\"},{\"name\":\"param2\",\"value\":\"10\"}]}",
"",
"",
"{\"metrics\":[{\"name\":\"f1_score\",\"value\":\"88.95\"},{\"name\":\"loss\",\"value\":\"0.5\"},{\"name\":\"precision\",\"value\":\"88.7\"},{\"name\":\"recall\",\"value\":\"89.2\"}]}",
api_pb.TrialStatus_RUNNING,
"2016-12-31 20:02:05.123456",
Expand Down
4 changes: 2 additions & 2 deletions test/e2e/v1alpha2/test-katib-manager.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,8 +16,7 @@ def register_experiment(stub):
algorithm=algo,
trial_template="run-mnist",
parallel_trial_count=2,
max_trial_count=9,
metrics_collector_type='2')
max_trial_count=9)
exp_status = api_pb2.ExperimentStatus(condition=1,
start_time="2019-04-28T14:09:15Z",
completion_time="2019-04-28T16:09:15Z")
Expand Down Expand Up @@ -64,6 +63,7 @@ def register_trial(stub):
parameters = api_pb2.TrialSpec.ParameterAssignments(assignments=[api_pb2.ParameterAssignment(name="rl", value="0.01")])
spec = api_pb2.TrialSpec(experiment_name=TEST_EXPERIMENT,
run_spec="a batch/job resource",
metrics_collector_spec="metrics/collector",
parameter_assignments=parameters)
observation = api_pb2.Observation(metrics=[api_pb2.Metric(name="loss", value="0.54")])
status = api_pb2.TrialStatus(condition=2,
Expand Down