Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add Field Status metadata to Online Serving #658

Merged
merged 65 commits into from
Jun 19, 2020
Merged

Add Field Status metadata to Online Serving #658

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
65 commits
Select commit Hold shift + click to select a range
377cb53
Update ServingService protobuf with additions to support online metad…
Apr 17, 2020
9170e22
Change error thrown by sendMultiGet() to Unknown as it describes the …
Apr 17, 2020
2db28c1
Return boolean primitive instead of Boolean object type in FeatureRow…
Apr 18, 2020
2b13bb5
Config sendAndProcessMultiGet() to throw an exception on feature row …
Apr 19, 2020
4ba034b
Refactor sendMultiGetAndProcess() to getFeaturesForFeatureSet().
Apr 19, 2020
607b681
Update ResponseJSONMapper to use new GetOnlineFeatures protobuf
Apr 20, 2020
2f4c710
Remove references to FieldValues in OnlineServingService & Tests
Apr 20, 2020
59692f2
Update java sdk's FieldClient to use the new GetOnlineFeatures protobuf
Apr 20, 2020
7e89555
Update OnlineRetriever to return null on missing retrieval (such as m…
Apr 20, 2020
38815bb
Refactor out unnecessary nesting of returned list in OnlineRetriever'…
Apr 21, 2020
437da45
Add online feature metadata to OnlineServingService's getOnlineFeatur…
Apr 23, 2020
4a78189
Update OnlineServingServiceTest to check that online metadata is set …
Apr 23, 2020
4d6437e
Added online metadata support java sdk's FeastClient.
Apr 23, 2020
9ca034c
Changed go sdk's Row to store a map of fields instead of just Values
Apr 24, 2020
f91c3fe
Update response.go to use updated Row and take into account FieldStatus
Apr 24, 2020
4df2d34
Added Field() to go sdk's types.go to construct fields from a Value.
Apr 24, 2020
e896f17
Update go sdk's request.go to use updated Row and Add GetOnlineFeatur…
Apr 24, 2020
5b23b55
Update go protobuf generated code for updated protobuf defintion
Apr 24, 2020
1ef7f4d
Update python sdk's client to document new fieldstatus metadata, adde…
Apr 24, 2020
240dc11
Fixed issue where getOnlineFeatures() returned Record in a non determ…
Apr 24, 2020
fea6f4a
Update e2e ingestion tests to support new get_online_features api and…
Apr 27, 2020
6c4968f
Fixed NullPointerException when feature row is null.
Apr 27, 2020
a7e97d6
Fixed issue in which unpackFields(entityRow) was not responding to in…
Apr 27, 2020
55e1ce4
Check FieldStatus in end to end tests to wait for ingestion of values…
Apr 27, 2020
d90cf5a
Moved test checks out of polling loop in e2e online ingestion tests
Apr 27, 2020
cd5ea47
Apply spotless java formatting
Apr 27, 2020
3443078
Fixed misplaced break in end to end tests
Apr 27, 2020
3cc3a8e
Fixed typos in e2e ingestion tests.
Apr 27, 2020
3dedf4f
Apply changes on RedisOnlineRetriever to RedisClusterOnlineRetriever
Apr 28, 2020
933e20d
Rename getFeaturesForFeatureSet() to getFeaturesFromRedis() for bette…
Apr 28, 2020
cf99f9c
Update Serving application.yml to include example on how to config Re…
Apr 28, 2020
3cd4e22
Correct missing include_meta flag in e2e tests get_online_features() …
Apr 28, 2020
044b27d
Correct minor comment typo in serving's application.yml
Apr 28, 2020
0a7c1fc
Correct another minor typo in e2e test: missing .status
Apr 28, 2020
2cdf558
Fix Feast core and serving compilation/test failures after rebase
mrzzy Jun 3, 2020
57babd5
Fix go SDK tests after rebase
mrzzy Jun 3, 2020
6e247db
Update ServingService protobuf to include metadata without breaking c…
mrzzy Jun 4, 2020
d012ee7
Revert Feast Serving's ServingService & ResponseJSONMapper to FieldVa…
mrzzy Jun 4, 2020
149a455
Revert serving's OnlineServingService to backwards compatible FieldVa…
mrzzy Jun 5, 2020
253ca28
Revert Java SDK's FeastClient to backwards compatible FieldStatus API.
mrzzy Jun 5, 2020
53ad9a0
Revert Go SDK's client.go to backwards compatible FieldStatus API.
mrzzy Jun 5, 2020
06bcca7
Fix Java SDK unit tests
mrzzy Jun 5, 2020
d9332ad
Added .Statuses() method to GO Sdk's OnlineFeaturesResponse to get me…
mrzzy Jun 5, 2020
5141a1e
Fixed Java SDK not stripping project from field values and statuses
mrzzy Jun 5, 2020
059ea9f
Fixed Go SDK's client.go not stripping project from field values and…
mrzzy Jun 5, 2020
8d5d031
Fixed Go SDK's Unit tests
mrzzy Jun 5, 2020
bf9c11b
Revert Python SDK to backward compatible FieldValues API & fix tests
mrzzy Jun 5, 2020
87249c8
Move hardcoded constants from job.py to constants.py in Python SDK to…
mrzzy Jun 7, 2020
ccd632d
Added wait backoff implementation in Python SDK: wait_retry_backoff()
mrzzy Jun 7, 2020
4580e04
Update Python SDK's job.py to use shared backoff wait implemetation
mrzzy Jun 7, 2020
95f5f6b
Clarify that maximum range in ServingService proto to max age
mrzzy Jun 12, 2020
3bf17f2
Fixed online serving e2e tests
mrzzy Jun 12, 2020
bd54fc7
Fix python lint.
mrzzy Jun 12, 2020
64f59ac
Collect entity refs in Python SDK as a single line.
mrzzy Jun 12, 2020
f1e52fc
Use guard style if statements in Go Sdk's types.go
mrzzy Jun 12, 2020
9958e7e
Make strip field values part of Python SDK's get_online_features() mo…
mrzzy Jun 12, 2020
6de8267
Fix go unit tests
mrzzy Jun 12, 2020
7952d98
Remove include_metadata_in_response flag from ServingService proto as…
mrzzy Jun 16, 2020
32cccaa
Readd missing span log call to log retrieved feature rows.
mrzzy Jun 16, 2020
c05614d
Use more modern Streams.zip() to iterate entity rows and feature rows…
mrzzy Jun 17, 2020
81422cc
Change utility methods in OnlineServingService to static.
mrzzy Jun 17, 2020
8df3e57
Use "outside max age" instead "stale" term in OnlineServingService as…
mrzzy Jun 17, 2020
a03a1a9
Refactor serving's OnlineRetriever to return a empty Optional<Feature…
mrzzy Jun 17, 2020
1e97264
Fixed typo in basic redis e2e tests
mrzzy Jun 17, 2020
d4ef594
Fixed typo OnlineServingService's logFeatureRowsTrace()
mrzzy Jun 17, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 1 addition & 2 deletions go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,8 +25,7 @@ require (
golang.org/x/lint v0.0.0-20200302205851-738671d3881b // indirect
golang.org/x/net v0.0.0-20200513185701-a91f0712d120
golang.org/x/sys v0.0.0-20200515095857-1151b9dac4a9 // indirect
golang.org/x/tools v0.0.0-20200521211927-2b542361a4fc // indirect
google.golang.org/genproto v0.0.0-20200515170657-fc4c6c6a6587 // indirect
golang.org/x/tools v0.0.0-20200604042327-9b20fe4cabe8 // indirect
google.golang.org/grpc v1.29.1
google.golang.org/protobuf v1.24.0 // indirect
gopkg.in/russross/blackfriday.v2 v2.0.0 // indirect
Expand Down
8 changes: 8 additions & 0 deletions go.sum
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -162,6 +162,7 @@ github.com/google/go-cmp v0.3.0 h1:crn/baboCvb5fXaQ0IJ1SGTsTVrWpDsCWC8EGETZijY=
github.com/google/go-cmp v0.3.0/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU=
github.com/google/go-cmp v0.3.1 h1:Xye71clBPdm5HgqGwUkwhbynsUJZhDbS20FvLhQ2izg=
github.com/google/go-cmp v0.3.1/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU=
github.com/google/go-cmp v0.4.0 h1:xsAVV57WRhGj6kEIi8ReJzQlHHqcBYCElAvkovg3B/4=
github.com/google/go-cmp v0.4.0/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/google/gofuzz v0.0.0-20161122191042-44d81051d367/go.mod h1:HP5RmnzzSNb993RKQDq4+1A4ia9nllfqcQFTQJedwGI=
github.com/google/gofuzz v0.0.0-20170612174753-24818f796faf/go.mod h1:HP5RmnzzSNb993RKQDq4+1A4ia9nllfqcQFTQJedwGI=
Expand Down Expand Up @@ -319,6 +320,7 @@ github.com/stretchr/testify v1.2.0/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXf
github.com/stretchr/testify v1.2.2 h1:bSDNvY7ZPG5RlJ8otE/7V6gMiyenm9RtJ7IUVIAoJ1w=
github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs=
github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
github.com/stretchr/testify v1.4.0 h1:2E4SXV/wtOkTonXsotYi4li6zVWxYlZuYNCXe9XRJyk=
github.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4=
github.com/tmc/grpc-websocket-proxy v0.0.0-20170815181823-89b8d40f7ca8/go.mod h1:ncp9v5uamzpCO7NfCPTXjqaC+bZgJeR0sMTm6dMHP7U=
github.com/tmc/grpc-websocket-proxy v0.0.0-20190109142713-0ad062ec5ee5/go.mod h1:ncp9v5uamzpCO7NfCPTXjqaC+bZgJeR0sMTm6dMHP7U=
Expand Down Expand Up @@ -466,8 +468,12 @@ golang.org/x/tools v0.0.0-20200519205726-57a9e4404bf7 h1:nm4zDh9WvH4jiuUpMY5RUsv
golang.org/x/tools v0.0.0-20200519205726-57a9e4404bf7/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE=
golang.org/x/tools v0.0.0-20200521211927-2b542361a4fc h1:6m2YO+AmBApbUOmhsghW+IfRyZOY4My4UYvQQrEpHfY=
golang.org/x/tools v0.0.0-20200521211927-2b542361a4fc/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE=
golang.org/x/tools v0.0.0-20200601175630-2caf76543d99/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE=
golang.org/x/tools v0.0.0-20200604042327-9b20fe4cabe8 h1:8Xr1qwxn90MXYKftwNxIO2g4J+26naghxFS5rYiTZww=
golang.org/x/tools v0.0.0-20200604042327-9b20fe4cabe8/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE=
golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543 h1:E7g+9GITq07hpfrRu66IVDexMakfv52eLZ2CXBWiKr4=
golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
gonum.org/v1/gonum v0.0.0-20190331200053-3d26580ed485/go.mod h1:2ltnJ7xHfj0zHS40VVPYEAAMTa3ZGguvHGBSJeRWqE0=
gonum.org/v1/netlib v0.0.0-20190313105609-8cb42192e0e0/go.mod h1:wa6Ws7BG/ESfp6dHfk7C6KdzKA7wR7u/rKwOGE66zvw=
Expand All @@ -491,6 +497,7 @@ google.golang.org/genproto v0.0.0-20200319113533-08878b785e9c h1:5aI3/f/3eCZps9x
google.golang.org/genproto v0.0.0-20200319113533-08878b785e9c/go.mod h1:55QSHmfGQM9UVYDPBsyGGes0y52j32PQ3BqQfXhyH3c=
google.golang.org/genproto v0.0.0-20200515170657-fc4c6c6a6587 h1:1Ym+vvUpq1ZHvxzn34gENJX8U4aKO+vhy2P/2+Xl6qQ=
google.golang.org/genproto v0.0.0-20200515170657-fc4c6c6a6587/go.mod h1:YsZOwe1myG/8QRHRsmBRE1LrgQY60beZKjly0O1fX9U=
google.golang.org/genproto v0.0.0-20200526211855-cb27e3aa2013 h1:+kGHl1aib/qcwaRi1CbqBZ1rk19r85MNUf8HaBghugY=
google.golang.org/genproto v0.0.0-20200526211855-cb27e3aa2013/go.mod h1:NbSheEEYHJ7i3ixzK3sjbqSGDJWnxyFXZblF3eUsNvo=
google.golang.org/grpc v1.13.0/go.mod h1:yo6s7OP7yaDglbqo1J04qKzAhqBH6lvTonzMVmEdcZw=
google.golang.org/grpc v1.19.0/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZiDr8c=
Expand Down Expand Up @@ -540,6 +547,7 @@ gopkg.in/yaml.v2 v2.0.0-20170812160011-eb3733d160e7/go.mod h1:JAlM8MvJe8wmxCU4Bl
gopkg.in/yaml.v2 v2.2.1/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v2 v2.2.2 h1:ZCJp+EgiOT7lHqUV2J862kp8Qj64Jo6az82+3Td9dZw=
gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v2 v2.2.4 h1:/eiJrUcujPVeJ3xlSWaiNi3uSVmDGBK1pDHUHAnao1I=
gopkg.in/yaml.v2 v2.2.4/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gotest.tools v2.2.0+incompatible/go.mod h1:DsYFclhRJ6vuDpmuTbkuFWG+y2sxOXAzmJt81HFBacw=
honnef.co/go/tools v0.0.0-20190102054323-c2f93a96b099/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
Expand Down
47 changes: 35 additions & 12 deletions protos/feast/serving/ServingService.proto
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -91,35 +91,58 @@ message GetOnlineFeaturesRequest {
bool omit_entities_in_response = 3;

message EntityRow {
// Request timestamp of this row. This value will be used, together with maxAge,
// to determine feature staleness.
// Request timestamp of this row. This value will be used,
// together with maxAge, to determine feature staleness.
google.protobuf.Timestamp entity_timestamp = 1;

// Map containing mapping of entity name to entity value.
map<string,feast.types.Value> fields = 2;
}
}

message GetBatchFeaturesRequest {
// List of features that are being retrieved
repeated FeatureReference features = 3;

// Source of the entity dataset containing the timestamps and entity keys to retrieve
// features for.
DatasetSource dataset_source = 2;
}

message GetOnlineFeaturesResponse {
// Feature values retrieved from feast.
repeated FieldValues field_values = 1;

message FieldValues {
// Map of feature or entity name to feature/entity values.
// Timestamps are not returned in this response.
map<string,feast.types.Value> fields = 1;
map<string, feast.types.Value> fields = 1;
// Map of feature or entity name to feature/entity statuses/metadata.
map<string, FieldStatus> statuses = 2;
}

enum FieldStatus {
// Status is unset for this field.
INVALID = 0;

// Field value is present for this field and age is within max age.
PRESENT = 1;

// Values could be found for entity key and age is within max age, but
// this field value is assigned a value on ingestion into feast.
NULL_VALUE = 2;

// Entity key did not return any values as they do not exist in Feast.
// This could suggest that the feature values have not yet been ingested
// into feast or the ingestion failed.
NOT_FOUND = 3;

// Values could be found for entity key, but field values are outside the maximum
// allowable range.
OUTSIDE_MAX_AGE = 4;
}
}

message GetBatchFeaturesRequest {
// List of features that are being retrieved
repeated FeatureReference features = 3;

// Source of the entity dataset containing the timestamps and entity keys to retrieve
// features for.
DatasetSource dataset_source = 2;
}

message GetBatchFeaturesResponse {
Job job = 1;
}
Expand Down
18 changes: 11 additions & 7 deletions sdk/go/client.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,21 +59,25 @@ func (fc *GrpcClient) GetOnlineFeatures(ctx context.Context, req *OnlineFeatures
}
}

// strip projects from to projects
for _, fieldValue := range resp.GetFieldValues() {
stripFields := make(map[string]*types.Value, len(fieldValue.Fields))
for refStr, value := range fieldValue.Fields {
// strip projects from feature refs recieved
for _, fieldValues := range resp.GetFieldValues() {
stripFields := make(map[string]*types.Value, len(fieldValues.Fields))
stripStatuses := make(map[string]serving.GetOnlineFeaturesResponse_FieldStatus, len(fieldValues.Statuses))
for refStr, _ := range fieldValues.Fields {
_, isEntity := entityRefs[refStr]
stripRefStr := refStr
if !isEntity { // is feature ref
featureRef, err := parseFeatureRef(refStr, true)
if err != nil {
return nil, err
}
refStr = toFeatureRefStr(featureRef)
stripRefStr = toFeatureRefStr(featureRef)
}
stripFields[refStr] = value

stripFields[stripRefStr] = fieldValues.Fields[refStr]
stripStatuses[stripRefStr] = fieldValues.Statuses[refStr]
}
fieldValue.Fields = stripFields
fieldValues.Fields, fieldValues.Statuses = stripFields, stripStatuses
}

return &OnlineFeaturesResponse{RawResponse: resp}, err
Expand Down
13 changes: 13 additions & 0 deletions sdk/go/client_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@ func TestGetOnlineFeatures(t *testing.T) {
Features: []string{
"driver:rating",
"rating",
"null_value",
},
Entities: []Row{
{"driver_id": Int64Val(1)},
Expand All @@ -41,6 +42,12 @@ func TestGetOnlineFeatures(t *testing.T) {
Fields: map[string]*types.Value{
"driver_project/driver:rating": Int64Val(1),
"driver_project/rating": Int64Val(1),
"driver_project/null_value": {},
},
Statuses: map[string]serving.GetOnlineFeaturesResponse_FieldStatus{
"driver_project/driver:rating": serving.GetOnlineFeaturesResponse_PRESENT,
"driver_project/rating": serving.GetOnlineFeaturesResponse_PRESENT,
"driver_project/null_value": serving.GetOnlineFeaturesResponse_NULL_VALUE,
},
},
},
Expand All @@ -53,6 +60,12 @@ func TestGetOnlineFeatures(t *testing.T) {
Fields: map[string]*types.Value{
"driver:rating": Int64Val(1),
"rating": Int64Val(1),
"null_value": {},
},
Statuses: map[string]serving.GetOnlineFeaturesResponse_FieldStatus{
"driver:rating": serving.GetOnlineFeaturesResponse_PRESENT,
"rating": serving.GetOnlineFeaturesResponse_PRESENT,
"null_value": serving.GetOnlineFeaturesResponse_NULL_VALUE,
},
},
},
Expand Down
Loading