Skip to content

Commit

Permalink
Range query for custom attribute (#5426)
Browse files Browse the repository at this point in the history
* Add pinot dual visibility manager and new advance visibility option

* update visibility store and implemented unit test

* update: go mod tidy

* update go.sum

* Fix the start options for pinot

* update unit test

* run make cmds to update files

* update one unit test case that had errors

* fix unit test

* Fix pinot config file and add ES config to make it run with workers and frontend

* add ES into docker compose file

* Fix kafka producer message

* revert kafka change, update producer message struct to match with schema

* cleanup

* change tableName and places used it accordingly, to allow pinot to recognize the name

* add a pinotClientInterface, refactor unit test

* NewPinotConnectionClient should return a genetic value

* fix a failed unit test & add mock genericClient component

* rename pinotConnectionClient tobe pinotClient

* change PinotClient to be public

* Update visibility manager to use the new pinot generic client

* Fix the log format

* update kafka config to separate kafka topics for pinot and ES, add pinot visibility triple manager

* Fix typo in pinot table config

* change json tags to start with upper case to match pinot schema

* Fix run ID json tag and remove unused kafka key from schema

* fix a naming issue that cause pinot can't receive CloseTime

* change json tag so that closeStatus won't be ignored when it is 0

* add attr into pinot

* update decoded attr

* Update reading from pinot dynamic config

* add single quote to query

* correct order Query formations

* Add log for debug purpose

* fix can't unmarshal request.Attr error

* fix nil pointer in isRecordValid

* clean up

* Remove unnecessary debug info and unused message fields

* solve can't unmarshal attr issue

* update unit test to pass

* Get pinot table from config

* Update table name in config

* use table name from config

* clean up

* update unit test

* change couple types in pinot message

* update pinot visibility triple manager to write to pinot and ES (#5229)

* Cdnc 4574 (#5230)

* update pinot config to enable text_index so that we could use Text_match function in query

* implement customized attribute search

* delete an unused function

* change name to avoid build fail

* use reg exp to split with case insensitive 'and'

* clean up comment

* clean up

* Add pagination and flatten customized search attributes (#5234)

* implement pagination

* add a test case for nextPageToken

* add more test cases for pagination

* change elastic search token into pinot token

* clean up

* fix count didn'w work issue

* try to implement flatten schema, add attributes into schema

* add attributes into schema

* add some code to convert time to unix time in customized search attributes

* refactor previously added code to a function

* customized search attributes and unit tests

* Remove attr column and minor clean up

* edge case for ORDER BY in customized search attribute

* Fix typo in name

* add one more condition for parseLastElement

* split one element of customized search attribute by operator instead of space; add a filter for customized attributes prefix

* update unit test

* update unit test

---------

Co-authored-by: Neil Xie <neil.xie@uber.com>

* Adds Dynamic-config type (#5261)

- Refactors the config-store library to include a new dimension called 'configType' which allows for multiple stores.
- Refactors the config-store slightly to reflect the fact that it is indeed actually a daemon

These changes are tested and part of a larger set of changes of the 'zonal-isolation' feature

* clean up: delete one unused function, and one line refactor

* update config file for deleting Attr

* fix a nil pointer after removing Attr

* update a test case to cover multiple order by clause case

* Fix fmt

* Add pinot integration test (#5316)

* Add integration test for pinot and fix bugs

* Fix frontend to read from pinot when set up pinot test cluster

* Add a new map for Attr in response

---------

Co-authored-by: Bowen Xiao <xbowen@uber.com>

* Cdnc 4589 (#5318)

* Add pinot dual visibility manager and new advance visibility option

* update visibility store and implemented unit test

* update: go mod tidy

* update go.sum

* Fix the start options for pinot

* update unit test

* run make cmds to update files

* update one unit test case that had errors

* fix unit test

* Fix pinot config file and add ES config to make it run with workers and frontend

* add ES into docker compose file

* Fix kafka producer message

* revert kafka change, update producer message struct to match with schema

* cleanup

* change tableName and places used it accordingly, to allow pinot to recognize the name

* add a pinotClientInterface, refactor unit test

* NewPinotConnectionClient should return a genetic value

* fix a failed unit test & add mock genericClient component

* rename pinotConnectionClient tobe pinotClient

* change PinotClient to be public

* Update visibility manager to use the new pinot generic client

* Fix the log format

* update kafka config to separate kafka topics for pinot and ES, add pinot visibility triple manager

* Fix typo in pinot table config

* change json tags to start with upper case to match pinot schema

* Fix run ID json tag and remove unused kafka key from schema

* fix a naming issue that cause pinot can't receive CloseTime

* change json tag so that closeStatus won't be ignored when it is 0

* add attr into pinot

* update decoded attr

* Update reading from pinot dynamic config

* add single quote to query

* correct order Query formations

* Add log for debug purpose

* fix can't unmarshal request.Attr error

* fix nil pointer in isRecordValid

* clean up

* Remove unnecessary debug info and unused message fields

* solve can't unmarshal attr issue

* update unit test to pass

* Get pinot table from config

* Update table name in config

* use table name from config

* clean up

* update unit test

* change couple types in pinot message

* update pinot visibility triple manager to write to pinot and ES (#5229)

* Cdnc 4574 (#5230)

* update pinot config to enable text_index so that we could use Text_match function in query

* implement customized attribute search

* delete an unused function

* change name to avoid build fail

* use reg exp to split with case insensitive 'and'

* clean up comment

* clean up

* Add pagination and flatten customized search attributes (#5234)

* implement pagination

* add a test case for nextPageToken

* add more test cases for pagination

* change elastic search token into pinot token

* clean up

* fix count didn'w work issue

* try to implement flatten schema, add attributes into schema

* add attributes into schema

* add some code to convert time to unix time in customized search attributes

* refactor previously added code to a function

* customized search attributes and unit tests

* Remove attr column and minor clean up

* edge case for ORDER BY in customized search attribute

* Fix typo in name

* add one more condition for parseLastElement

* split one element of customized search attribute by operator instead of space; add a filter for customized attributes prefix

* update unit test

* update unit test

---------

Co-authored-by: Neil Xie <neil.xie@uber.com>

* Adds Dynamic-config type (#5261)

- Refactors the config-store library to include a new dimension called 'configType' which allows for multiple stores.
- Refactors the config-store slightly to reflect the fact that it is indeed actually a daemon

These changes are tested and part of a larger set of changes of the 'zonal-isolation' feature

* clean up: delete one unused function, and one line refactor

* update config file for deleting Attr

* fix a nil pointer after removing Attr

* update a test case to cover multiple order by clause case

* Fix fmt

* Add integration test for pinot

* Fix frontend to read from pinot when set up pinot test cluster

* Update history to load pinot config correctly

* add more test cases/most of them are failing

* fix WorkflowExecutionCloseStatus unmarshal panic

* add a new map for Attr in response

* pass upsert tests

* scanWorkflow pass

* pass listOpenWorkflow

* pass pagination tests

* all test passed

* BinaryChecksums support arrays

* make go-generate && make fmt && make lint && make copyright

* Minor tweaks

* Clean up

* handle CloseTime = missing case

* only deal with system keys

* delete single quote for search val if it is not string

* cleanup

* refactor utility functions to a new file for both OSS and Mono repo to use

* run make go-generate && make fmt && make lint && make copyright

---------

Co-authored-by: Neil Xie <neil.xie@uber.com>
Co-authored-by: neil-xie <104041627+neil-xie@users.noreply.github.com>
Co-authored-by: David Porter <david.porter@uber.com>

* Update Pinot query to order by closetime when query closed wf, order by runID when query open wf

* refactor pinotClient to pass in pinotConfig

* Revert "refactor pinotClient to pass in pinotConfig"

This reverts commit 4f403b2.

* refactor pinotClient to pass in pinotConfig

* PinotQueryValidator (#5333)

* add pinotQueryValidator, and change default order by to be StartTime

* clean up: delete comments

* clean up: run make copy right things

* refactor PinotQueryValidator

* refactor

* clean up: run copyright things

* Add limit clause to pinot queries (#5337)

* add limit clause to queries

* cleanup: run make copyright things

* Update all queries to order by startTime

* Adding a PInot/ES response comparator (#5353)

* add response comparator

*when comparison fails, it will return es result and log the error

* if can't start both mgr, use only one mgr

* Refactor code to determine read mode and which visibility manager to use when read mode is both

* Return valid response when one of the source is broken

---------

Co-authored-by: Neil Xie <neil.xie@uber.com>

* Fix rebase and lint

* Fix integration test and minor clean up

* more clean up

* Add more comments and more clean up

* Update to use constants for visibility store name instead of strings

* Enable json index (#5390)

* enable json index, change the quries for exact/partial match, update the unit tests

* make it pass integration test: found that order by is not supported in json index column

* update partial match query. It didn't work in the mono repo

* Implemented deletion method for Pinot visibility store (#5404)

* Rebase

* Uncomment code that caused error by idl changes

* More clean up

* turn off comparator

* Add pinot metrics client and update pinot visibility manager to use it (#5411)

* Add pinot metrics client and update pinot visibility manager to use it

* Update read and write mode to prepare for migration

* Add SecondsSinceEpoch field and update Pinot schema (#5418)

* Address comments part 1

* Address comments and fix Pinot integration test

* remove temporarily to rename folder

* Add back with new folder name

* Fix

* Minor fix for stopwatch

* Add more comments

* add range query and unit test

* support <, >, >=, <= for custom attributes

* remove dead code

* add unit tests

* add comment for range query function

---------

Co-authored-by: Neil Xie <neil.xie@uber.com>
Co-authored-by: neil-xie <104041627+neil-xie@users.noreply.github.com>
Co-authored-by: David Porter <david.porter@uber.com>
Co-authored-by: Shijie Sheng <shengs@uber.com>
  • Loading branch information
5 people authored Nov 3, 2023
1 parent 4ece98a commit 1649929
Show file tree
Hide file tree
Showing 3 changed files with 122 additions and 27 deletions.
4 changes: 2 additions & 2 deletions common/persistence/pinot/pinotVisibilityStore_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -185,7 +185,7 @@ LIMIT 0, 10
FROM %s
WHERE DomainID = 'bfd5c907-f899-4baf-a7b2-2ab85e623ebd'
AND IsDeleted = false
AND WorkflowID = 'wid' and ((JSON_MATCH(Attr, '"$.CustomStringField" is not null') AND REGEXP_LIKE(JSON_EXTRACT_SCALAR(Attr, '$.CustomStringField', 'string'), 'custom and custom2 or custom3 order by*')) or CustomIntField between 1 and 10)
AND WorkflowID = 'wid' and ((JSON_MATCH(Attr, '"$.CustomStringField" is not null') AND REGEXP_LIKE(JSON_EXTRACT_SCALAR(Attr, '$.CustomStringField', 'string'), 'custom and custom2 or custom3 order by*')) or (JSON_MATCH(Attr, '"$.CustomIntField" is not null') AND CAST(JSON_EXTRACT_SCALAR(Attr, '$.CustomIntField') AS INT) >= 1 AND CAST(JSON_EXTRACT_SCALAR(Attr, '$.CustomIntField') AS INT) <= 10))
Order BY StartTime DESC
LIMIT 0, 10
`, testTableName),
Expand Down Expand Up @@ -239,7 +239,7 @@ LIMIT 0, 10
FROM %s
WHERE DomainID = 'bfd5c907-f899-4baf-a7b2-2ab85e623ebd'
AND IsDeleted = false
AND CloseStatus < 0 and (JSON_MATCH(Attr, '"$.CustomKeywordField"=''keywordCustomized''') or JSON_MATCH(Attr, '"$.CustomKeywordField[*]"=''keywordCustomized''')) and JSON_MATCH(Attr, '"$.CustomIntField"=''10''') and (JSON_MATCH(Attr, '"$.CustomStringField" is not null') AND REGEXP_LIKE(JSON_EXTRACT_SCALAR(Attr, '$.CustomStringField', 'string'), 'String field is for text*'))
AND CloseStatus < 0 and (JSON_MATCH(Attr, '"$.CustomKeywordField"=''keywordCustomized''') or JSON_MATCH(Attr, '"$.CustomKeywordField[*]"=''keywordCustomized''')) and (JSON_MATCH(Attr, '"$.CustomIntField" is not null') AND CAST(JSON_EXTRACT_SCALAR(Attr, '$.CustomIntField') AS INT) <= 10) and (JSON_MATCH(Attr, '"$.CustomStringField" is not null') AND REGEXP_LIKE(JSON_EXTRACT_SCALAR(Attr, '$.CustomStringField', 'string'), 'String field is for text*'))
Order by DomainID Desc
LIMIT 11, 10
`, testTableName),
Expand Down
104 changes: 81 additions & 23 deletions common/pinot/pinotQueryValidator.go
Original file line number Diff line number Diff line change
Expand Up @@ -91,24 +91,58 @@ func (qv *VisibilityQueryValidator) validateWhereExpr(expr sqlparser.Expr) (stri
if expr == nil {
return "", nil
}
buf := sqlparser.NewTrackedBuffer(nil)

switch expr := expr.(type) {
case *sqlparser.AndExpr, *sqlparser.OrExpr:
return qv.validateAndOrExpr(expr)
case *sqlparser.ComparisonExpr:
return qv.validateComparisonExpr(expr)
case *sqlparser.RangeCond:
expr.Format(buf)
return buf.String(), nil
//return qv.validateRangeExpr(expr)
return qv.validateRangeExpr(expr)
case *sqlparser.ParenExpr:
return qv.validateWhereExpr(expr.Expr)
default:
return "", errors.New("invalid where clause")
}
}

// for "between...and..." only
// <, >, >=, <= are included in validateComparisonExpr()
func (qv *VisibilityQueryValidator) validateRangeExpr(expr sqlparser.Expr) (string, error) {
buf := sqlparser.NewTrackedBuffer(nil)
rangeCond := expr.(*sqlparser.RangeCond)
colName, ok := rangeCond.Left.(*sqlparser.ColName)
if !ok {
return "", errors.New("invalid range expression: fail to get colname")
}
colNameStr := colName.Name.String()

if !qv.isValidSearchAttributes(colNameStr) {
return "", fmt.Errorf("invalid search attribute %q", colNameStr)
}

if definition.IsSystemIndexedKey(colNameStr) {
expr.Format(buf)
return buf.String(), nil
}

//lowerBound, ok := rangeCond.From.(*sqlparser.ColName)
lowerBound, ok := rangeCond.From.(*sqlparser.SQLVal)
if !ok {
return "", errors.New("invalid range expression: fail to get lowerbound")
}
lowerBoundString := string(lowerBound.Val)

upperBound, ok := rangeCond.To.(*sqlparser.SQLVal)
if !ok {
return "", errors.New("invalid range expression: fail to get upperbound")
}
upperBoundString := string(upperBound.Val)

return fmt.Sprintf("(JSON_MATCH(Attr, '\"$.%s\" is not null') "+
"AND CAST(JSON_EXTRACT_SCALAR(Attr, '$.%s') AS INT) >= %s "+
"AND CAST(JSON_EXTRACT_SCALAR(Attr, '$.%s') AS INT) <= %s)", colNameStr, colNameStr, lowerBoundString, colNameStr, upperBoundString), nil
}

func (qv *VisibilityQueryValidator) validateAndOrExpr(expr sqlparser.Expr) (string, error) {
var leftExpr sqlparser.Expr
var rightExpr sqlparser.Expr
Expand Down Expand Up @@ -248,24 +282,48 @@ func (qv *VisibilityQueryValidator) processCustomKey(expr sqlparser.Expr) (strin
// get the value type
indexValType := common.ConvertIndexedValueTypeToInternalType(valType, log.NewNoop())

// Case2-1: when it is string, need partial match
if indexValType == types.IndexedValueTypeString {
// change to like statement for partial match
comparisonExpr.Operator = sqlparser.LikeStr
comparisonExpr.Right = &sqlparser.SQLVal{
Type: sqlparser.StrVal,
Val: []byte("%" + colValStr + "%"),
}
//return fmt.Sprintf("JSON_EXTRACT_SCALAR(Attr, '$.%s', 'STRING') LIKE '%%%s%%'", colNameStr, colValStr), nil
return fmt.Sprintf("(JSON_MATCH(Attr, '\"$.%s\" is not null') "+
"AND REGEXP_LIKE(JSON_EXTRACT_SCALAR(Attr, '$.%s', 'string'), '%s*'))", colNameStr, colNameStr, colValStr), nil
operator := comparisonExpr.Operator

switch indexValType {
case types.IndexedValueTypeString:
return processCustomString(comparisonExpr, colNameStr, colValStr), nil
case types.IndexedValueTypeKeyword:
return processCustomKeyword(operator, colNameStr, colValStr), nil
case types.IndexedValueTypeDatetime:
return processCustomNum(operator, colNameStr, colValStr, "BIGINT"), nil
case types.IndexedValueTypeDouble:
return processCustomNum(operator, colNameStr, colValStr, "DOUBLE"), nil
case types.IndexedValueTypeInt:
return processCustomNum(operator, colNameStr, colValStr, "INT"), nil
default:
return processEqual(colNameStr, colValStr), nil
}
// case2-2: otherwise, exact match
// case2-2-1: if it is keyword, need to deal with a situation when value is an array
if indexValType == types.IndexedValueTypeKeyword {
return fmt.Sprintf("(JSON_MATCH(Attr, '\"$.%s\"=''%s''') or JSON_MATCH(Attr, '\"$.%s[*]\"=''%s'''))",
colNameStr, colValStr, colNameStr, colValStr), nil
}

func processCustomNum(operator string, colNameStr string, colValStr string, valType string) string {
if operator == sqlparser.EqualStr {
return processEqual(colNameStr, colValStr)
}
return fmt.Sprintf("(JSON_MATCH(Attr, '\"$.%s\" is not null') "+
"AND CAST(JSON_EXTRACT_SCALAR(Attr, '$.%s') AS %s) %s %s)", colNameStr, colNameStr, valType, operator, colValStr)
}

func processEqual(colNameStr string, colValStr string) string {
return fmt.Sprintf("JSON_MATCH(Attr, '\"$.%s\"=''%s''')", colNameStr, colValStr)
}

func processCustomKeyword(operator string, colNameStr string, colValStr string) string {
return fmt.Sprintf("(JSON_MATCH(Attr, '\"$.%s\"%s''%s''') or JSON_MATCH(Attr, '\"$.%s[*]\"%s''%s'''))",
colNameStr, operator, colValStr, colNameStr, operator, colValStr)
}

func processCustomString(comparisonExpr *sqlparser.ComparisonExpr, colNameStr string, colValStr string) string {
// change to like statement for partial match
comparisonExpr.Operator = sqlparser.LikeStr
comparisonExpr.Right = &sqlparser.SQLVal{
Type: sqlparser.StrVal,
Val: []byte("%" + colValStr + "%"),
}
// case2-2-2: other cases:
return fmt.Sprintf("JSON_MATCH(Attr, '\"$.%s\"=''%s''')", colNameStr, colValStr), nil
return fmt.Sprintf("(JSON_MATCH(Attr, '\"$.%s\" is not null') "+
"AND REGEXP_LIKE(JSON_EXTRACT_SCALAR(Attr, '$.%s', 'string'), '%s*'))", colNameStr, colNameStr, colValStr)
}
41 changes: 39 additions & 2 deletions common/pinot/pinotQueryValidator_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,7 @@ func TestValidateQuery(t *testing.T) {
},
"Case6-1: complex query I: with parenthesis": {
query: "(CustomStringField = 'custom and custom2 or custom3 order by') or CustomIntField between 1 and 10",
validated: "((JSON_MATCH(Attr, '\"$.CustomStringField\" is not null') AND REGEXP_LIKE(JSON_EXTRACT_SCALAR(Attr, '$.CustomStringField', 'string'), 'custom and custom2 or custom3 order by*')) or CustomIntField between 1 and 10)",
validated: "((JSON_MATCH(Attr, '\"$.CustomStringField\" is not null') AND REGEXP_LIKE(JSON_EXTRACT_SCALAR(Attr, '$.CustomStringField', 'string'), 'custom and custom2 or custom3 order by*')) or (JSON_MATCH(Attr, '\"$.CustomIntField\" is not null') AND CAST(JSON_EXTRACT_SCALAR(Attr, '$.CustomIntField') AS INT) >= 1 AND CAST(JSON_EXTRACT_SCALAR(Attr, '$.CustomIntField') AS INT) <= 10))",
},
"Case6-2: complex query II: with only system keys": {
query: "DomainID = 'd-id' and (RunID = 'run-id' or WorkflowID = 'wid')",
Expand All @@ -71,7 +71,7 @@ func TestValidateQuery(t *testing.T) {
},
"Case6-4: complex query IV": {
query: "WorkflowID = 'wid' and (CustomStringField = 'custom and custom2 or custom3 order by' or CustomIntField between 1 and 10)",
validated: "WorkflowID = 'wid' and ((JSON_MATCH(Attr, '\"$.CustomStringField\" is not null') AND REGEXP_LIKE(JSON_EXTRACT_SCALAR(Attr, '$.CustomStringField', 'string'), 'custom and custom2 or custom3 order by*')) or CustomIntField between 1 and 10)",
validated: "WorkflowID = 'wid' and ((JSON_MATCH(Attr, '\"$.CustomStringField\" is not null') AND REGEXP_LIKE(JSON_EXTRACT_SCALAR(Attr, '$.CustomStringField', 'string'), 'custom and custom2 or custom3 order by*')) or (JSON_MATCH(Attr, '\"$.CustomIntField\" is not null') AND CAST(JSON_EXTRACT_SCALAR(Attr, '$.CustomIntField') AS INT) >= 1 AND CAST(JSON_EXTRACT_SCALAR(Attr, '$.CustomIntField') AS INT) <= 10))",
},
"Case7: invalid sql query": {
query: "Invalid SQL",
Expand Down Expand Up @@ -113,6 +113,43 @@ func TestValidateQuery(t *testing.T) {
query: "CustomIntField = 1 or CustomIntField = 2",
validated: "(JSON_MATCH(Attr, '\"$.CustomIntField\"=''1''') or JSON_MATCH(Attr, '\"$.CustomIntField\"=''2'''))",
},
"Case14-1: range query: custom filed": {
query: "CustomIntField BETWEEN 1 AND 2",
validated: "(JSON_MATCH(Attr, '\"$.CustomIntField\" is not null') AND CAST(JSON_EXTRACT_SCALAR(Attr, '$.CustomIntField') AS INT) >= 1 AND CAST(JSON_EXTRACT_SCALAR(Attr, '$.CustomIntField') AS INT) <= 2)",
},
"Case14-2: range query: system filed": {
query: "NumClusters BETWEEN 1 AND 2",
validated: "NumClusters between 1 and 2",
},
"Case15-1: custom date attribute less than": {
query: "CustomDatetimeField < 1697754674",
validated: "(JSON_MATCH(Attr, '\"$.CustomDatetimeField\" is not null') AND CAST(JSON_EXTRACT_SCALAR(Attr, '$.CustomDatetimeField') AS BIGINT) < 1697754674)",
},
"Case15-2: custom date attribute greater than or equal to": {
query: "CustomDatetimeField >= 1697754674",
validated: "(JSON_MATCH(Attr, '\"$.CustomDatetimeField\" is not null') AND CAST(JSON_EXTRACT_SCALAR(Attr, '$.CustomDatetimeField') AS BIGINT) >= 1697754674)",
},
"Case15-3: system date attribute greater than or equal to": {
query: "StartTime >= 1697754674",
validated: "StartTime >= 1697754674",
},
"Case16-1: custom int attribute greater than or equal to": {
query: "CustomIntField >= 0",
validated: "(JSON_MATCH(Attr, '\"$.CustomIntField\" is not null') AND CAST(JSON_EXTRACT_SCALAR(Attr, '$.CustomIntField') AS INT) >= 0)",
},
"Case16-2: custom double attribute greater than or equal to": {
query: "CustomDoubleField >= 0",
validated: "(JSON_MATCH(Attr, '\"$.CustomDoubleField\" is not null') AND CAST(JSON_EXTRACT_SCALAR(Attr, '$.CustomDoubleField') AS DOUBLE) >= 0)",
},
"Case17: custom keyword attribute greater than or equal to. Will return error run time": {
query: "CustomKeywordField < 0",
validated: "(JSON_MATCH(Attr, '\"$.CustomKeywordField\"<''0''') or JSON_MATCH(Attr, '\"$.CustomKeywordField[*]\"<''0'''))",
},
// TODO
"Case18: custom int order by. Will have errors at run time. Doesn't support for now": {
query: "CustomIntField = 0 order by CustomIntField desc",
validated: "JSON_MATCH(Attr, '\"$.CustomIntField\"=''0''') order by CustomIntField desc",
},
}

for name, test := range tests {
Expand Down

0 comments on commit 1649929

Please sign in to comment.