…nd frontend

…cognize the name

…not visibility triple manager

* update pinot config to enable text_index so that we could use Text_match function in query

* implement customized attribute search

* delete an unused function

* change name to avoid build fail

* use reg exp to split with case insensitive 'and'

* clean up comment

* clean up

* implement pagination

* add a test case for nextPageToken

* add more test cases for pagination

* change elastic search token into pinot token

* clean up

* fix count didn'w work issue

* try to implement flatten schema, add attributes into schema

* add attributes into schema

* add some code to convert time to unix time in customized search attributes

* refactor previously added code to a function

* customized search attributes and unit tests

* Remove attr column and minor clean up

* edge case for ORDER BY in customized search attribute

* Fix typo in name

* add one more condition for parseLastElement

* split one element of customized search attribute by operator instead of space; add a filter for customized attributes prefix

* update unit test

* update unit test

---------

Co-authored-by: Neil Xie <neil.xie@uber.com>

- Refactors the config-store library to include a new dimension called 'configType' which allows for multiple stores.
- Refactors the config-store slightly to reflect the fact that it is indeed actually a daemon

These changes are tested and part of a larger set of changes of the 'zonal-isolation' feature

* Add integration test for pinot and fix bugs

* Fix frontend to read from pinot when set up pinot test cluster

* Add a new map for Attr in response

---------

Co-authored-by: Bowen Xiao <xbowen@uber.com>

* Add pinot dual visibility manager and new advance visibility option

* update visibility store and implemented unit test

* update: go mod tidy

* update go.sum

* Fix the start options for pinot

* update unit test

* run make cmds to update files

* update one unit test case that had errors

* fix unit test

* Fix pinot config file and add ES config to make it run with workers and frontend

* add ES into docker compose file

* Fix kafka producer message

* revert kafka change, update producer message struct to match with schema

* cleanup

* change tableName and places used it accordingly, to allow pinot to recognize the name

* add a pinotClientInterface, refactor unit test

* NewPinotConnectionClient should return a genetic value

* fix a failed unit test & add mock genericClient component

* rename pinotConnectionClient tobe pinotClient

* change PinotClient to be public

* Update visibility manager to use the new pinot generic client

* Fix the log format

* update kafka config to separate kafka topics for pinot and ES, add pinot visibility triple manager

* Fix typo in pinot table config

* change json tags to start with upper case to match pinot schema

* Fix run ID json tag and remove unused kafka key from schema

* fix a naming issue that cause pinot can't receive CloseTime

* change json tag so that closeStatus won't be ignored when it is 0

* add attr into pinot

* update decoded attr

* Update reading from pinot dynamic config

* add single quote to query

* correct order Query formations

* Add log for debug purpose

* fix can't unmarshal request.Attr error

* fix nil pointer in isRecordValid

* clean up

* Remove unnecessary debug info and unused message fields

* solve can't unmarshal attr issue

* update unit test to pass

* Get pinot table from config

* Update table name in config

* use table name from config

* clean up

* update unit test

* change couple types in pinot message

* update pinot visibility triple manager to write to pinot and ES (#5229)

* Cdnc 4574 (#5230)

* update pinot config to enable text_index so that we could use Text_match function in query

* implement customized attribute search

* delete an unused function

* change name to avoid build fail

* use reg exp to split with case insensitive 'and'

* clean up comment

* clean up

* Add pagination and flatten customized search attributes (#5234)

* implement pagination

* add a test case for nextPageToken

* add more test cases for pagination

* change elastic search token into pinot token

* clean up

* fix count didn'w work issue

* try to implement flatten schema, add attributes into schema

* add attributes into schema

* add some code to convert time to unix time in customized search attributes

* refactor previously added code to a function

* customized search attributes and unit tests

* Remove attr column and minor clean up

* edge case for ORDER BY in customized search attribute

* Fix typo in name

* add one more condition for parseLastElement

* split one element of customized search attribute by operator instead of space; add a filter for customized attributes prefix

* update unit test

* update unit test

---------

Co-authored-by: Neil Xie <neil.xie@uber.com>

* Adds Dynamic-config type (#5261)

- Refactors the config-store library to include a new dimension called 'configType' which allows for multiple stores.
- Refactors the config-store slightly to reflect the fact that it is indeed actually a daemon

These changes are tested and part of a larger set of changes of the 'zonal-isolation' feature

* clean up: delete one unused function, and one line refactor

* update config file for deleting Attr

* fix a nil pointer after removing Attr

* update a test case to cover multiple order by clause case

* Fix fmt

* Add integration test for pinot

* Fix frontend to read from pinot when set up pinot test cluster

* Update history to load pinot config correctly

* add more test cases/most of them are failing

* fix WorkflowExecutionCloseStatus unmarshal panic

* add a new map for Attr in response

* pass upsert tests

* scanWorkflow pass

* pass listOpenWorkflow

* pass pagination tests

* all test passed

* BinaryChecksums support arrays

* make go-generate && make fmt && make lint && make copyright

* Minor tweaks

* Clean up

* handle CloseTime = missing case

* only deal with system keys

* delete single quote for search val if it is not string

* cleanup

* refactor utility functions to a new file for both OSS and Mono repo to use

* run make go-generate && make fmt && make lint && make copyright

---------

Co-authored-by: Neil Xie <neil.xie@uber.com>
Co-authored-by: neil-xie <104041627+neil-xie@users.noreply.github.com>
Co-authored-by: David Porter <david.porter@uber.com>

…by runID when query open wf

This reverts commit 4f403b2.

* add pinotQueryValidator, and change default order by to be StartTime

* clean up: delete comments

* clean up: run make copy right things

* refactor PinotQueryValidator

* refactor

* clean up: run copyright things

* add limit clause to queries

* cleanup: run make copyright things

* add response comparator

*when comparison fails, it will return es result and log the error

* if can't start both mgr, use only one mgr

* Refactor code to determine read mode and which visibility manager to use when read mode is both

* Return valid response when one of the source is broken

---------

Co-authored-by: Neil Xie <neil.xie@uber.com>

* enable json index, change the quries for exact/partial match, update the unit tests

* make it pass integration test: found that order by is not supported in json index column

* update partial match query. It didn't work in the mono repo

* Implemented deletion method for Pinot visibility store (#5404)

#5411)

* Add pinot metrics client and update pinot visibility manager to use it

…uery

…DNC_5946_RangeQuery