[Epic]: Add OpenSearch to the Cluster for many Analytics and Search Functions

[mrhegemon]

Dependent on

• Getting MoCap done
• Adding Metadata to Scens
• Adding Metadata to Static Resources

Stories

V1

Send tons of logs and data streams to Open Search

• Full-text search Static Resources / Scenes
  • Metadata tags
  • Full text
  • 2D / 3D
  • geodata / AR

Send mocap frames to OpenSearch

• schemas:
  • MoCap Frame
  • IK body
  • 1 point+ object tracking

V2

• Make some pre-canned DSL searches
  • Make from Aggregation - https://opensearch.org/docs/latest/aggregations/

V3

• Add Vector Search for crazy machine learning stuff
  • Hook up to Kubeflow
  • https://opensearch.org/docs/latest/ml-commons-plugin/algorithms/
  • https://opensearch.org/docs/latest/search-plugins/neural-search/
  • [RFC] OpenSearch neural-search plugin opensearch-project/neural-search#11

Notes

Make it run on K8s

• https://opensearch.org/blog/setup-multinode-cluster-kubernetes/
• https://opensearch.org/docs/latest/tools/k8s-operator/

Improve logging

We need to integrate it into the engine, our development, and DevOps workflow. The tasks for this are:

• Set up a separate kubernetes cluster dedicated to logging and monitoring using Open Search.
• Ensure our Open Search implementation has access to all of Ethereal Engine and monitors all deployments (currently dev, app, exp)
• Add additional logs and log levels to cover all levels of Ethereal Engine (information to come)
• Build a native Open Search dashboard
• Link Open Search to the appropriate Ethereal Engine deployment's admin panel and surface hot statistics like instance health, crashes, etc. that will display in a new area of admin panel https://www.flaticon.com/free-icon/log_1960242?term=logging&page=1&position=1&origin=tag&related_id=1960242
• Revamp the user analytics portion of the admin panel to surface user relevant KPIs
  - [ ] new users
  - [ ] active users within the last 15, 30 days
  - [ ] average user time spent for different locations
  - [ ] number of page views by users with accounts and guests
  - [ ] Unique Visitors
  - [ ] Session Duration
  - [ ] Collect UserAgent strings(useful for VR devices)

Issues

• Pods that crash should post their error to the analytics server to be preserved and read in the /admin/servers route #7221
• Ability to collect logs of crashed pods ir-engine-launcher#69
• Move analytics entirely to opensearch #9699