Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Revamp Obsrv Sink to Index All Data in Elastic search #5

Open
6 tasks
manjudr opened this issue May 3, 2023 · 2 comments
Open
6 tasks

Revamp Obsrv Sink to Index All Data in Elastic search #5

manjudr opened this issue May 3, 2023 · 2 comments

Comments

@manjudr
Copy link
Collaborator

manjudr commented May 3, 2023

Project Details

What is Sunbird Obsrv?

Sunbird Obsrv comprises several pluggable tools and microservices that come together to enable observability features on any platform/solution. This includes the ability to capture granular events via telemetry, create measures, and observe various events/actions carried out by the system/users/devices (like IoT devices) on any platform/solution.

Sunbird Obsrv can be utilized as an independent building block by adopters or as part of a system that employs other Sunbird building blocks. Sunbird Obsrv comes with a set of microservices, APIs, and some utility SDKs to make it easy for adopters to rapidly enable powerful data processing and aggregation infrastructure to process telemetry data, validate telemetry stream data, as well as aggregate and generate actionable insights via APIs. It also has built-in open data cataloguing and publishing capability. It is built keeping extensibility in mind so that adopters have the flexibility to adapt the telemetry and tools to their specific use cases.
More details are here

Features to be implemented

Provision the Elasticsearch using a Helm chart and configuring the Obsrv sink to index all data to Elasticsearch.

  • Current Scenario: At present, Obsrv reads messages from the Kafka sink topic to index all data into analytical storage apache druid.

  • Acceptance Criteria: Improve Obsrv by modifying its configuration to enable the reading of messages from the sink topic and indexing all messages to Elasticsearch.

Learning Path

Product Set Up

Milestones

  • Understanding of the existing apache druid sink configuration
  • Understanding elastic search index mapping spec
  • Understanding of elastic search Basic APIs
  • Create a helm chart to provision the elastic search
  • Create an index mapping file to index the data into an elastic search by reading from sink topic
  • Query the result from the elastic search and validate the result.
@wasup-yash
Copy link

I can work on this in case #4 is not free to be assigned.

@NEGRAH-S
Copy link

I am interested to contribute on the above issue

manjudr pushed a commit that referenced this issue Jun 1, 2023
#5 [SV] - Master dataset processing pipeline
manjudr pushed a commit that referenced this issue Jun 1, 2023
fear: updated actions as per terraform changes
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants