Exploration of SunBird Obsrv platform

[Alka1703]

Hello ,

I am trying to use the obsrv platform by first sending some data to create a dataset, data source and then trying to send some data for data ingestion. The sequence for doing this is:

1. Create Dataset using obsrv/v1/datasets endpoint
2. Create Datasource using obsrv/v1/datasources endpoint
3. Ingest data event using obsrv/v1/data/mosip-telemetry-dataset endpoint

I am getting error while trying to ingest data in the third step. attaching the error response.
I am attaching the dataset, datasource and data sent using these endpoints for reference. Please validate if there is some error in the process.

Also, once a dataset/datasource is created successfully, can we see it somewhere visually other than postman by hitting the list API.

DataIngestErrorResponse.txt
DataSet.txt
DataSource.txt
Event_Data_Ingeset.txt

[manjudr]

@Alka1703 - We encountered a Kafka connection issue with the API in the sandbox environment, but it has been resolved now. You should be able to ingest the Mosip data through API now.

[Alka1703]

@manjudr Once a DataSet, DataSource is created and data is ingested, how do we visually look at it? I am not able to view it in the Superset link under the dataset section.

[manjudr]

@Alka1703 - Please ensure following these steps.

• Kindly prepare the ingestion specification for your dataset. For reference, you can consult the documentation available at this link. Additionally, we have provided some examples in the Sunbird-Obsrv repository that you can also use as a guide.

• Proceed to create the data source record in the table by using the obsrv/v1/datasources API endpoint.

• The Datasource API requires both the ingestion spec and configurations.

• Once the data source records are successfully created, you can submit the ingestion spec using the provided API.

To submit the ingestion spec, use the sample curl command below:

curl --location 'http://localhost:8888/druid/indexer/v1/supervisor' \
--header 'Content-Type: application/json' \
--data '{
  "type": "kafka",
  "spec": {}
}'

I will share the druid ingestion URL over the mail.

[manjudr]

@Alka1703 - Also, please refer to this documentation to understand dataset creation workflow and configurations - https://github.com/Sunbird-Obsrv/obsrv-automation/blob/main/INSTALLATION.md#create-a-dataset

[manjudr]

@Alka1703 - Can we close this discussion if you are able to ingest the data?

[Alka1703]

Yes @manjudr . Please close the discussion. Thank you.