Druid Ingestion Spec Support for CSP Changes

[Shakthieshwari]

Hi Team,

@anandp504 @manjudr @sowmya-dixit Can you please help us clarify the below query ?

In the druid ingestion spec, do we need to do any changes to the druid spec to support dynamically any CSP's(Cloud Service Provider)

For Ex :-
{"type":"index","spec":{"ioConfig":{"type":"index","inputSource":{"type": "azure","uris": ["azure://telemetry-data-store/observation/status/sl_observation_status.json"]},"inputFormat":{"type":"json"}},"tuningConfig":{"type":"index","partitionsSpec":{"type":"dynamic"}},"dataSchema":{"dataSource":"sl-observation-status","granularitySpec":{"type":"uniform","queryGranularity":"DAY","rollup":false,"segmentGranularity":"DAY"},"timestampSpec":{"column":"updatedAt","format":"auto"},"dimensionsSpec":{"dimensions":["status","entity_externalId","entity_id","entity_type","solution_id","solution_externalId","submission_id","entity_name","solution_name","role_title","school_name","school_code","school_externalId","state_name","state_code","state_externalId","district_name","district_code","district_externalId","block_name","block_code","block_externalId","cluster_name","cluster_code","cluster_externalId","completedDate","channel","parent_channel","program_id","program_externalId","program_name","app_name","user_id","private_program","solution_type","organisation_name","ecm_marked_na","board_name","updatedAt","organisation_id","user_type","observed_school_name","observed_school_id","observed_school_code","observed_state_name","observed_state_id","observed_state_code","observed_district_name","observed_district_id","observed_district_code","observed_block_name","observed_block_id","observed_block_code","observed_cluster_name","observed_cluster_id","observed_cluster_code"]},"metricsSpec":[]}}}

In this druid ingestion spec, should we make inputSource.type and uri's variablised dynamically, so that it support's any csp (or) since druid ingestion spec is part of configuration, we will keep as it and change it manually based on the CSP

Cc- @Ashwiniev95 @vijiurs @snehangsude

[rhwarrier]

@anandp504 @manjudr please take a look

[manjudr]

Yes. The druid supports ingesting the data from any cloud storage service (S3, Azure and GCP).
here are some documentation where you can configure the input sources - https://druid.apache.org/docs/latest/ingestion/native-batch.html#splittable-input-sources

Just you need to configure your input source in the ingestion spec to pull the data from the source.

[Shakthieshwari]

@manjudr How are we planning to support Oracle Cloud Storage service ? because in the druid documentation i could not find a support for oracle for the ingestion spec? can you please help me out here ?

Thanks

[manjudr]

I believe the OCI Object store is equivalent to AWS S3 Object Store. You might probably try the S3 Input source once. https://druid.apache.org/docs/latest/ingestion/native-batch-input-sources.html#s3-input-source

[Shakthieshwari]

ok, will check and get back @manjudr .

Thanks

[manjudr]

@Shakthieshwari - If this issue is resolved, can we close this?