You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
recently we have deployed multiple instances of OpenSearch via K8S operator . We are fairly new to OS so we're trying to understand how the things work under the hood, so that we can investigate properly in case of problems. At this moment we're trying to understand how the data ingestion process works internally in the OpenSearch. Unfortunately there's literally nothing in the OS documentation how the data ingestion process is handled in detail. Our current setup looks like below (we don't use Data Prepper):
FluentBit -> OpenSearch ingest service LB -> Dedicated ingest nodes -> ?? (master-> data nodes)
What we are trying to understand is:
Do we need dedicated ingest nodes if there's no ingest pipeline configured? Based on the docs, dedicated ingest nodes are only useful to run ingest pipelines, is there anything else the dedicated ingest nodes do?
How is the ingest process working in general? Can this be described in detail in the official documentation? We'd like to understand the whole process, let's say we have an OS cluster where we have:
How does the ingest flow look like once the data is received? I'd assume it's something like:
ingest node -> ask master node which data node(s) to use -> data node(s).
But this is not described in the documentation at all (among with who and how is it decided which data nodes to use).
Also based on the official docs, each node is a coordinating node unless dedicated coordinating nodes are specified. How to measure / based on what to decide whether dedicated coordinating node is necessary?
I think we can consider this issue as documentation request type as well.
Ingest process information missing
Hey all,
recently we have deployed multiple instances of OpenSearch via K8S operator . We are fairly new to OS so we're trying to understand how the things work under the hood, so that we can investigate properly in case of problems. At this moment we're trying to understand how the data ingestion process works internally in the OpenSearch. Unfortunately there's literally nothing in the OS documentation how the data ingestion process is handled in detail. Our current setup looks like below (we don't use Data Prepper):
FluentBit -> OpenSearch ingest service LB -> Dedicated ingest nodes -> ?? (master-> data nodes)
What we are trying to understand is:
3 dedicated cluster manager nodes
3 dedicated ingest nodes
x dedicated datanodes
How does the ingest flow look like once the data is received? I'd assume it's something like:
ingest node -> ask master node which data node(s) to use -> data node(s).
But this is not described in the documentation at all (among with who and how is it decided which data nodes to use).
Also based on the official docs, each node is a coordinating node unless dedicated coordinating nodes are specified. How to measure / based on what to decide whether dedicated coordinating node is necessary?
I think we can consider this issue as documentation request type as well.
Many thanks!
Related component
Other
To Reproduce
Expected behavior
Additional Details
No response
The text was updated successfully, but these errors were encountered: