New component: Acknowledgement Extension

[splunkericl]

The purpose and use-cases of the new component

purpose

The purpose of this extension is to improve the overall data resiliency of otel collector data ingestion. When an upstream agent sends data to otel collector, the agent might want to ensure the the event is processed(processed can be defined by how it is configured). If the event is not processed(e.g: host process killed and restarted) and acknowledged, the upstream agent can choose to send the event again.

Use cases

• Splunk HEC receiver would like to use this extension to track what event has been persisted to queue, then expose an ack endpoint for upstream agent to query
• Future consideration: E2E Ack for users that use Splunk HEC Receiver -> processors -> Splunk HEC exporters to perform E2E ACK for an event.

Example configuration for the component

Interface

// AckExtension is an extension to check whether an event has been processed. The processing can be done either synchronously or asynchronously.
type AckExtension interface {
	extension.Extension

      // ProcessEvent marks the beginning of processing an event. It generates an ack ID for the associated parition ID. ACK IDs are only unique within a partition. Two partitions can have the same ACK IDs but they are generated for different events.
      ProcessEvent(partitionID string) (ackID uint64)

      // Ack acknowledges an event has been processed.
      Ack(partitionID string, ackID uint64)

      // QueryAcks checks the statuses of given ackIDs for a partition.
      QueryAcks(partitionID string, ackIDs []uint64) map[uint64]bool
}

Configurations

Extension Configurations

type AckExtensionConfig struct {
    // StorageType defines the storage type of the extension. Currently planned for In memory type. Future consideration is disk type.
    StorageType string `mapstructure:"StorageType,omitempty"`
}

How a component can use this extension

type Config struct {
   ....

   // if Extension is provided, loads extension to ACK events. Otherwise, disable ACK
   AckExtension *component.ID `mapstructure:"ackExtension"`
}

Telemetry data types supported

Generic extension and they should be supported for any data types.

Is this a vendor-specific component?

• This is a vendor-specific component
• If this is a vendor-specific component, I am proposing to contribute and support it as a representative of the vendor.

Code Owner(s)

@splunkericl

Sponsor (optional)

@dmitryax

Additional context

No response

[atoulme]

I will sponsor this proposal.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

[github-actions]

This issue has been closed as inactive because it has been stale for 120 days with no activity.

[splunkericl]

Starting this work this sprint

[andrzej-stencel]

I don't undertstand why such extension is required. It seems that a client sending data to the otelcol's Splunk HEC receiver should be able to rely on the HTTP response from the receiver. If the receiver responds with 200 OK, this means the message was successfully processed (either sent out via an exporter or stored in a queue in the collector).

Can you explain why this is not enough and this extension is needed for the Splunk HEC receiver? Is this a specific need of this receiver or could other receivers also benefit from it, and if so, how?

[atoulme]

This component is now present in the contrib distro, closing.