SourcesAvro

The avro source accepts avro encoded events. This source will listen on both IPv4 and IPv6 TCP sockets with a configurable port.

Each accepted connection receives its own dedicated thread which:

• Decodes the given payload's header, ensuring the payload is valid Avro.
• Records the payload, as published, as metric::event::Raw types into Hopper queues for later processing.

Configuration

• enabled :: whether the source is enabled [default: true]
• host :: the hostname to accept connections on [default: localhost]
• port :: the port for the source to listen on, both IPv4 and IPv6 [default: 2002]
• forwards :: the filters and/or sinks to forward into [default: []]

The forward concept is described in detail here.

You may configure multiple avro sources.

Example

[sources]
  [sources.avro.primary]
  enabled = true
  port = 2002
  forwards = []

  [sources.avro.secondary]
  enabled = true
  port = 2003
  forwards = []

  [sources.avro.disabled]
  enabled = false
  port = 2004
  forwards = []

Wire Protocol

The avro source accepts length prefixed transmissions of the form:

| length - 4 bytes, big-endian | header - N bytes | avro_blob - M bytes |

where:

• length :: A 32 bit, big-endian unsigned integer representing the size in bytes of the header and the subsequent avro_blob.
• header :: A version prefixed [header] governing how to publish the given avro_blob. See Headers.
• avro_blob :: An avro encoded binary blob.

Headers

Version prefixed headers control which publication features are available to a client.

The following versions are currently supported:

• V1

Available Clients

• pycernan.avro.Client (Python, Coming Soon)