Field type conflict blocks the output buffer

[kostasb]

High level description: A type mismatch for a single point totally blocks the output buffer for outputs.influxdb.

Version tested: v1.1.2

input plugins: any
output plugins: outputs.influxdb in HTTP mode

Conditions:

A field key for a measurement already exists with a defined context type (e.g. float) in the backend InfluxDB
A metric arrives to Telegraf for that measurement & field with a different context type (e.g. int)

Issue:

The output queue gets stuck due to type mismatch. Telegraf indefinitely retries to write the mismatching point and does not flush the output buffer at all. Any following points are stacked up and are not written to InfluxDB.

e.g., using the Telegraf inputs.http_listener:

curl -v "http://localhost:8186/write?db=telegraf" --data-binary "test value=1"
curl -v "http://localhost:8186/write?db=telegraf" --data-binary "test value=1i"

2017/01/10 13:46:46 I! Output [influxdb] buffer fullness: 1 / 10000 metrics. Total gathered metrics: 2. Total dropped metrics: 0.
2017/01/10 13:46:46 E! InfluxDB Output Error: {"error":"field type conflict: input field \"value\" on measurement \"test\" is type integer, already exists as type float"}

 curl -v "http://localhost:8186/write?db=telegraf" --data-binary "test value=1"

2017/01/10 13:47:08 I! Output [influxdb] buffer fullness: 2 / 10000 metrics. Total gathered metrics: 3. Total dropped metrics: 0.
2017/01/10 13:47:08 E! InfluxDB Output Error: {"error":"field type conflict: input field \"value\" on measurement \"test\" is type integer, already exists as type float"}

Result: Output buffer fullness increases indefinitely.

Sample config used for the above test:

 [[inputs.http_listener]]
   service_address = ":8186"
   read_timeout = "10s"
   write_timeout = "10s"
   max_body_size = 0
   max_line_size = 0

[[outputs.influxdb]]
  urls = ["http://localhost:8086"] # required
  database = "telegraf" # required
  retention_policy = ""
  write_consistency = "any"
  timeout = "5s"

Proposal: add a max-attempts parameter for type conflicts to avoid indefinitely blocking the buffer.

[wiebeytec]

I guess it would be too much against the design philosophy of Telegraf do bookkeeping of the (type of) measurements as they come it? Influx DB creates the measurement with a type when it sees it first. Telegraf could do the same, so it would be able to do type checking.

[kostasb]

@wiebeytec As of the current version Telegraf has no persistence so upon restart it would have no way to know what schema already exists in InfluxDB, unless it performs schema exploration.

[sparrc]

Telegraf can't track the types of points as they flow through the system, as this doesn't provide any guarantees with data that comes from a separate source anyways.

I'm not sure the best way to handle this, maybe if InfluxDB fails the write then we should discard that batch (Influx does write the well-formed points even when it returns an error code).

[kostasb]

@sparrc we need to further test this, in my repro case once the buffer gets blocked due to the type mismatch no new points make it into InfluxDB.

[phemmer]

Influx does write the well-formed points even when it returns an error code

Actually, this is false. see influxdata/influxdb#4856 (comment)

[sparrc]

@phemmer, you're right, I thought that InfluxDB handled mismatched types the same as malformed points (it doesn't)

I created an issue but it might be a dupe: influxdata/influxdb#7814

[sparrc]

Unfortunately I don't think there is anything telegraf can do to fix this at the moment.

I'd like to hear what other users think, but it might be best to simply drop batches when receiving a 400. This could lead to dropped metrics in the case of mismatched types, but the only alternative is to let the mismatched point get stuck in the buffers, which can only be recovered from by restarting telegraf (and thus dropping even more points).

as @phemmer mentioned, the only sure-fire workaround for now will be to use 1-metric batch sizes until influxdata/influxdb#7814 is fixed.

[kostasb]

+1 on waiting for Influxdb issue 7814 to be fixed and then go with dropping the points in a batch that receives a 400

[sparrc]

I have a fix for this at #2311.

One caveat is that this fix will only work in combination with InfluxDB version 1.2+