fluent-plugin-grafana-loki can generate invalid requests

[candlerb]

Describe the bug
Under certain circumstances, fluent-plugin-grafana-loki can generate invalid requests

1. If a label contains a double quote, it's not escaped in the POST data sent to loki. As a result, loki rejects it with:

   HTTP/1.1 500 Internal Server Error
   Content-Type: text/plain; charset=utf-8
   X-Content-Type-Options: nosniff
   Date: Wed, 21 Aug 2019 09:34:34 GMT
   Content-Length: 54
   
   parse error at line 1, col 71: literal not terminated
   

   (Aside: IMO loki should generate a 400 error here, not a 500 error. A 500 error suggests that the error is temporary and the request should be resubmitted).

2. If a label is nil or empty string, it's still included in the label set as an empty string.

It's unclear whether (2) is a problem or not. I'm not sure if/when loki itself suppresses empty labels, or indeed whether empty labels have value or are expected to be preserved. However it's convenient to suppress them because fluentd's own record_transformer does not.

e.g. extracting labels from elastic beats, where the field may or may not be present in a given record:

<filter **>
  @type record_transformer
  enable_ruby
  <record>
    job ${tag}
    source ${record.dig('log', 'source', 'address')}
    facility ${record.dig('syslog', 'facility_label')}
    severity ${record.dig('syslog', 'severity_label')}
    program ${record.dig('process', 'program')}
  </record>
</filter>

To Reproduce

Most easily replicated using:

    extra_labels {"foo":"\"", "bar":"", "baz":null}

(Of course that's silly, but in general, labels are from unsanitised data and could contain anything)

Expected behavior
At minimum: double-quotes in labels to be either escaped or removed.

Environment:

• Ubuntu 18.04 lxd container
• Loki v0.3.0 installed from binary package
• fluentd installed by gem install fluentd fluent-plugin-grafana-loki --no-doc

fluent-plugin-grafana-loki (1.0.2)
fluentd (1.6.3)