consume libhoney responses channel in a thread, preventing a deadlock

We found that libhoney provides a responses() channel which MUST be read to
prevent eventual deadlock of the process. libhoney-rust writes to the responses
channel for every event written to honeycomb. This eventually causes the
work_sender thread to block, which causes the trace-data sending thread to
block.

We now consume the honeycomb responses in a tight loop, eventually exiting
if we detect that the channel has disconnected and all messages have been
consumed (the `responses.recv.is_err()` case.)

tracing-honeycomb/src/honeycomb.rs

@@ -24,6 +24,20 @@ impl HoneycombTelemetry {
     pub(crate) fn new(cfg: libhoney::Config, sample_rate: Option<u32>) -> Self {
         let honeycomb_client = libhoney::init(cfg);
 
+        // Handle the libhoney response channel by consuming and ignoring messages. This prevents a
+        // deadlock because the responses() channel is bounded and gains an item for every event
+        // emitted.
+        let responses = honeycomb_client.responses();
+        std::thread::spawn(move || {
+            loop {
+                if responses.recv().is_err() {
+                    // If we receive an error, the channel is empty & disconnected. No need to keep
+                    // this thread around.
+                    break;
+                }
+            }
+        });
+
         // publishing requires &mut so just mutex-wrap it
         // FIXME: may not be performant, investigate options (eg mpsc)
         let honeycomb_client = Mutex::new(honeycomb_client);

tracing-honeycomb/src/visitor.rs

@@ -140,7 +140,10 @@ pub(crate) fn span_to_values(
     match span.completed_at.duration_since(span.initialized_at) {
         Ok(d) => {
             // honeycomb-special (I think, todo: get full list of known values)
-            values.insert("duration_ms".to_string(), json!(d.as_secs_f64() * MILLIS_PER_SECOND));
+            values.insert(
+                "duration_ms".to_string(),
+                json!(d.as_secs_f64() * MILLIS_PER_SECOND),
+            );
         }
         Err(e) => {
             eprintln!("error comparing system times in tracing-honeycomb, indicates possible clock skew: {:?}", e);