EOF exceptions on empty responses w/ :as :clojure [1.1.0]

[hlship]

I have hit an issue where, when I supply the :as :clojure option, clj-http will always attempt to parse the response, even when blank.

This is not ideal for me, and is generally problematic for any HTTP request that may return a body (to be parsed as JSON or EDN, etc.) or may respond with a status code and empty body.

This has only just come up because of ring-middleware-format:0.5.0, which now allows responses that are truly empty (in the past, a nil body would be converted to the string "nil" or "null").

Here's the exception I'm hitting:

                      users-spec/eval21796/fn/fn/fn    users_spec.clj:  178
                       users-spec/eval21796/fn/show    users_spec.clj:   59
                                                ...                        
                                clj-http.client/get        client.clj:  739
               clj-http.client/wrap-unknown-host/fn        client.clj:  636
                       clj-http.links/wrap-links/fn         links.clj:   50
                   clj-http.cookies/wrap-cookies/fn       cookies.clj:  121
                     clj-http.client/wrap-method/fn        client.clj:  576
              clj-http.client/wrap-nested-params/fn        client.clj:  617
                clj-http.client/wrap-form-params/fn        client.clj:  593
               clj-http.client/wrap-content-type/fn        client.clj:  477
            clj-http.client/wrap-accept-encoding/fn        client.clj:  502
                     clj-http.client/wrap-accept/fn        client.clj:  487
                 clj-http.client/wrap-exceptions/fn        client.clj:  149
            clj-http.client/wrap-output-coercion/fn        client.clj:  366
                                                ...                        
                       clj-http.client/eval15959/fn        client.clj:  348
                clj-http.client/coerce-clojure-body        client.clj:  310
                                                ...                        
                          clj-http.client/parse-edn        client.clj:   46
                                 clojure.core/apply          core.clj:  628
                                                ...                        
               clojure.tools.reader.edn/read-string           edn.clj:  406
                      clojure.tools.reader.edn/read           edn.clj:  357
                      clojure.tools.reader.edn/read           edn.clj:  364
                                                ...                        
     clojure.tools.reader.reader-types/reader-error  reader_types.clj:  337
                               clojure.core/ex-info          core.clj: 4577
     clojure.lang.ExceptionInfo: EOF
         data: {:type :reader-exception}

My suggestion is that the :as functionality return nil when the body is empty.

I'm attempting a workaround where I do not use :as, but do supply a custom middleware to do the same thing ... with the "empty" check.

[hlship]

Hm. This code makes it look like it's doing the right thing:

(defn wrap-output-coercion
  "Middleware converting a response body from a byte-array to a different
  object. Defaults to a String if no :as key is specified, the
  `coerce-response-body` multimethod may be extended to add
  additional coercions."
  [client]
  (fn [req]
    (let [{:keys [body] :as resp} (client req)]
      (if body
        (coerce-response-body req resp)
        resp))))

Let me check to see exactly what's in my response that's causing this kind of problem.

[hlship]

The issue is that even with an empty body, there's still an non-nil :body, a FilterStream.

so the conversion code is called, even when the FilterStream is empty.

At the very least, a quick check of the content-length header would be an improvement.

After that it gets more complicated. If mark is supported, you can attempt to read a single byte, then reset.

Ideally, lower layers of clj-http would ensure that :body was nil if the actual response is empty.

[hlship]

Looks like the lower levels do know that there is no input stream, no real body.

[hlship]

Verified this occurs in 1.1.0.

[dakrone]

Yes, I think this is a byproduct of the switch from a String body to a stream, in the past it the body could be nil as you said, but now it is always a stream (even if an empty one).

I will work on fixing this.

[hlship]

Ok, thanks. I think I can do a workaround by defining a custom coerce-response-body method that eats the EOF exception.

[bilus]

For anyone who may come across this problem, the quick & dirty workaround below may save a few minutes:

(ns my-ns
  (:require [clj-http.client :as http]))

(defmethod http/coerce-response-body :clojure [req resp]
  (if (> (Integer/parseInt (or (get-in resp [:headers "Content-Length"]) "0")) 0)
    (http/coerce-clojure-body req resp)
    (assoc resp :body nil)))

It is specific to :clojure but should be pretty easy to adapt to any format. Not the best example of sound engineering practices but I expect it to work till the problem is fixed.

[kamituel]

I just encountered that issue too. For now, @bilus's workaround seems to be working (thanks).

[smee]

The same problem occurs if a server sends an empty body with "content-encoding: gzip".

clj-http/src/clj_http/client.clj

Line 311 in 3da50bf

(defmethod decompress-body "gzip"

tries to unzip the empty body, which fails. According to rabbitmq/rabbitmq-management#154 (comment) this behaviour looks like a client error.

[dakrone]

I finally pushed a commit for this, thanks for your patience!