Using VCR.py causes 404 for Flickr upload specifically :S

[fatuhoku]

More problems with posting. This is very strange behaviour:

        def should_talk_to_Flickr():
            r = requests.post("http://api.flickr.com/services/upload/")
            print(r.status_code) # returns with HTTP 200, body contains XML complaining about lack of api_key parameter. Fine. This is normal behaviour

Instead of transparently recording the requests that occur, VCR has completely hi-jacked the call somehow:

        def should_still_talk_to_Flickr():
            with vcr.use_cassette('fixtures/flickr/reach.yaml'): # this file doesn't exist yet
                r = requests.post("http://api.flickr.com/services/upload/")
                print(r.status_code) # returns with HTTP 404 with empty body

[fatuhoku]

Let me know if you can reproduce this problem.

In a similar vein, I've written a single test to demonstrate the problem:

        def flickr_should_respond_with_200():
            r = requests.post("http://api.flickr.com/services/upload")
            r.status_code.should.be.equal(200)  # FINE
            with vcr.use_cassette('fixtures/flickr/reach.yaml'):
                r = requests.post("http://api.flickr.com/services/upload")
                r.status_code.should.be.equal(200) # NOT FINE

I ran this test with vcrpy==0.2.1; it passes before and after the fixture is generated. Great.
I ran this test again with vcrpy==0.3.0 and the HTTP response code just suddenly becomes 404. Bizarre.

Wireshark says that no request is made within the with block. This leads me to think that this is a problem within the requests patching code.

I'm so close to being a happy user of vcrpy, but it just needs to be more robust! The patching and hackery of HTTP connection classes make it hard for me to contribute back to the project in a timely manner.

[fatuhoku]

Using another URL, such as requestb.in is fine; I don't get 404s. It's just this Flickr URL. What's wrong with it!

[kevin1024]

Yes, I can reproduce the problem. How bizarre!

[kevin1024]

Ahh it looks like the Host header is disappearing somehow. That would explain the 404.

[kevin1024]

Never mind, I think the problem is that it is sending the POST to the full URL including protocol. Here is the raw data sent down the socket by VCR:

POST http://api.flickr.com/services/upload HTTP/1.1
Host: api.flickr.com
Content-Length: 0
Accept-Encoding: gzip, deflate, compress
Accept: */*
User-Agent: python-requests/1.2.3 CPython/2.7.3 Darwin/12.4.0

Here is what unVCR'd httplib sends:

POST /services/upload HTTP/1.1
Host: api.flickr.com
Content-Length: 0
Accept-Encoding: gzip, deflate, compress
Accept: */*
User-Agent: python-requests/1.2.3 CPython/2.7.3 Darwin/12.4.0

Here is what curl sends:

POST /services/upload HTTP/1.1
User-Agent: curl/7.24.0 (x86_64-apple-darwin12.0) libcurl/7.24.0 OpenSSL/0.9.8x zlib/1.2.5
Host: api.flickr.com
Accept: */*

[kevin1024]

Gotta run to work but that must be the problem!

[kevin1024]

Looks like Flickr's API server is actually out of spec:

To allow for transition to absoluteURIs in all requests in future versions of HTTP, all HTTP/1.1 servers MUST accept the absoluteURI form in requests, even though HTTP/1.1 clients will only generate them in requests to proxies.

To be fair, VCR is out of spec too :)

[fatuhoku]

Ah cool! Impressive skill. For some reason I had Wireshark and it didn't occur to me that I should compare the requests.

Well, now the mystery is solved, what's the fix? Will VCR have to use the path segment rather than the full URL to save all those poor out-of-spec servers of our Internet from getting confused? =]