amex broken under requests but not urllib

[dtbartle]

When using requests, content type is sent as "Content-type" while urllib sends "Content-Type". Per https://datatracker.ietf.org/doc/html/rfc7231#section-3.1.1.5, the latter is correct. This evidently breaks amex starting a couple days ago.

[csingley]

Nice catch, thanks for reporting. Should be fixed in 655549a

[dtbartle]

Actually, not sure that's it, though removing "Content-type": mimetype, from http_headers does get it working...

[csingley]

No your're right, it's probably not. If you search Client.py (or this bug tracker) for "Amex" you'll find comments from the last round of header-related incantation engineering regarding XML quality scores etc. If urllib works for you and requests doesn't, you'd probably be better served by isolating ofxtools inside a virtualenv. OFX is deprecated as a transmission protocol, and life is short.

[dtbartle]

I managed to reproduce this with curl; it's pretty wacky, and seems to to do with header ordering:

OK:
curl --http1.1 -X POST -v 'https://online.americanexpress.com/myca/ofxdl/desktop/desktopDownload.do?request_type=nl_ofxdownload' -H 'Accept: */*, application/x-ofx, application/xml;q=2.9' -d @file -H 'User-Agent: InetClntApp/5.0' -H 'Content-Type: application/x-ofx' -H 'Accept-Encoding: identity'

Forbidden:
curl --http1.1 -X POST -v 'https://online.americanexpress.com/myca/ofxdl/desktop/desktopDownload.do?request_type=nl_ofxdownload' -H 'User-Agent: InetClntApp/5.0' -H 'Accept: */*, application/x-ofx, application/xml;q=2.9' -d @file -H 'Content-Type: application/x-ofx' -H 'Accept-Encoding: identity'

[dtbartle]

The requests library puts User-Agent first, I assume because it just updates the default headers, which list that first:

def default_headers():
    """
    :rtype: requests.structures.CaseInsensitiveDict
    """
    return CaseInsensitiveDict({
        'User-Agent': default_user_agent(),
        'Accept-Encoding': ', '.join(('gzip', 'deflate')),
        'Accept': '*/*',
        'Connection': 'keep-alive',
    })

You can work around this by setting headers on session, i.e. sess.headers = self.http_headers.

Forbidden:
send: b'POST /myca/ofxdl/desktop/desktopDownload.do?request_type=nl_ofxdownload HTTP/1.1\r\nHost: online.americanexpress.com\r\nUser-Agent: InetClntApp/3.0\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*, application/x-ofx, application/xml;q=2.9\r\nConnection: keep-alive\r\nContent-Type: application/x-ofx\r\nContent-Length: 694\r\n\r\n'

OK:
send: b'POST /myca/ofxdl/desktop/desktopDownload.do?request_type=nl_ofxdownload HTTP/1.1\r\nHost: online.americanexpress.com\r\nAccept-Encoding: identity\r\nUser-Agent: InetClntApp/3.0\r\nContent-type: application/ x-ofx\r\nAccept: */*, application/x-ofx, application/xml;q=0.9\r\nContent-Length: 694\r\n\r\n'

And for reference, what urllib sends (also OK):
send: b'POST /myca/ofxdl/desktop/desktopDownload.do?request_type=nl_ofxdownload HTTP/1.1\r\nAccept-Encoding: identity\r\nContent-Length: 694\r\nHost: online.americanexpress.com\r\nUser-Agent: InetClntApp/3.0\r\nContent-Type: application/x-ofx\r\nAccept: */*, application/x-ofx, application/xml;q=0.9\r\nConnection: close\r\n\r\n'

[csingley]

You're in it to win it!

Whatever homegrown jank Amex has up on the web seems to be in the van of the "code to Quicken not the spec" movement. If you're hardcore, you'll want to install Quicken & mitmproxy to directly observe the header sequence being set by the genuine article, rather than repeatedly lobbing codebreaker style guesses at their server.

But if you want to break up the code behind the USE_REQUESTS switch in OFXClient.post_request(), and kludge the headers after constructing session but before POSTing the request... I mean OK, I'd take a diff, especially if you were intent enough to include a unit test of this behavior.

But, and I'm gonna keep bringing this up, you could also just avoid using requests, by running ofxtools in a virtual env that doesn't have requests installed. Everything about OFX download is fairly terrible, and there's just not that much life left in the system over which to amortize a bunch of dev work.

[dtbartle]

Yeah, I ended up just setting USE_REQUESTS = False for amex. I wonder if there should be a CLI flag or Client arg to disable the requests module, though?