bytecode string returned when page has charset=UTF-8

[j0hnsmith]

I had a situation (with both requests and urllib2) where a page that had <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" /> was being returned as a bytestring <type 'str'> but did contain unicode characters (due to a server misconfiguration I assume). So when I tried to use it I got the classic UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 419: ordinal not in range(128)

Is this something that requests could fix? Is this something that requests would want to fix?

[kennethreitz]

Requests only attempts to decode charsets specified in HTTP Headers (in the upcoming release).

However, there is a utility function that will attempt to decode based on the HTML tags. If the content isn't actually in the specified encoding, however, there's nothing that can be done (aside from ignoring the invalid charecters).