Readme downloaded with wrong encoding

[Bilge]

Downloading this readme results in a string with the wrong encoding (should be UTF-8).

client.readme('ScriptFUSION/Porter', accept: 'application/vnd.github.raw')

encoding = ASCII-8BIT

This problem manifests itself when trying to parse it with a markdown processor such as Kramdown.

Kramdown::Document.new(
  client.readme('ScriptFUSION/Porter', accept: 'application/vnd.github.raw')
)

Error: "\xE2" from ASCII-8BIT to UTF-8

[tarebyte]

@Bilge are you able to reproduce this bug if you make the request via CURL? And then would you mind pasting your findings?

If you could paste your results using curl -v so that we can verify the headers that would be ✨

[Bilge]

How could I possibly reproduce it using cURL given that this is a Ruby encoding error?

[kytrinyx]

@Bilge Sorry—our request was unclear. Seeing the verbose curl output is always our first step in troubleshooting these things, even if we suspect that the problem is in Ruby.

We're seeing Content-Type: application/vnd.github.raw; charset=utf-8, which indeed looks correct.

curl -v -H "Authorization: token $GITHUB_TOKEN" -H "Accept: application/vnd.github.raw" https://api.github.com/repos/ScriptFUSION/Porter/readme

* TCP_NODELAY set
* Connected to api.github.com (192.30.253.117) port 443 (#0)
* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate: *.github.com
* Server certificate: DigiCert SHA2 High Assurance Server CA
* Server certificate: DigiCert High Assurance EV Root CA
> GET /repos/ScriptFUSION/Porter/readme HTTP/1.1
> Host: api.github.com
> User-Agent: curl/7.54.0
> Authorization: token :-)
> Accept: application/vnd.github.raw
>
< HTTP/1.1 200 OK
< Server: GitHub.com
< Date: Thu, 14 Sep 2017 17:29:25 GMT
< Content-Type: application/vnd.github.raw; charset=utf-8
< Content-Length: 28025
< Status: 200 OK
< X-RateLimit-Limit: 5000
< X-RateLimit-Remaining: 4993
< X-RateLimit-Reset: 1505413429
< Cache-Control: private, max-age=60, s-maxage=60
< Vary: Accept, Authorization, Cookie, X-GitHub-OTP
< ETag: "37c51935bde28a31cbbfc6e5451c1e41"
< Last-Modified: Sat, 26 Aug 2017 17:08:37 GMT
< X-OAuth-Scopes: admin:org, admin:org_hook, admin:public_key, admin:repo_hook, delete_repo, gist, notifications, repo, user
< X-Accepted-OAuth-Scopes:
< X-GitHub-Media-Type: github.v3; param=raw
< Access-Control-Expose-Headers: ETag, Link, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval
< Access-Control-Allow-Origin: *
< Content-Security-Policy: default-src 'none'
< Strict-Transport-Security: max-age=31536000; includeSubdomains; preload
< X-Content-Type-Options: nosniff
< X-Frame-Options: deny
< X-XSS-Protection: 1; mode=block
< X-Runtime-rack: 0.058001
< X-GitHub-Request-Id: A008:3DED:25EF253:4EA3277:59BABC75
<

Classifying this as a bug in the library, thanks for reporting it.

[hahmed]

@kytrinyx can you point me along the right direction in fixing this?

Looks like all we need to do is check if the response has base64 inside the encoding then encode content as base64.

def readme(repo, options={})
  response = get "#{Repository.path repo}/readme", options
  if response[:encoding] == "base64"
    response[:content] = Base64.strict_encode64(response[:content])
   end
  response
end

hahmed@cfb7f54#diff-c5bbd24771f20209a3fa4e3766ba01afL19

[kytrinyx]

Looks like all we need to do is check if the response has base64 inside the encoding then encode content as base64.

This looks like a different issue. Did you mean to solve this one?
#663