fread showing HTTP error when read.csv does not

[dkgaraujo]

fread is able to correctly load a csv file that was downloaded to a local folder, but not the same csv file from the web. Function read.csv is able to correctly load the same csv file from the web. The issue was perceived in the following code as seen in the attached PDF, which also contains the session info. Also, I would note that I'm not running this behind any corporate firewall (although I also got the same error when I ran from behind a corporate firewall in another computer.)

fread_bug_report.pdf

[ben-schwen]

It seems that curl has problems with unencoded urls.

url = "https://github.com/OpportunityInsights/EconomicTracker/raw/main/data/Google Mobility - County - Daily.csv"

# fails
dt = fread(url)
# works
dt = fread(utils::URLencode(url))

df = read.csv(url, nrows = 10)

To not rely on utils::URLencode, one could use

invalid_chars = "[^][!$&'()*+,;=:/?@#ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789._~-]"
valid_url = gsub(invalid_chars, "%20\\1", url)
fread(valid_url)

[MichaelChirico]

thanks for the investigation... why then does read.csv succeed? does it have such a redirect/is it doing url encoding?

…

On Wed, Aug 5, 2020, 12:45 PM ben-schwen ***@***.***> wrote: It seems that curl has problems with unencoded urls. url = "https://github.com/OpportunityInsights/EconomicTracker/raw/main/data/Google Mobility - County - Daily.csv" # fails dt = fread(url) # works dt = fread(utils::URLencode(url)) df = read.csv(url, nrows = 10) To not rely on utils, one could use invalid_chars = "[^][!$&'()*+,;=:/?@#ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789._~-]" valid_url = gsub(invalid_chars, "%20\\1", url) fread(valid_url) — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#4659 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB2BA5JN2WVKCDGLKGF6YOLR7GED3ANCNFSM4PUGMHTA> .

[ben-schwen]

thanks for the investigation... why then does read.csv succeed? does it have such a redirect/is it doing url encoding?
…
On Wed, Aug 5, 2020, 12:45 PM ben-schwen @.**> wrote: It seems that curl has problems with unencoded urls. url = "https://github.com/OpportunityInsights/EconomicTracker/raw/main/data/Google Mobility - County - Daily.csv" # fails dt = fread(url) # works dt = fread(utils::URLencode(url)) df = read.csv(url, nrows = 10) To not rely on utils, one could use invalid_chars = "[^][!$&'()+,;=:/?@#ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789._~-]" valid_url = gsub(invalid_chars, "%20\1", url) fread(valid_url) — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#4659 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB2BA5JN2WVKCDGLKGF6YOLR7GED3ANCNFSM4PUGMHTA .

As far as I understand: read.table (read.csv is just a wrapper function) opens a connection to an url and then proceeds with calling base::scan. This itself calls do_scan from base scan.c. fread in contrary, tries not to read the file directly but downloads a tempfile first.

Furthermore it is also mentioned in the base::connections docs 'Most methods do not percent-encode special characters such as spaces in http:// URLs (see URLencode), but it seems the "wininet" method does.'

It seems like this is another follow-up on #1686

Strange enough it appears that read.csv is working with a not encoded url on Windows but not on Linux.