Error in http_statuses[[as.character(status)]] : subscript out of bounds

[Javdat]

Sometimes I am getting an error Error in http_statuses[[as.character(status)]] : subscript out of bounds while calling s3saveRDS() in aws.s3. As far as I could pinpoint, the issue originates from httr, more specifically in http_status.

This is the part of the code that is (I think) is problematic:

  status <- status_code(x)

  status_desc <- http_statuses[[as.character(status)]]
  if (is.na(status_desc)) {
    stop("Unknown http status code: ", status, call. = FALSE)
  }

This assumes that http_statuses[[as.character(status)]] will return NA when it is faced with a status code not present in http_statuses, and thus will be handed with: "Unknown http status code: ".

However, in reality, it seems http_statuses[[as.character(status)]] will just throw subscript out of bounds error.

Reproducible(ish) example:

Status present in http_statuses, works as expected:

> status <- status_code(100)
> status_desc <- httr:::http_statuses[[as.character(status)]]
> status_desc
[1] "Continue"

Unknown status, fails before reaching "Unknown http status code: " handler:

> status <- status_code(1000)
> status_desc <- httr:::http_statuses[[as.character(status)]]
Error in httr:::http_statuses[[as.character(status)]] : 
  subscript out of bounds
> status_desc
Error: object 'status_desc' not found

So, I think either httr:::http_statuses[[as.character(status)]] has to be called safely and error handled as NA, or indexing should happen one level only.

> status <- status_code(1000)
> status_desc <- httr:::http_statuses[as.character(status)] # Notice [] instead of [[]]
> status_desc
<NA> 
  NA 
> as.character(status_desc) # remove variable name
[1] NA

[sheffe]

Found this existing issue after lots of similar error messages. For what it's worth, I make ~100k read/write calls through aws.s3 every week, and I see this error on something like 1 in 200 of those calls. I've done some limited diagnostic checks to see whether this error happens more often on specific types of calls, payload sizes, etc., but it seems pretty random.

(I haven't been able to create a full reprex without doing something objectively dumb, like "write 100k tiny RDS files with aws.s3::s3saveRDS and see what happens." The example above is a nice demo of what happens.)

From my vantage point, this low-level error from httr is a complication because most of the aws.s3 high-level functions have been difficult to wrap in purrr adverbs like safely/slowly/insistently. By that I mean errors like this inside a package wrapper function can still cause a hard interruption, for reasons I haven't been able to trace yet. Any research to that end probably belongs in aws.s3 directly. I don't use many other high-level API packages that wrap httr and have this kind of call volume, but I suspect the general problem won't be exclusive to dealing with AWS resources.

Both httr and aws.s3 have large enough user footprints that I'd appreciate a maintainer's opinion on where to submit a PR -- I think @Javdat 's solution above makes sense, but after some digging I can't trace what else would be impacted.

(PS: For my use-cases, 100k read/writes a week with only a 0.5% error rate is pretty remarkable. Thanks for httr!)

[hadley]

I think is a fairly simple fix if someone wants to do a PR. Just check before subsetting:

  if (!http_statuses %in% http_statuses) {
    stop("Unknown http status code: ", status, call. = FALSE)
  }
  status_desc <- http_statuses[[as.character(status)]]