Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Combine error from vec_c() requiring columns to be same type #337

Closed
asadow opened this issue Oct 10, 2023 · 4 comments
Closed

Combine error from vec_c() requiring columns to be same type #337

asadow opened this issue Oct 10, 2023 · 4 comments

Comments

@asadow
Copy link

asadow commented Oct 10, 2023

When using pagination, an error is returned from paginate_req_perform() if columns of the same name have different types across pages.

Error in `vctrs::list_unchop()`:                          
! Can't combine `x[[1]]$y` <list> and `x[[5]]$y` <character>.
Run `rlang::last_trace()` to see where the error occurred.
@mgirlich
Copy link
Collaborator

It makes sense that this returns an error (type stability is a core principle of {vctrs}) and to me this is more a feature than a bug. Otherwise you won't know whether y will be a character or a list in the end.

So, this should clearly be handled by the function you use to parse the response.

Out of curiosity: do you have an example where this happened?

@asadow
Copy link
Author

asadow commented Oct 11, 2023

Unfortunately I can't make a reprex. The first page happened to have only scalars under column X. Another page happened to have said column containing scalars and vectors. The same thing happened for another column Y when I was pulling different dates (hence different pages).

In our case we don't have a data dictionary outlining types. However, I'm not convinced that types (i.e. having a complete data dictionary of endpoint columns) should be imposed before pulling from an API. Our API is in development such that columns are added depending on users' needs. To account for that in R functions would add unnecessary complexity.

I recall @hadley suggested a paginated perform could return a list so that it is on the user on how to handle it. Indeed that's the route I took by using paginate_next_request(). But this seems a roundabout way of dealing with the above.

@mgirlich
Copy link
Collaborator

You don't need to use paginate_next_request(). You can simply use something like

``` r
devtools::load_all("~/GitHub/httr2/")
#> ℹ Loading httr2

page_size <- 40

parse_flower_resp <- function(resp) {
  parsed <- resp_body_json(resp)
  results <- parsed$data
  data <- data.frame(
    Sepal.Length = sapply(results, `[[`, "Sepal.Length"),
    Sepal.Width = sapply(results, `[[`, "Sepal.Width"),
    Petal.Length = sapply(results, `[[`, "Petal.Length"),
    Petal.Width = sapply(results, `[[`, "Petal.Width"),
    Species = sapply(results, `[[`, "Species")
  )
  
  # list(data = data, count = parsed$count)
  # wrap data in a list
  list(data = list(data), count = parsed$count)
}

req_flowers <- request(example_url()) %>%
  req_url_path("/iris") %>%
  req_url_query(limit = page_size) %>%
  req_paginate_page_index(
    page_index = function(req, page) {
      req %>% req_url_query(page_index = page)
    },
    parse_resp = parse_flower_resp,
    n_pages = function(parsed) {
      total <- parsed$count
      ceiling(total / page_size)
    }
  )

paginate_req_perform(req_flowers)
#> [[1]]
#>    Sepal.Length Sepal.Width Petal.Length Petal.Width Species
#> 1           5.1         3.5          1.4         0.2  setosa
#> 2           4.9         3.0          1.4         0.2  setosa
#> 3           4.7         3.2          1.3         0.2  setosa
#> 4           4.6         3.1          1.5         0.2  setosa
#> 5           5.0         3.6          1.4         0.2  setosa
#> 6           5.4         3.9          1.7         0.4  setosa
#> 7           4.6         3.4          1.4         0.3  setosa
#> 8           5.0         3.4          1.5         0.2  setosa
#> 9           4.4         2.9          1.4         0.2  setosa
#> 10          4.9         3.1          1.5         0.1  setosa
#> 11          5.4         3.7          1.5         0.2  setosa
#> 12          4.8         3.4          1.6         0.2  setosa
#> 13          4.8         3.0          1.4         0.1  setosa
#> 14          4.3         3.0          1.1         0.1  setosa
#> 15          5.8         4.0          1.2         0.2  setosa
#> 16          5.7         4.4          1.5         0.4  setosa
#> 17          5.4         3.9          1.3         0.4  setosa
#> 18          5.1         3.5          1.4         0.3  setosa
#> 19          5.7         3.8          1.7         0.3  setosa
#> 20          5.1         3.8          1.5         0.3  setosa
#> 21          5.4         3.4          1.7         0.2  setosa
#> 22          5.1         3.7          1.5         0.4  setosa
#> 23          4.6         3.6          1.0         0.2  setosa
#> 24          5.1         3.3          1.7         0.5  setosa
#> 25          4.8         3.4          1.9         0.2  setosa
#> 26          5.0         3.0          1.6         0.2  setosa
#> 27          5.0         3.4          1.6         0.4  setosa
#> 28          5.2         3.5          1.5         0.2  setosa
#> 29          5.2         3.4          1.4         0.2  setosa
#> 30          4.7         3.2          1.6         0.2  setosa
#> 31          4.8         3.1          1.6         0.2  setosa
#> 32          5.4         3.4          1.5         0.4  setosa
#> 33          5.2         4.1          1.5         0.1  setosa
#> 34          5.5         4.2          1.4         0.2  setosa
#> 35          4.9         3.1          1.5         0.2  setosa
#> 36          5.0         3.2          1.2         0.2  setosa
#> 37          5.5         3.5          1.3         0.2  setosa
#> 38          4.9         3.6          1.4         0.1  setosa
#> 39          4.4         3.0          1.3         0.2  setosa
#> 40          5.1         3.4          1.5         0.2  setosa
#> 
#> [[2]]
#>    Sepal.Length Sepal.Width Petal.Length Petal.Width    Species
#> 1           5.0         3.5          1.3         0.3     setosa
#> 2           4.5         2.3          1.3         0.3     setosa
#> 3           4.4         3.2          1.3         0.2     setosa
#> 4           5.0         3.5          1.6         0.6     setosa
#> 5           5.1         3.8          1.9         0.4     setosa
#> 6           4.8         3.0          1.4         0.3     setosa
#> 7           5.1         3.8          1.6         0.2     setosa
#> 8           4.6         3.2          1.4         0.2     setosa
#> 9           5.3         3.7          1.5         0.2     setosa
#> 10          5.0         3.3          1.4         0.2     setosa
#> 11          7.0         3.2          4.7         1.4 versicolor
#> 12          6.4         3.2          4.5         1.5 versicolor
#> 13          6.9         3.1          4.9         1.5 versicolor
#> 14          5.5         2.3          4.0         1.3 versicolor
#> 15          6.5         2.8          4.6         1.5 versicolor
#> 16          5.7         2.8          4.5         1.3 versicolor
#> 17          6.3         3.3          4.7         1.6 versicolor
#> 18          4.9         2.4          3.3         1.0 versicolor
#> 19          6.6         2.9          4.6         1.3 versicolor
#> 20          5.2         2.7          3.9         1.4 versicolor
#> 21          5.0         2.0          3.5         1.0 versicolor
#> 22          5.9         3.0          4.2         1.5 versicolor
#> 23          6.0         2.2          4.0         1.0 versicolor
#> 24          6.1         2.9          4.7         1.4 versicolor
#> 25          5.6         2.9          3.6         1.3 versicolor
#> 26          6.7         3.1          4.4         1.4 versicolor
#> 27          5.6         3.0          4.5         1.5 versicolor
#> 28          5.8         2.7          4.1         1.0 versicolor
#> 29          6.2         2.2          4.5         1.5 versicolor
#> 30          5.6         2.5          3.9         1.1 versicolor
#> 31          5.9         3.2          4.8         1.8 versicolor
#> 32          6.1         2.8          4.0         1.3 versicolor
#> 33          6.3         2.5          4.9         1.5 versicolor
#> 34          6.1         2.8          4.7         1.2 versicolor
#> 35          6.4         2.9          4.3         1.3 versicolor
#> 36          6.6         3.0          4.4         1.4 versicolor
#> 37          6.8         2.8          4.8         1.4 versicolor
#> 38          6.7         3.0          5.0         1.7 versicolor
#> 39          6.0         2.9          4.5         1.5 versicolor
#> 40          5.7         2.6          3.5         1.0 versicolor
#> 
#> [[3]]
#>    Sepal.Length Sepal.Width Petal.Length Petal.Width    Species
#> 1           5.5         2.4          3.8         1.1 versicolor
#> 2           5.5         2.4          3.7         1.0 versicolor
#> 3           5.8         2.7          3.9         1.2 versicolor
#> 4           6.0         2.7          5.1         1.6 versicolor
#> 5           5.4         3.0          4.5         1.5 versicolor
#> 6           6.0         3.4          4.5         1.6 versicolor
#> 7           6.7         3.1          4.7         1.5 versicolor
#> 8           6.3         2.3          4.4         1.3 versicolor
#> 9           5.6         3.0          4.1         1.3 versicolor
#> 10          5.5         2.5          4.0         1.3 versicolor
#> 11          5.5         2.6          4.4         1.2 versicolor
#> 12          6.1         3.0          4.6         1.4 versicolor
#> 13          5.8         2.6          4.0         1.2 versicolor
#> 14          5.0         2.3          3.3         1.0 versicolor
#> 15          5.6         2.7          4.2         1.3 versicolor
#> 16          5.7         3.0          4.2         1.2 versicolor
#> 17          5.7         2.9          4.2         1.3 versicolor
#> 18          6.2         2.9          4.3         1.3 versicolor
#> 19          5.1         2.5          3.0         1.1 versicolor
#> 20          5.7         2.8          4.1         1.3 versicolor
#> 21          6.3         3.3          6.0         2.5  virginica
#> 22          5.8         2.7          5.1         1.9  virginica
#> 23          7.1         3.0          5.9         2.1  virginica
#> 24          6.3         2.9          5.6         1.8  virginica
#> 25          6.5         3.0          5.8         2.2  virginica
#> 26          7.6         3.0          6.6         2.1  virginica
#> 27          4.9         2.5          4.5         1.7  virginica
#> 28          7.3         2.9          6.3         1.8  virginica
#> 29          6.7         2.5          5.8         1.8  virginica
#> 30          7.2         3.6          6.1         2.5  virginica
#> 31          6.5         3.2          5.1         2.0  virginica
#> 32          6.4         2.7          5.3         1.9  virginica
#> 33          6.8         3.0          5.5         2.1  virginica
#> 34          5.7         2.5          5.0         2.0  virginica
#> 35          5.8         2.8          5.1         2.4  virginica
#> 36          6.4         3.2          5.3         2.3  virginica
#> 37          6.5         3.0          5.5         1.8  virginica
#> 38          7.7         3.8          6.7         2.2  virginica
#> 39          7.7         2.6          6.9         2.3  virginica
#> 40          6.0         2.2          5.0         1.5  virginica
#> 
#> [[4]]
#>    Sepal.Length Sepal.Width Petal.Length Petal.Width   Species
#> 1           6.9         3.2          5.7         2.3 virginica
#> 2           5.6         2.8          4.9         2.0 virginica
#> 3           7.7         2.8          6.7         2.0 virginica
#> 4           6.3         2.7          4.9         1.8 virginica
#> 5           6.7         3.3          5.7         2.1 virginica
#> 6           7.2         3.2          6.0         1.8 virginica
#> 7           6.2         2.8          4.8         1.8 virginica
#> 8           6.1         3.0          4.9         1.8 virginica
#> 9           6.4         2.8          5.6         2.1 virginica
#> 10          7.2         3.0          5.8         1.6 virginica
#> 11          7.4         2.8          6.1         1.9 virginica
#> 12          7.9         3.8          6.4         2.0 virginica
#> 13          6.4         2.8          5.6         2.2 virginica
#> 14          6.3         2.8          5.1         1.5 virginica
#> 15          6.1         2.6          5.6         1.4 virginica
#> 16          7.7         3.0          6.1         2.3 virginica
#> 17          6.3         3.4          5.6         2.4 virginica
#> 18          6.4         3.1          5.5         1.8 virginica
#> 19          6.0         3.0          4.8         1.8 virginica
#> 20          6.9         3.1          5.4         2.1 virginica
#> 21          6.7         3.1          5.6         2.4 virginica
#> 22          6.9         3.1          5.1         2.3 virginica
#> 23          5.8         2.7          5.1         1.9 virginica
#> 24          6.8         3.2          5.9         2.3 virginica
#> 25          6.7         3.3          5.7         2.5 virginica
#> 26          6.7         3.0          5.2         2.3 virginica
#> 27          6.3         2.5          5.0         1.9 virginica
#> 28          6.5         3.0          5.2         2.0 virginica
#> 29          6.2         3.4          5.4         2.3 virginica
#> 30          5.9         3.0          5.1         1.8 virginica

Created on 2023-10-11 with reprex v2.0.2


This way you will get a list of responses.

> However, I'm not convinced that types (i.e. having a complete data dictionary of endpoint columns) should be imposed before pulling from an API. Our API is in development such that columns are added depending on users' needs. To account for that in R functions would add unnecessary complexity.

I don't think we impose anything. You get total freedom how (or even whether) you want to parse the data. You could simply use

```r
parse_resp <- function(resp) {
  list(
    data = resp,
    ... # whatever else you need for the pagination
  )
}

and parse it in the end.

Does that help you?

@asadow
Copy link
Author

asadow commented Oct 11, 2023

Thank you! Wrapping data in a list is brilliant. Being unfamiliar with vctrs, I assumed vec_c() was more like bind_rows() or rbind().

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants