Error in bq_table_upload - Error: Invalid JSON payload received. Unknown name "autodetect" at 'job.configuration.load'

[selesnow]

Hello!

I have an error when try loading data to BigQuery into new table.

Error: Invalid JSON payload received. Unknown name "autodetect" at 'job.configuration.load': Proto field is not repeating, cannot start list. [invalid]

If i load data into exist table i have't error.

reprex

library(bigrquery)
library(magrittr)

bq_auth(path = "C:/my_develop_workshop/ppc_digest/token_bigquery/bq_service_key.json")

bq_table(project = "netpeak-1079",
         dataset = "concert_ua",
         table   = "mytable") %>%
  bq_table_upload(values = mtcars,
                  create_disposition = "CREATE_IF_NEEDED",
                  write_disposition = "WRITE_APPEND")
#> Error: Invalid JSON payload received. Unknown name "autodetect" at 'job.configuration.load': Proto field is not repeating, cannot start list. [invalid]

^{Created on 2019-11-14 by the reprex package (v0.3.0)}

Session info

devtools::session_info()
#> - Session info ----------------------------------------------------------
#>  setting  value                       
#>  version  R version 3.5.3 (2019-03-11)
#>  os       Windows Server x64          
#>  system   x86_64, mingw32             
#>  ui       RTerm                       
#>  language (EN)                        
#>  collate  English_United States.1252  
#>  ctype    English_United States.1252  
#>  tz       Europe/Helsinki             
#>  date     2019-11-14                  
#> 
#> - Packages --------------------------------------------------------------
#>  package     * version    date       lib source                          
#>  askpass       1.1        2019-01-13 [1] CRAN (R 3.5.3)                  
#>  assertthat    0.2.1      2019-03-21 [1] CRAN (R 3.5.3)                  
#>  backports     1.1.5      2019-10-02 [1] CRAN (R 3.5.3)                  
#>  bigrquery   * 1.2.0.9000 2019-08-07 [1] Github (r-dbi/bigrquery@5f43228)
#>  bit           1.1-14     2018-05-29 [1] CRAN (R 3.5.2)                  
#>  bit64         0.9-7      2017-05-08 [1] CRAN (R 3.5.2)                  
#>  callr         3.2.0      2019-03-15 [1] CRAN (R 3.5.3)                  
#>  cli           1.1.0      2019-03-19 [1] CRAN (R 3.5.3)                  
#>  crayon        1.3.4      2017-09-16 [1] CRAN (R 3.5.3)                  
#>  curl          4.2        2019-09-24 [1] CRAN (R 3.5.3)                  
#>  DBI           1.0.0      2018-05-02 [1] CRAN (R 3.5.3)                  
#>  desc          1.2.0      2018-05-01 [1] CRAN (R 3.5.3)                  
#>  devtools      2.0.2      2019-04-08 [1] CRAN (R 3.5.3)                  
#>  digest        0.6.22     2019-10-21 [1] CRAN (R 3.5.3)                  
#>  evaluate      0.13       2019-02-12 [1] CRAN (R 3.5.3)                  
#>  fs            1.3.1      2019-05-06 [1] CRAN (R 3.5.3)                  
#>  gargle        0.4.0      2019-10-04 [1] CRAN (R 3.5.3)                  
#>  glue          1.3.1      2019-03-12 [1] CRAN (R 3.5.3)                  
#>  highr         0.8        2019-03-20 [1] CRAN (R 3.5.3)                  
#>  htmltools     0.3.6      2017-04-28 [1] CRAN (R 3.5.3)                  
#>  httr          1.4.1      2019-08-05 [1] CRAN (R 3.5.3)                  
#>  jsonlite      1.6        2018-12-07 [1] CRAN (R 3.5.3)                  
#>  knitr         1.23       2019-05-18 [1] CRAN (R 3.5.3)                  
#>  magrittr    * 1.5        2014-11-22 [1] CRAN (R 3.5.3)                  
#>  memoise       1.1.0      2017-04-21 [1] CRAN (R 3.5.3)                  
#>  openssl       1.4.1      2019-07-18 [1] CRAN (R 3.5.3)                  
#>  pillar        1.4.2      2019-06-29 [1] CRAN (R 3.5.3)                  
#>  pkgbuild      1.0.3      2019-03-20 [1] CRAN (R 3.5.3)                  
#>  pkgconfig     2.0.3      2019-09-22 [1] CRAN (R 3.5.3)                  
#>  pkgload       1.0.2      2018-10-29 [1] CRAN (R 3.5.3)                  
#>  prettyunits   1.0.2      2015-07-13 [1] CRAN (R 3.5.3)                  
#>  processx      3.3.1      2019-05-08 [1] CRAN (R 3.5.3)                  
#>  ps            1.3.0      2018-12-21 [1] CRAN (R 3.5.3)                  
#>  R6            2.4.0      2019-02-14 [1] CRAN (R 3.5.3)                  
#>  Rcpp          1.0.2      2019-07-25 [1] CRAN (R 3.5.3)                  
#>  remotes       2.0.4      2019-04-10 [1] CRAN (R 3.5.3)                  
#>  rlang         0.4.0      2019-06-25 [1] CRAN (R 3.5.3)                  
#>  rmarkdown     1.12       2019-03-14 [1] CRAN (R 3.5.3)                  
#>  rprojroot     1.3-2      2018-01-03 [1] CRAN (R 3.5.3)                  
#>  sessioninfo   1.1.1      2018-11-05 [1] CRAN (R 3.5.3)                  
#>  stringi       1.4.3      2019-03-12 [1] CRAN (R 3.5.3)                  
#>  stringr       1.4.0      2019-02-10 [1] CRAN (R 3.5.3)                  
#>  tibble        2.1.3      2019-06-06 [1] CRAN (R 3.5.3)                  
#>  usethis       1.5.0      2019-04-07 [1] CRAN (R 3.5.3)                  
#>  withr         2.1.2      2018-03-15 [1] CRAN (R 3.5.3)                  
#>  xfun          0.7        2019-05-14 [1] CRAN (R 3.5.3)                  
#>  yaml          2.2.0      2018-07-25 [1] CRAN (R 3.5.2)                  
#> 
#> [1] C:/Users/Alsey/Documents/R/win-library/3.5
#> [2] C:/Program Files/R/R-3.5.3/library

[georgewoodhead]

We are getting the same error when trying to upload data to BigQuery into a new table.

[ChrisJohnsonUMG]

I'm seeing the same issue with bq_table_upload

I believe in my case it was an issue with gcloud Version 270.0.0

A restore to 269.0.0 or an update to 271.0.0 (current latest version) fixed the issue for me

[ChrisJohnsonUMG]

My mistake it wasn't just a gcloud version issue

The fix for me was to add fields=as_bq_fields(dataset_name) to the bq_table_upload

[EdwinTh]

Just to confirm, we have experienced the same error and indeed as_bq_fields(dataset_name) resolves it.

[david-jankoski]

i'm strangely hitting either the same error as OP (Error: Invalid JSON payload received... ) or when trying the proposed workarounds Error: Unsupported type: list.
The error comes from the as_bq_fields() function.
I tried upgrading to the latest version of the package as well as updating gcloud components but did not help.
I tried with this old example issue

library("bigrquery")

z <- tibble::tibble(
  x = 1:3,
  y = list(1, 1:2, 1:3)
)

as_bq_fields(z)
#> Error: Unsupported type: list

^{Created on 2019-12-05 by the reprex package (v0.3.0)}

Session info

devtools::session_info()
#> ─ Session info ───────────────────────────────────────────────────────────────
#>  setting  value                       
#>  version  R version 3.6.1 (2019-07-05)
#>  os       macOS Mojave 10.14.6        
#>  system   x86_64, darwin18.6.0        
#>  ui       unknown                     
#>  language (EN)                        
#>  collate  en_US.UTF-8                 
#>  ctype    en_US.UTF-8                 
#>  tz       Europe/Amsterdam            
#>  date     2019-12-05                  
#> 
#> ─ Packages ───────────────────────────────────────────────────────────────────
#>  package     * version      date       lib source                           
#>  assertthat    0.2.1        2019-03-21 [1] CRAN (R 3.6.0)                   
#>  backports     1.1.5        2019-10-02 [1] CRAN (R 3.6.1)                   
#>  bigrquery   * 1.2.0.9000   2019-12-05 [1] Github (r-dbi/bigrquery@5f43228) 
#>  bit           1.1-14       2018-05-29 [1] CRAN (R 3.6.0)                   
#>  bit64         0.9-7        2017-05-08 [1] CRAN (R 3.6.0)                   
#>  callr         3.3.2        2019-09-22 [1] CRAN (R 3.6.1)                   
#>  cli           1.1.0        2019-03-19 [1] CRAN (R 3.6.0)                   
#>  crayon        1.3.4        2017-09-16 [1] CRAN (R 3.6.0)                   
#>  DBI           1.0.0        2018-05-02 [1] CRAN (R 3.6.0)                   
#>  desc          1.2.0        2018-05-01 [1] CRAN (R 3.6.0)                   
#>  devtools      2.0.2        2019-04-08 [1] CRAN (R 3.6.0)                   
#>  digest        0.6.23       2019-11-23 [1] CRAN (R 3.6.1)                   
#>  evaluate      0.14         2019-05-28 [1] CRAN (R 3.6.1)                   
#>  fs            1.3.1        2019-05-06 [1] CRAN (R 3.6.0)                   
#>  glue          1.3.1        2019-03-12 [1] CRAN (R 3.6.0)                   
#>  highr         0.8          2019-03-20 [1] CRAN (R 3.6.0)                   
#>  htmltools     0.4.0        2019-10-04 [1] CRAN (R 3.6.1)                   
#>  httr          1.4.1.9000   2019-12-05 [1] Github (r-lib/httr@844c8c7)      
#>  jsonlite      1.6          2018-12-07 [1] CRAN (R 3.6.0)                   
#>  knitr         1.26         2019-11-12 [1] CRAN (R 3.6.1)                   
#>  magrittr      1.5          2014-11-22 [1] CRAN (R 3.6.0)                   
#>  memoise       1.1.0.9000   2019-11-10 [1] Github (hadley/memoise@d7782b1)  
#>  pillar        1.4.2.9001   2019-11-20 [1] Github (r-lib/pillar@82370d7)    
#>  pkgbuild      1.0.6        2019-10-09 [1] CRAN (R 3.6.1)                   
#>  pkgconfig     2.0.3        2019-09-22 [1] CRAN (R 3.6.1)                   
#>  pkgload       1.0.2        2018-10-29 [1] CRAN (R 3.6.0)                   
#>  prettyunits   1.0.2        2015-07-13 [1] CRAN (R 3.6.0)                   
#>  processx      3.4.1        2019-07-18 [1] CRAN (R 3.6.1)                   
#>  ps            1.3.0        2018-12-21 [1] CRAN (R 3.6.0)                   
#>  R6            2.4.1        2019-11-12 [1] CRAN (R 3.6.1)                   
#>  Rcpp          1.0.3        2019-11-08 [1] CRAN (R 3.6.1)                   
#>  remotes       2.1.0.9000   2019-07-02 [1] Github (r-lib/remotes@b8213b4)   
#>  rlang         0.4.2.9000   2019-12-03 [1] Github (r-lib/rlang@1be25e7)     
#>  rmarkdown     1.12         2019-03-14 [1] CRAN (R 3.6.0)                   
#>  rprojroot     1.3-2        2018-01-03 [1] CRAN (R 3.6.0)                   
#>  sessioninfo   1.1.1        2018-11-05 [1] CRAN (R 3.6.0)                   
#>  stringi       1.4.3        2019-03-12 [1] CRAN (R 3.6.0)                   
#>  stringr       1.4.0        2019-02-10 [1] CRAN (R 3.6.0)                   
#>  testthat      2.3.1        2019-12-01 [1] CRAN (R 3.6.1)                   
#>  tibble        2.99.99.9010 2019-12-04 [1] Github (tidyverse/tibble@f4365f7)
#>  usethis       1.5.0        2019-04-07 [1] CRAN (R 3.6.0)                   
#>  vctrs         0.2.0.9007   2019-12-03 [1] Github (r-lib/vctrs@311410e)     
#>  withr         2.1.2        2018-03-15 [1] CRAN (R 3.6.0)                   
#>  xfun          0.11         2019-11-12 [1] CRAN (R 3.6.1)                   
#>  yaml          2.2.0        2018-07-25 [1] CRAN (R 3.6.0)                   
#> 
#> [1] /usr/local/lib/R/3.6/site-library
#> [2] /usr/local/Cellar/r/3.6.1_1/lib/R/library

[electricstorm]

My mistake it wasn't just a gcloud version issue

The fix for me was to add fields=as_bq_fields(dataset_name) to the bq_table_upload

Just dropping by to say this worked for me as well.

Weirdly, there is a different code which does the same thing, and is continuing to work without me having added this extra parameter. If anyone knows why this discrepant behavior is happening, pls share.