[R] Proactively remove "problems" attributes

[asfimport]

I cannot able to downgrade arrow from version 2.0.0 to 1.0.1 since arrow::install_arrow() always installs the latest version.

Steps to reproduce:

 

devtools::install_version("arrow", version = "1.0.1")
arrow::install_arrow()

 

If I skip the arrow::install_arrow(), I am not able to use the gzip compression (WARNING: Arrow Gzip is not available, try using arrow::install_arrow())

 

Environment: R version 3.6.1 (2019-07-05)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 19.10

Reporter: Vinicius Pinto
Assignee: Jonathan Keane / @jonkeane

Original Issue Attachments:

• debug-arrow.7z

PRs and other links:

• GitHub Pull Request #9092

_{Note: This issue was originally created as ARROW-10624. Please see the migration documentation for further details.}

[asfimport]

Neal Richardson / @nealrichardson:
Try setting the environment variable NOT_CRAN=true prior to install_version. That should give you a build with all of the compression libraries, so there's no need to arrow::install_arrow().

[asfimport]

Neal Richardson / @nealrichardson:
See http://arrow.apache.org/docs/r/articles/install.html for details

[asfimport]

Neal Richardson / @nealrichardson:
FTR that warning message is not coming from the arrow package: https://github.com/schnorr/starvz/blob/87e1fbc8b152e720096fc3d9b9387528c9b617f2/R/write_functions.R#L24

[asfimport]

Neal Richardson / @nealrichardson:
Out of curiosity, why do you want to downgrade?

[asfimport]

Vinicius Pinto:
Thanks @nealrichardson
I just add Sys.setenv(NOT_CRAN = "true") before the install_version and that works.

[asfimport]

Vinicius Pinto:
I am debugging an error (IOError: Couldn't deserialize thrift: TProtocolException: Exceeded size limit) that appears when I use the latest versions of arrow (2.0) and readr(1.4). A tiny starvz data frame with only 4 rows becomes a parquet file with several MB (it seems a memory leak since the final size varies with the size of other starvz data frames) and this does not happen when using older versions.

[asfimport]

Neal Richardson / @nealrichardson:
Hmm. Can you share a tiny data frame that triggers this? Sounds like something that should be fixed.

[asfimport]

Vinicius Pinto:
I prepared a "small case" (8M) that triggers this error. With smaller data frames, it works, but the parquet files are larger than expected.
Download link with the data and the commands to reproduce the issue: https://filesender.rnp.br/?s=download&token=cee89b89-3c75-49e6-b69f-db47cc2ab737
With this dataset, the total size of the parquet files increases from 8M with arrow 1.0.1 to 620M with arrow 2.0.0

 

 

[asfimport]

Neal Richardson / @nealrichardson:
Issue resolved by pull request 9092
#9092