Data hashes seem to differ between Windows and Linux

[florisvdh]

This issue uses the reprex from issue #47 .

While not getting those errors, my output - in Linux - is always as:

3e6fbe383532f4312bd0f5c9f30976f64d00e9cc e5e6ed33018f669308297f2f3d66512b3fa8c1b6 
                     "../data/df_vc.tsv"                      "../data/df_vc.yml" 

Which is a different data_hash (stored in the yml file) than the Windows-generated one.

Session Info

R version 3.6.1 (2019-07-05) Platform: x86_64-pc-linux-gnu (64-bit) Running under: Linux Mint 18.1

Matrix products: default
BLAS: /usr/lib/libblas/libblas.so.3.6.0
LAPACK: /usr/lib/lapack/liblapack.so.3.6.0

locale:
[1] LC_CTYPE=nl_BE.UTF-8 LC_NUMERIC=C LC_TIME=nl_BE.UTF-8
[4] LC_COLLATE=nl_BE.UTF-8 LC_MONETARY=nl_BE.UTF-8 LC_MESSAGES=nl_BE.UTF-8
[7] LC_PAPER=nl_BE.UTF-8 LC_NAME=C LC_ADDRESS=C
[10] LC_TELEPHONE=C LC_MEASUREMENT=nl_BE.UTF-8 LC_IDENTIFICATION=C

attached base packages:
[1] stats graphics grDevices utils datasets methods base

other attached packages:
[1] git2rdata_0.1

loaded via a namespace (and not attached):
[1] drat_0.1.5 compiler_3.6.1 assertthat_0.2.1 tools_3.6.1 yaml_2.2.0
[6] git2r_0.26.1 packrat_0.5.0 fortunes_1.5-4

[ThierryO]

I can reproduce this. It seems like git2r::hashfile() yields a different output under Linux and Windows

filename <- tempfile("os-bug")
writeLines(
  c("x\ty", "1\t1", "2\t2", "3\t3", "4\t4", "5\t5", "6\t6", "7\t7", 
    "8\t8", "9\t9", "10\t10", "11\t11", "12\t12", "13\t13", "14\t14", 
    "15\t15", "16\t16", "17\t17", "18\t18", "19\t19", "20\t20", "21\t21", 
    "22\t22", "23\t23", "24\t24", "25\t25", "26\t26"),
  filename
)
git2r::hashfile(filename)

Output:

• Windows: 1de50dce6d5139f98a8e69d4d45d26ae7d32c64f
• Linux: 3e6fbe383532f4312bd0f5c9f30976f64d00e9cc

Session info on Windows

─ Session info ──────────────────────────────────────────────────────────────────────────────────
 setting  value                       
 version  R version 3.5.2 (2018-12-20)
 os       Windows >= 8 x64            
 system   x86_64, mingw32             
 ui       RStudio                     
 language (EN)                        
 collate  Dutch_Belgium.1252          
 ctype    Dutch_Belgium.1252          
 tz       Europe/Paris                
 date     2019-08-14                  

- Packages ---------------------------------------------------------------------------------------------
 package     * version date       lib source        
 assertthat    0.2.1   2019-03-21 [1] CRAN (R 3.5.3)
 cli           1.1.0   2019-03-19 [1] CRAN (R 3.5.3)
 crayon        1.3.4   2017-09-16 [1] CRAN (R 3.5.3)
 drat          0.1.4   2017-12-16 [1] CRAN (R 3.5.3)
 fortunes      1.5-4   2016-12-29 [1] CRAN (R 3.5.2)
 git2r       * 0.25.2  2019-03-19 [1] CRAN (R 3.5.3)
 rstudioapi    0.10    2019-03-19 [1] CRAN (R 3.5.3)
 sessioninfo   1.1.1   2018-11-05 [1] CRAN (R 3.5.3)
 withr         2.1.2   2018-03-15 [1] CRAN (R 3.5.3)
 yaml          2.2.0   2018-07-25 [1] CRAN (R 3.5.2)

[1] C:/R/library
[2] C:/Program Files/R/R-3.5.2/library

Session info on Linux

─ Session info ──────────────────────────────────────────────────────────────────────────────────
 setting  value                       
 version  R version 3.6.1 (2019-07-05)
 os       Ubuntu 18.04.3 LTS          
 system   x86_64, linux-gnu           
 ui       RStudio                     
 language nl:en                       
 collate  nl_NL.UTF-8                 
 ctype    nl_NL.UTF-8                 
 tz       Europe/Brussels             
 date     2019-08-14                  

─ Packages ──────────────────────────────────────────────────────────────────────────────────────
 package     * version date       lib source        
 assertthat    0.2.1   2019-03-21 [1] CRAN (R 3.6.0)
 cli           1.1.0   2019-03-19 [2] CRAN (R 3.5.3)
 crayon        1.3.4   2017-09-16 [2] CRAN (R 3.5.3)
 drat          0.1.5   2019-03-28 [1] CRAN (R 3.6.0)
 fortunes      1.5-4   2016-12-29 [1] CRAN (R 3.6.0)
 git2r         0.26.1  2019-06-29 [1] CRAN (R 3.6.0)
 packrat       0.5.0   2018-11-14 [1] CRAN (R 3.6.0)
 rstudioapi    0.10    2019-03-19 [2] CRAN (R 3.5.3)
 sessioninfo   1.1.1   2018-11-05 [2] CRAN (R 3.5.3)
 withr         2.1.2   2018-03-15 [2] CRAN (R 3.5.3)

[1] /home/thierry_onkelinx/R/x86_64-pc-linux-gnu-library/3.5
[2] /usr/local/lib/R/site-library
[3] /usr/lib/R/site-library
[4] /usr/lib/R/library

[ThierryO]

According to @stewid, the difference in hash is due to the difference in line endings on Linux and Windows (ropensci/git2r#397).

Below is a reprex using write.table() on Linux.

library(git2r)
x <- seq(1:26)
y <- letters
df <- data.frame(x, y, stringsAsFactors = FALSE)
filename <- tempfile("os-bug")

# unix style line endings
write.table(
  x = df, file = filename, append = FALSE, quote = FALSE,
  sep = "\t", eol = "\n", na = "NA", dec = ".", row.names = FALSE,
  col.names = TRUE, fileEncoding = "UTF-8"
)
hashfile(filename) # "50aabdcd96bd742fdcc41edcc6b3efdf8e63f498"

# windows style line endings
write.table(
  x = df, file = filename, append = FALSE, quote = FALSE,
  sep = "\t", eol = "\r\n", na = "NA", dec = ".", row.names = FALSE,
  col.names = TRUE, fileEncoding = "UTF-8"
)
hashfile(filename) # "1783ed10fa5035a3963abf4202f42fe6ca88f046"

[ThierryO]

@florisvdh and @w-jan can you check if PR #53 solves this issue? use remotes::install_github("ropensci/git2rdata@datahash")

[florisvdh]

Didn't check Windows yet, but in Linux I now get a different hash than before, is this expected?

library(git2rdata)
x <- seq(1:26)
y <- letters
df <- data.frame(x,y)
write_vc(df, "df_vc", sorting = c("x"), strict =  FALSE)
# b2658819ed189ec4496b4b25c55404f7d0918b6a 3514e919bcca45b232268c650a04db36a18aa6b5
#                              "df_vc.tsv"                              #"df_vc.yml"

[ThierryO]

Yes. This is possible. The hashes are now calculated based on the content instead of the file.

[florisvdh]

I checked in Windows and the same datahash is produced. Good work! I think it's OK to close the issue.

See some further comments in PR #53 .