realize_ does not work if the path points toward a network drive

[courtiol]

Dear fs developers,
the function realize_ does not work in my working environment if the path points toward a network drive:

fs:::realize_('M:/')
Error: [EPERM] Failed to realize 'M:/': operation not permitted

Yet, I can easily write or read file on this drive from R using usual functions.
I figured this out since it makes usethis::use_data() crash.

I cannot produce a reproductible example unless you are willing to visit me in my working place in Berlin... If that is an option, let me know!

++
Alex

[TylerGrantSmith]

I am also having the same issue.

[jimhester]

Hmm, I cannot reproduce this, with a SMB mapped network drive from my macOS machine to my windows machine I get the following.

fs::path_real("Y:/")
#> //192.168.109/jhester

Maybe there is an issue if you don't have permissions to access directories higher up in the path? Could you explain the situation a little better?

[bjcairns]

I also have this problem, again found via usethis::use_data(), where for example I would see:

usethis::use_data(object)
#> Error: [EPERM] Failed to realize 'J:/path-to-package-dir': operation not permitted
#> In addition: Warning message:
#> In path_file(base_path) : restarting interrupted promise evaluation

At the mapped drive level I have full access, although it is mapping to my user area on an underlying networked file system to which I don't have full access. There are no symbolic links, relative elements like "..", or other oddities in my actual path string; it's just a standard sequence of directory names separated by "/". Just for comparison, running your test example on my mapped drive and on C:/, I get:

fs::path_real("J:/")
#> Error: [EPERM] Failed to realize 'J:/': operation not permitted

fs::path_real("C:/")
#> C:/

What other information would be helpful? I keep hitting this problem whenever I want to work with packages (and usethis) on this system and am keen to help.

Thanks!

[bjcairns]

Encountering this again, I have dug further. The bug appears to occur in realize_, in src/path.cc. But, I think that's not the origin of the bug. For that it appears that we can look at libuv's uv_fs_realpath:

http://docs.libuv.org/en/v1.x/fs.html#c.uv_fs_realpath

which notes the bug and concludes:

While this function can still be used, it’s not recommended if scenarios such as [subst'd drives] need to be supported.

In short, libuv has known problems with Windows network drives mapped to a drive letter. The problem seems to have been around a while and is yet unsolved, c.f. open issue libuv/issues/1877.

I also note that Node.js seems to have reverted to a previous JavaScript implementation of realpath to avoid this problem (see node/pull/7899).

I admit I don't have the particular skills to verify that it is the uv_fs_realpath bug that is definitely the culprit, but I hope this is at least helpful in finding the root cause.

[bjcairns]

I've had a go at a work-around for this - basically a fallback to path_norm() in case of error. My first attempt (referenced above) added a new argument, but I realised that a much better approach would be a global option that the user could set either forever, or dynamically using withr::with_options().

@jimhester I'm still a bit out of my depth here so can I ask your opinion on the work around in the second of the above? Bascially, it tries realize_ from libuv and on error, iff option fs.path.real.robust is TRUE, will throw an informative warning and try path_norm() instead. I also updated the help text.

[jimhester]

@bjcairs, thank you for working on this, the next time I am working on fs I can see how difficult replicating the js implementation in R would be. Although figuring out an easy way to replicate the issue would be the biggest help.

[bjcairns]

@jimhester Thank you for the reply. Yes, it doesn't look simple at all. I will keep looking for a solution - what are the requirements? Creating a test for Travis or Appveyor? Would you be happy just for it to work on a local machine (local to you)?

I tried subst at the command prompt to generate a virtual drive but this doesn't seem to throw an error. I've played about with permissions but it doesn't replicate the same error ("[EPERM] Failed to stat" not "[EPERM] Failed to realize" and I can only do it with a directory I can't write to (though I can setwd() it). It seems that the user-level Windows file permissions propagate such that it might not be possible to replicate the specific network-drive issue I and others have been finding.

[bjcairns]

For the benefit of anyone who has this issue, and pending a real solution, I've written a guide on how to (easily) install the work-around directly from my fork.

Full description is here at the bjcairns/fs wiki.

I will try to update this periodically to keep up with CRAN and (perhaps less often) the latest development version.

[Not sure if this is the best way to do this - I removed the issue references in the commit messages to avoid them popping up on this issue whenever I rebase, but I don't want to generate other inconveniences for @jimhester or other fs devs. If there is a better way, please let me know.]

[ckirkeby]

I also have problems with this. Tried the suggested work-around, as well as a number of other solutions, with no luck. Seems like my windows network drive is locked from the university side. Please let me know if anyone has another solution to this.

[bjcairns]

@ckirkeby The solution I suggested should work if base::normalizePath(path, mustWork = TRUE) works, where path is a path to the network drive.

If that function does work for you, I've enabled issues on my fork and would be happy to discuss further there (instead of here).

If normalizePath does not work, it sounds like your difficulty is not an fs issue but an R/base issure, or as you suggest a networking or filesystem issue, and I've not encountered it myself.

[bjcairns]

Just to note, the commit workflowr/workflowr@6d98dc98, which includes a function to recursively resolve symlinks on Windows with fs::path_real(), does not solve the EPERM error that I get here.

In particular, fs::file_exists() returns TRUE for paths that would otherwise trigger an error with fs::path_real(), so the code (in function workflowr:::resolve_symlink_) simply calls fs::path_real().

@jdblischak

[ckirkeby]

Thank you for the response. I am very sure this was a network related issue at the university, due to security on the private network drives. So I copied the project onto my C-drive, and it worked well.

[bollard]

EDIT: I take all (well almost all) of my previous comment back. The workaround here by @bjcairns does indeed work (thank you very much!), but crucially only the path-real-fallback-cran branch. I had previously only tried the path-real-fallback branch (seeing as it has a more recent commit), but that did not resolve the issue.

There is a little bit of hackery involved becuase as of verion 2.4.0 devtools requires fs >= 1.5.0. In lieu of the fix being merged, it would be really excellent if the latest fs master changes could be merged into path-real-fallback-cran to make it play nicer with other packages downstream. For anyone else, my quick fix was to run:

install.packages('versions')
library(versions)
install.versions('devtools', '2.3.2')

Thank you for all your help and efforts on this issue