win api and fs module: filepath too long, ntfs on windows 64

[viperscape]

rust is unable to list directory contents on mounted ntfs drive. ntfs can hold extremely long folder structures but the native win API cannot browse them. This is not always a problem, for example: cygwin's LS executable compiled for windows, or java's file modules can browse these just fine since they do not rely on the win api. linux and mac osx can browse these window server shared folders as well. Does Rust for windows rely on the win api? it's unable to list contents in a deeply nested folder.

See below example, where K is a mounted SMB share that resides on a windows server (this is compile on latest rust 32bit for windows)

use std::io::fs;
fn main () {
    let path = "K:/IT/testing/";
    let sub1 = "wejhflkjwehfhwelhfkljwehfhkwehfkjlhwekjlfhkjwehfhwejkhfuiweh nweuiyfhuih4389578347589069045uyrt98y78yrf78gyv78yr78f76345yughvuidfu89yg7845y78tyg8y9845y78gy7845hy5678346578y78ghvcx jxbzcvjsbguyfg764wtr763 85474678 69346 35/";

    let sub2 = "4375848v 9t489bu89cb n89fg89gf7g89fn89g7df98 g79r 7g9re78976re78er78 t60we6re7er6w7878w46t7834545bytbxcvu8v6r7 fye49th4ui 7878t6r78tg645t6 46uygt78g6xrt645jk6h78t7y8dxzg78e5 ht6455ytr78sr6e4tnovn47654t6/";

    let fullpath = path.to_string() + sub1.to_string() + sub2.to_string(); // fails!
    let mostpath = path.to_string() + sub1.to_string(); // works!

    let path1 = fs::readdir(&Path::new(fullpath.to_string())).unwrap(); //fails with fullpath!

    for src in path1.iter() {
        println!("{}",src.filename_str().unwrap());
    }
}

cargo run --verbose
       Fresh rust_test v0.0.1 (file:///C:/Users/chris/rust_test)
     Running `..\target\rust_test`
task '<main>' panicked at 'called `Result::unwrap()` on an `Err` value: couldn't read directory (OS Error 3: The system cannot find the path specified.
; path=K:\IT\testing\wejhflkjwehfhwelhfkljwehfhkwehfkjlhwekjlfhkjwehfhwejkhfuiweh nweuiyfhuih4389578347589069045uyrt98y78yrf78gyv78yr78f76345yughvuidfu89yg7845y78tyg8y9845y78gy7845hy5678346578y78ghvcx jxbzcvjsbguyfg764wtr763 85474678 69346 35\4375848v 9t489bu89cb n89fg89gf7g89fn89g7df98 g79r 7g9re78976re78er78 t60we6re7er6w7878w46t7834545bytbxcvu8v6r7 fye49th4ui 7878t6r78tg645t6 46uygt78g6xrt645jk6h78t7y8dxzg78e5 ht6455ytr78sr6e4tnovn47654t6)', C:\bot\slave\nightly-win-32\build\src\libcore\result.rs:743
An unknown error occurred
Process didn't exit successfully: `..\target\rust_test` (status=101)

[vharavy]

WinAPI does support very long file names, but one need to use Unicode variants of file functions (that ends with W) add append \\?\ prefix to all such names. See, for example, documentation on MSDN for CreateFile.

[viperscape]

Thank you for the tip. I was able to solve this by changing all the fwd slashes to escaped backslashes, as well as providing the prefix (escaped as well). I could have also done a raw string: r("\\?\...") however fwd slashes did not work, even though the typical windows api can take either; as well my editor does not know anything about raw strings. Thanks again. Does it make sense to include this by default in Rust for windows (unc conversion for win fs unicode api compatibility)? This issue could be closed, though I am interested in hearing the last question answered by someone, if possible

    let path = "\\\\?\\K:\\IT\\testing\\";
    let sub1 = "wejhflkjwehfhwelhfkljwehfhkwehfkjlhwekjlfhkjwehfhwejkhfuiweh nweuiyfhuih4389578347589069045uyrt98y78yrf78gyv78yr78f76345yughvuidfu89yg7845y78tyg8y9845y78gy7845hy5678346578y78ghvcx jxbzcvjsbguyfg764wtr763 85474678 69346 35\\";

    let sub2 = "4375848v 9t489bu89cb n89fg89gf7g89fn89g7df98 g79r 7g9re78976re78er78 t60we6re7er6w7878w46t7834545bytbxcvu8v6r7 fye49th4ui 7878t6r78tg645t6 46uygt78g6xrt645jk6h78t7y8dxzg78e5 ht6455ytr78sr6e4tnovn47654t6\\";

[nodakai]

There is an ongoing effort to overhaul the path module: rust-lang/rfcs#474 Let's talk to the core team to make sure the new version can correctly handle a path longer than MAX_PATH on Windows.

[retep998]

Using \\?\ prefixed paths means that the API does not perform any normalization of paths whatsoever. This means that things like .. are treated literally and that / isn't automatically translated to \.
Also how do Cygwin or Java avoid using WinAPI? I'd be really interested in seeing what method they use to avoid WinAPI because that is the backbone of everything on Windows.

[steveklabnik]

Path reform has now landed, but I'm not sure if this was addresed or not. @alexcrichton ?

[retep998]

Unless you explicitly use verbatim paths then this issue still exists. We'd need to do magic behind the scenes to convert non-verbatim paths to verbatim paths in order to really resolve this issue.

[alexcrichton]

While proposed in #27916, the standard library is pretty unlikely to transform paths by default. That being said all the functions we have today can consume verbatim paths, so I'm going to close this as basically "not much to do here". There may be a few points in the compiler which need to be updated, but follow-up issues can be opened for those.