windows encoding support

[vtjnash]

except for limited usage cases, julia code on windows uses an incorrect encoding (including the clipboard code). solution: always use utf16 format. windows provides a function to make this straightforward. see libuv/src/win/util.c uv_utf16_to_utf8 and uv_utf8_to_utf16 for usage.

(replaces #1919)

[vtjnash]

TODO list: doesn't seem to bad, actually:

• ./env.jl:_getenvlen(var::String) = ccall(:GetEnvironmentVariableA,stdcall,Uint32,(Ptr{Uint8},Ptr{Uint8},Uint32),var,C_NULL,0)
• ./env.jl: ret=ccall(:GetEnvironmentVariableA,stdcall,Uint32,(Ptr{Uint8},Ptr{Uint8},Uint32),s,val,len)
• ./env.jl: ret = ccall(:SetEnvironmentVariableA,stdcall,Int32,(Ptr{Uint8},Ptr{Uint8}),var,val)
• ./env.jl: ret = ccall(:SetEnvironmentVariableA,stdcall,Int32,(Ptr{Uint8},Ptr{Uint8}),var,C_NULL)
• ./env.jl:start(hash::EnvHash) = (hash.block = ccall(:GetEnvironmentStringsA,stdcall,Ptr{Uint8},()))
• ./env.jl: ccall(:FreeEnvironmentStringsA,stdcall,Int32,(Ptr{Uint8},),hash.block)
• ./file.jl: lentemppath = ccall(:GetTempPathA,stdcall,Uint32,(Uint32,Ptr{Uint8}),length(temppath),temppath)
• ./file.jl: uunique = ccall(:GetTempFileNameA,stdcall,Uint32,(Ptr{Uint8},Ptr{Uint8},Uint32,Ptr{Uint8}),temppath,"julia",uunique,tname)
• ./interactiveutil.jl: res = ccall((:URLDownloadToFileA,:urlmon),stdcall,Cuint,
• ./mmap.jl: mmaphandle = ccall(:CreateFileMappingA, stdcall, Ptr{Void}, (Ptr{Void}, Ptr{Void}, Cint, Cint, Cint, Ptr{Void}), shandle.handle, C_NULL, flprotect, szfile>>32, szfile&0xffffffff, C_NULL)
• ./path.jl: p = ccall((:GetFullPathNameA, "Kernel32"), stdcall,
• ./path.jl: p = ccall((:GetFullPathNameA, "Kernel32"), stdcall,
• ./dlload.c: if (!GetModuleHandleExA(GET_MODULE_HANDLE_EX_FLAG_FROM_ADDRESS | GET_MODULE_HANDLE_EX_FLAG_UNCHANGED_REFCOUNT,
• ./sys.c: int len = GetModuleFileNameA(handle,tclfile,sizeof(tclfile));
• ./file.jl: @windows_only ret = ccall(:_mkdir, Int32, (Ptr{Uint8},), bytestring(path))
• ./file.jl: @windows_only ret = ccall(:_rmdir, Int32, (Ptr{Uint8},), bytestring(path))
• ./file.jl: @windows_only p = ccall(:_tempnam, Ptr{Uint8}, (Ptr{Uint8},Ptr{Uint8}), d, "julia")
• ./file.jl: ret = ccall(:_mkdir, Int32, (Ptr{Uint8},), filename)
• ./io.c
• flisp getenv/setenv
• setlocale
• clipboard()
• ./interactiveutils.jl: res = ccall((:URLDownloadToFileA,:urlmon),stdcall,Cuint,
• main (http://sourceforge.net/apps/trac/mingw-w64/wiki/Unicode%20apps)

[vtjnash]

note, part two of this probably requires using the windows API for long filenames, where that API is easily accessed by only using normalized, absolute filenames and prefixing them with \\?\ (I think there is multiple C++/COM APIs also, but I forget the details)

[vtjnash]

the Microsoft team who write the .NET libraries also write a blog. 7 years ago, they explained that until the windows API is fixed to stop rejecting long paths that are supported the underlying NT API (but not by windows explorer), .NET won't be able to add support for long paths because it is just too broken (http://blogs.msdn.com/b/bclteam/archive/2007/02/13/long-paths-in-net-part-1-of-3-kim-hamilton.aspx)

the official solution? For now, our suggested workaround for users that encounter the MAX_PATH issue is to rearrange directories We definitely agree that users shouldn't have to handle the workarounds we're forcing -- i.e. rewriting System.IO or changing the dir structure. A few users have attempted this appealing workaround [of using DOS paths], but we foil it because we expand short paths as part of file name normalization, throwing once it exceeds MAX_PATH.

yuck. I sort of knew this, but was also hoping we could leap-frog julia past 1990-era limitations (like it does for Unicode strings), so that this would never need to be true

[StefanKarpinski]

Jameson, I'm endlessly grateful that you're willing to tackle these things. Thank you.

[vtjnash]

hrm, msys-git did it msysgit/git#122 (with a disclaimer that cmd / explorer / most windows programs won't be able to access the file, and a few API's like chdir don't support them). so maybe it is possible.