'No such file or directory' error when path contains Chinese characters

[azurefx]

using Images
using FileIO
img=load("input.png")
println(img[1,1])

C:\>cd English

C:\English>julia transparent_png.jl
RGBA{N0f8}(0.0,1.0,0.0,1.0)

C:\English>cd ..

C:\>cd 中文

C:\中文>julia transparent_png.jl
Error encountered while loading "C:\\中文\\input.png".
Fatal error:
ERROR: LoadError: UnableToOpenBlob `C:\中文\input.png': No such file or directory @ error/blob.c/OpenBlob/2873
Stacktrace:
 [1] handle_error(::ErrorException, ::File{DataFormat{:PNG}}) at C:\Users\Azure\.julia\packages\FileIO\Y0Lre\src\error_handling.jl:80
 [2] handle_exceptions(::Array{Any,1}, ::String) at C:\Users\Azure\.julia\packages\FileIO\Y0Lre\src\error_handling.jl:75
 [3] #load#27(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::File{DataFormat{:PNG}}) at C:\Users\Azure\.julia\packages\FileIO\Y0Lre\src\loadsave.jl:193
 [4] load(::File{DataFormat{:PNG}}) at C:\Users\Azure\.julia\packages\FileIO\Y0Lre\src\loadsave.jl:172
 [5] #load#13(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::String) at C:\Users\Azure\.julia\packages\FileIO\Y0Lre\src\loadsave.jl:118
 [6] load(::String) at C:\Users\Azure\.julia\packages\FileIO\Y0Lre\src\loadsave.jl:118
 [7] top-level scope at none:0
 [8] include at .\boot.jl:317 [inlined]
 [9] include_relative(::Module, ::String) at .\loading.jl:1038
 [10] include(::Module, ::String) at .\sysimg.jl:29
 [11] exec_options(::Base.JLOptions) at .\client.jl:229
 [12] _start() at .\client.jl:421
in expression starting at C:\中文\transparent_png.jl:3

[azurefx]

FYI The system codepage for non-Unicode programs is 936(GBK)

julia> Sys.windows_version()
v"10.0.17134"

julia> versioninfo()
Julia Version 1.0.0
Commit 5d4eaca0c9 (2018-08-08 20:58 UTC)
Platform Info:
  OS: Windows (x86_64-w64-mingw32)
  CPU: Intel(R) Core(TM) i7-4712MQ CPU @ 2.30GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-6.0.0 (ORCJIT, haswell)

[timholy]

Is that first line in your top post the contents of transparent_png.jl? What happens if you change it to println(isfile("input.png"))?

My main concern is that it looks like the error may be being thrown by the ImageMagick C library rather than Julia. If so, it may be an error in the library rather than in any of the Julia code. You might want to try some of the IM command-line tools, if possible, and see if they work. If so, then I'm afraid I'm going to have to ask you to debug this yourself, I don't have any idea how to go about setting up my machine to have Chinese-character directory names. (ImageMagick.jl is not a big or complicated package, so hopefully you'll find this fairly straightforward.)

[azurefx]

Is that first line in your top post the contents of transparent_png.jl?
Yes.

What happens if you change it to println(isfile("input.png"))?
Both true for English and Chinese dir names. In fact, All Julia file functions works properly.

Thanks for your reply and suggestions. I'm going to debug on my own first.

[azurefx]

I've figured out that this problem is caused by ImageMagick's MagickWand C interface. The MagickReadImage function accepts a char* filename parameter, and unfortunately the CRT on Windows treats filename as MBCS encoding (The active codepage for non-Unicode programs, which is CP936 (GBK) on Simplified Chinese Windows installations). Since GBK and UTF-8 are both compatible with ASCII, this function only works on full ASCII path names, and other characters become gibberish.

(In libmagickwand.jl, pkg ImageMagick)

using StringEncodings

function readimage(wand::MagickWand, filename::AbstractString)
    filename = encode(filename, "GBK")
    status = ccall((:MagickReadImage, libwand), Cint, (Ptr{Cvoid}, Ptr{UInt8}), wand, filename)
    status == 0 && error(wand)
    nothing
end

If we convert the string to MBCS codepage explicitly, most Chinese path names can be recognized correctly. However, it still fails when path contains non-GBK characters (Japanese, Korean, etc.).

A better approach is using Julia's IO facilities to bypass the CRT's file functions:

using Images
using FileIO
io=open("img1.jpg","r")
img=load(io)
show(img)
close(io)

I think some functions need to be re-designed to work with encodings correctly, but I can't come up with a good solution. Should we wrap those functions which accept filename with Julia IO or convert the string encoding to make IM happy?

@timholy

[timholy]

Nice work! You clearly know a lot more about this than I do, so the guidance I can offer is limited. Others should feel free to chime in here.

Naively, I'd suggest trying the JuliaIO route first and see if tests pass; if that works, also try loading images from TestImages.jl. If those pass, perhaps that's the way to go.

[azurefx]

Nice work! You clearly know a lot more about this than I do, so the guidance I can offer is limited. Others should feel free to chime in here.

Naively, I'd suggest trying the JuliaIO route first and see if tests pass; if that works, also try loading images from TestImages.jl. If those pass, perhaps that's the way to go.

Fixed in my PR.

#using fixed ImageMagick.jl
using TestImages
using FileIO
dir = joinpath(tempdir(), "中文路径")
if !isdir(dir)
    mkdir(dir)
end
filenames = TestImages.remotefiles
for fn in filenames
    imgpath = joinpath(dir, fn)
    cp(joinpath(TestImages.imagedir, fn), imgpath;force = true)
    img = load(imgpath)
    println(size(img))
end

Test images also pass.

However I encountered a new problem with IM...
JuliaIO/ImageMagick.jl#145
@timholy

[fonsp]

It looks like this is still an issue, this post suggest that Images.jl cannot be used by people with a Chinese username on windows :(

https://stackoverflow.com/questions/79032378/big-problems-caused-by-chinese-usernames-in-julia