Strange rotation & tiling image display

[timholy]

There must be something about the display framework that I'm missing, because with the new version of Images I'm stumped by a strange IJulia bug. Here's the summary:

• For both the current master and v0.3.3, writemime for image/png produces a binary blob that, when written to a file, seems fine when viewed with an image file viewer (gwenview) or via IJulia with a "bare" wrapper
• For the new version of Images, the same binary blob viewed as a direct output of writemime for an Image type causes a strange rotation and tiling of pixels.

Steps to test:

• Pkg.test("Images"). The purpose of this step is simply to download a small test image, deposited in /tmp/Images/rose.png. If you've done this once, you don't have to do it again.
• Define the following raw image container:

type BareImage
    buf::Vector{Uint8}
end
function BareImage(filename::String)
    str = open(filename) do file
        readall(file)
    end
    BareImage(str.data)
end

Base.mimewritable(::MIME"image/png", ::BareImage) = true
Base.writemime(stream::IO, ::MIME"image/png", b::BareImage) = write(stream, b.buf)

saved to a file bareimage.jl.

• In Images, let's make a change so that we store a record of the output of writemime: right before this line, add the following:

    @show typeof(blob)
    open("/tmp/rosei.png", "w") do file
        write(file, blob)
    end

• Launch IJulia from within the /tmp/Images folder. Run the commands in this screenshot:
  
  This already illustrates the surprising result. First, from the typeof(blob) = Array{Uint8,1} and the creation of the /tmp/rosei.png file, it's clear the correct version of writemime is being called. The resulting image is rotated clockwise 90 degrees, and the individual pixels are "tiled" into a larger object and have a gap of white space between them. But only when viewed as the direct output of Images.writemime; viewing the same file as a BareImage produces the expected results.
• For comparison, within the Images folder do a git checkout v0.3.3. Make the same changes to src/io.jl that you did above, although here I named it rosei_old.png. Restart the IJulia kernel and load the same commands. Here's what I get:
  

Viewing both rosei.png and rosei_old.png with gwenview produces the same image. ImageMagick's compare tool reveals no differences, and identify -verbose does not reveal any differences other than the datestamp.

I confess to find this quite mysterious. FYI I'm running a recent master of 0.4. However, this same bug has been reported by another user JuliaImages/Images.jl#181 (comment); I've reproduced his result on two machines, one running firefox & julia 0.3. (There you can't see the tiles because for a larger image they get squished together, but it seems to be the same bug.)

[timholy]

Hah! After having worked on this for quite some time, seconds after hitting "submit" I finally figured out where the problem is coming from: this writemime method in Color. If I comment that out, display works as expected. The key difference is that the old version of Images loaded that png as a 3-by-70-by-46 array of Uint8 and so didn't invoke the method in Color, whereas the new Images loads it as a 70-by-46 array of RGB{Ufixed8}.

If I insert printlns, what's clearly happening is that both variants of writemime are being called for the same object. (Color's gets called first, then Images'.) That's a little weird. Why doesn't it pick just one?

CC @dcjones.

[timholy]

Also I should add that for an image, Color's "svg swatches" output causes a massive performance problem. (See JuliaImages/Images.jl#181 (comment).) Clearly we need the variant in Images to be preferred.

[stevengj]

@timholy, the way the Jupyter protocol works is that kernels can send multiple representations of data to display, in the form of a dictionary from mime types to the corresponding data. This way, the front-end can decide what version to display (and a future version of the notebook front-end may allow you to switch between representations, e.g. by right-clicking).

The IJulia kernel therefore sends several representations of the output, depending on what mime types are mimewritable. It always sends text/plain, but if an object has both image/png and image/xml+svg representations then it will send both: some front-ends don't support SVG display, but when SVG is available then it is typically preferred (because it can be resolution-independent). e.g. the notebook front-end will use the SVG data in preference to PNG data.

As you note, one difficulty with SVG is that, for large/complex images, it can be much slower than PNG. In consequence, PyPlot defaults to not sending SVG, for example.

[stevengj]

One option would be to define mimewritable(::MIME"image/svg+xml", ::AbstractMatrix{RGB{Ufixed8}}) = false (etcetera) in Images, which will tell IJulia to ignore the SVG writemime method for those image types.

[stevengj]

(Another option would be to remove the AbstractMatrix{<:ColorValue} methods from writemime in the Color package. Though it's kinda nice to be able to easily view small 2d color arrays as swatches.)

[timholy]

Thanks for the explanation. Of the various options on the table, disabling via mimewritable sounds like the best choice. I suppose the other choice might be, in Color, to define that mimewritable to return true only for sufficiently small arrays.