openCV BGR format

[kmsquire]

I'm reading some video files through PyCall -> OpenCV. By default, OpenCV provides a column-major, h x w x d Uint8 BGR image (why!?). For the project I'm working on, I've also found it useful to use BGRA.

It's unclear to me where/how I should handle this format. Should I simply deal with it in my own code? Create a PyOpenCV package (competing with the OpenCV work in GSoC)?

Would you want to handle the format here? Is there a reason RGB8 is defined here and not in Color.jl?

I'd love to hear your thoughts. Cheers, Kevin

Cc: @dcjones

[dcjones]

I'd be open to defining types in Color.jl for different RGB memory layouts. We already have RGB24.

[ihnorton]

(I don't believe there is any explicit OpenCV project among the GSoC-funded projects, unfortunately)

[kmsquire]

@dcjones, thanks. I'll probably add them there.

I'll have to look at how Images handles this currently, but the image format isn't packed--the images have one plane of blue, one of red, and one of green. So it might make more sense to keep most things here in Images.jl.

(@ihnorton, you're right. Considering there were so many, I thought one of them had gotten through.)

[timholy]

Handling it Images makes sense. The general philosophy of the representation in Images is "no ambiguity." As long as you set

img["colordim"] = 3
img["colorspace"] = "BGR"
img["spatialorder"] = ["y", "x"]

then the basics of the job are done. Of course, presumably you'd want to be able to display these images, etc, and for that a little bit of support for "BGR" (e.g., in uint32color) might be needed.

[lucasb-eyer]

h x w x d Uint8 BGR image (why!?).

In C world, h x w x d is contiguous in memory. BGR is the standard order for bitmaps in windows, the reason of that is well explained in this SO answer.

[timholy]

Right, but it's a little odd that it's not stored in color-fastest order. You naturally get BGRA out of a statement equivalent to reinterpret(Uint8, img, (4,size(img,1),size(img,2))) where img is an Array{Uint32,2}, but then @kmsquire finds that it is reordered to make the color dimension slowest. Most image file formats also store in "horizontal-major" order, another sign this has been reordered internally.

OT, but @kmsquire, I take it your experiments with a Julia libav wrapper are not yet working? I plan to get back to that someday myself, but I haven't yet needed video. Also, given that a Julia wrapper of OpenCV is not in GSoC, perhaps we should write a glue package for PyOpenCV.

[lucasb-eyer]

"Color-fastest order" is what I actually meant when I said "contiguous in memory," I was missing the right words. In C (and default numpy) land, the last coordinate iterates fastest, and thus h x w x d really is color-fastest there and it's the right thing to do for both OpenCV and its Python wrapper.

But when the array reaches Julia, the reordering should be done by PyCall, so that it's d x w x h in Julia, which is color-fastest order in Julia.

Re-reading the OP, I think @kmsquire doesn't actually have a problem with the dimension ordering, but only with the color ordering, right?

[timholy]

@lucasb-eyer, thanks for figuring this out. It would be far better if Julia could just get a pointer to the same buffer available to Python, and work with it directly. I really haven't used PyCall, is there a way to avoid the permutedims?

[kmsquire]

Thanks all, for the explanations.

Re-reading the OP, I think @kmsquire doesn't actually have a problem with the dimension ordering, but only with the color ordering, right?

@lucasb-eyer, both, actually, but the BGR part did throw me for a loop when I first started using OpenCV.

But when the array reaches Julia, the reordering should be done by PyCall, so that it's d x w x h in Julia, which is color-fastest order in Julia.

Tim just posted, but in fact, the format in memory is already d x w x h (from Julia's perspective), so it would be better (here) if PyCall didn't do the transpose.

@timholy, I've gotten libav to the point of initialization, but it does a bunch of memory allocation and deallocation, and I haven't gotten that part working yet. There are also a few circular references in the data structures, and Clang doesn't order them properly yet, so I've been hacking on Clang a little to deal with these issues, to try to make it easier to regenerate the interface. I do plan to get back to libav soon, though.

On the other hand, an OpenCV wrapper would still be useful, if somewhat limiting. There are a lot of useful algorithms there that aren't available in Julia yet. That said, I've found using it from Python a little tedious: many of the algorithms I'm using require that the data be float32, and I'm finding I have to convert things all over the place. Plus, it's a bit more verbose than Julia. But it is faster/easier than C++.

[timholy]

Re OpenCV wrapper, yes. I've also thought it would be good for benchmarking, to provide a standard to measure our own pure-Julia progress against. Clearly, one could write that in Python, so this isn't a reason by itself to provide a Julia wrapper.

[lucasb-eyer]

You can avoid PyCall transposing by calling the function through pycall and specifying PyArray as return-value. It's a bit cludgy in practice, though, because you need to

np = pyimport(:numpy)
a = pycall(np[:random][:rand], PyArray, 5, 6)
# a is now a not-reordered PyArray, subtype of AbstractArray

above code is untested, from memory.

Edit: I also think an OpenCV-wrapper would be immediately useful in getting more productive with Julia, but it might hinder Image.jl's longer-term progress, e.g. I wouldn't have looked at Image.jl much if there was an OpenCV.jl.

[glennsweeney]

I can't comment on much of the above, but I wanted to chime in about OpenCV support in Julia.
I'm excited to see Julia evolving a good image processing library separate from OpenCV, and I personally want to see how far that development can go; I think that OpenCV (though an excellent project) doesn't need to be the "end-all, be-all" of image processing.

[timholy]

Glad to hear there's so much support for native solutions! I have to say, even though wrapping a C library is easy in Julia (C++ is of course different), distributing code that relies on C is a pain in the neck, especially when multiple platforms are involved. I've gone through more hassle from ImageMagick than any other part of Images.jl. So, these kinds of messages are nice to get.

[glennsweeney]

I don't want to take this issue too far away from the original purpose, but I think that your point is very poignant. Almost all of image processing is still built on foundations of C/C++. There is still a huge use for these libraries, especially because many are heavily optimized. However, for most scientific imaging cases, "real-time" is not normally vital. For this reason I think moving away from ImageMagick gradually by adding Julia-native support is a good thing, if "redundant" from some perspectives.

But, OpenCV isn't going anywhere, and so getting Julia/Images to play will will be beneficial for everyone, I think.

[timholy]

This is fixed on #172, which I'm hoping will be merged soon.