Memory usage of printing DArrays

[cpcloud]

I was playing around with @parallel doing the comprehension example from the recently posted videos on the Julia blog and I noticed something strange. I happened to have htop running at the time and I noticed that when I evaluated the DArray at the command line (just printing itself), the amount of memory used on my machine would increase by about the number of bytes that the array takes up in memory. Just as a sanity check I tried the same thing with a regular old Array and it works fine.

I then greped the code for "\u22ee" (the vertical dots unicode symbol) to try to find where the issue might lie in show.jl.

The show_vector method of AbstractArrays calls the DArray reshape method which may be making a copy of the chunks making up the darray in darray2.jl on line 188. I have no idea how to go about fixing this.

[quinnj]

Sorry this hasn't really been addressed @cpcloud, do you have a code example we can try and reproduce the issue here? There's been a lot of changes in the last year+, so this may not be an issue any more.

[cpcloud]

Hm been a while since I've spun up Julia, let me see if I can come up with something :)

[ViralBShah]

Cc @amitm

[malmaud]

Should this issue be closed and moved to https://github.com/JuliaParallel/DistributedArrays.jl, or is it something that needs to be fixed in Base?

[ViralBShah]

Yes, this should be moved there - and have a specific issue opened here if there is a dependency on fixing something in Base.

[malmaud]

OK, moved.

[jakebolewski]

This uses so much memory because of the way we do array printing in Base. The way we print matrices could be made more efficient.

[jakebolewski]

It would be useful to see if #13598 helps this situation or not.

[amitmurthy]

DArrays are out of Base now.