Binary output proposal #237
Conversation
…still output in 2d, however
|
@donnaaboise: Since you reverted to writing out the ghost cells, why not just revert to the original versions of But more generally we should resolve the issue of how to handle ghost cells following the discussion in #236 and be consistent in 2d and 3d. If we decide not to write ghost cells in 3d, or allow the user to specify how many ghost cells to include, then there has to be a corresponding change made to Finally, I would avoid using the term "slice" when talking about the interior grid cells without ghost cells. This doesn't correspond to what I think of as a slice of the data, and conflicts with the way @cjvogl and I are using the term in another branch that I've started using again for 3d output (and that we should try to clean up and merge in soon). This version has a |
|
I could have left the original 2d The main reason I see for not including the ghost cells is that they take more storage and takes longer to output the data. This is particularly noticable in 3d. In 2d, it makes less of a difference, and in any case, it sounds like the 2d Python graphics expects ghost cells, so may not be worth stripping them before outputting data. I used the term "slice" only because that is the term I use in the Matlab graphics. I have no particular attachment to that term, and can change it. In any case, the Matlab graphics can read the 2d/3d binary data with or without ghost cells. Currently I have hardwired the selection, but can easily make this a user choice, depending on what In any case, I would think it is okay if 2d wrote out ghost cells (since, at the very least Python graphics depends on them), but the 3d doesn't. In 3d, there is a big performance hit, and there is no corresponding plotting routines that depends on ghost cells. |
|
Just a point of clarification about reading binary files and plotting with visclaw: The reading is done by pyclaw.fileio.binary, which can read in 1d, 2d, or 3d data. It currently always assumes ghost cells are included (though this could be changed pretty easily). Ghost cells are stripped off in creating the pyclaw.solution.Solution object that is returned. The 2d plotting routines in visclaw use this stripped down version. So it is not the plotting that depends on the ghost cells being in the binary files, but the fileio.binary routines, which already handle 3d (you can read in the solution and do your own thing with it) even though we don't have Python plotting routines in visclaw for 3d yet. So I suggest that whatever we do should be consistent between 2d and 3d. (And 1d, which also supports binary in the same way.) |
|
Thanks @rjleveque for clarification - I had been assuming that Visclaw didn't read 3d files at all. I still think it might be worth considering the option of not printing out ghost cells, especially in 3d. IN In the one example I have looked at, the ratio is of timing/storage without verses with ghost cells is about 60%. Timing (without ghost cells) Storage : (without ghost cells) Timing (with ghost cells) Storage (with ghost cells) So while the timing for the output isn't that much (0.091s vs. 0.153s, or about 59%), the storage is significant (23M vs. 36M, or about 63%). Regardless of what AMRClaw decides, I'll go ahead and make an option in Matlab so the user can decide whether or not to read in ghost cells. |
|
The timing results for the 2d swirl example show that the timing is about 55% faster when not writing out ghost cells (0.052s vs. 0.096s) and 80% for storage (5.0Mb vs. 6.2Mb). |
|
It seems misleading to say it is faster by such a huge margin when ghost cells aren't included since you're only looking at the valout time, which is a tiny fraction of the total run time. In fact for the 3d example you show, the total time without ghost cells was greater than the total time with ghost cells by a couple seconds, so the uncertainty in the timings is much greater than the total time spent in valout. But I agree the storage saving is considerable and as long as it doesn't significantly slow down valout to strip out the ghost cells, I think it's great to include this as an option. But we do need to modify |
|
In this example (the 3d advection example), the time in As a second data point, I did a run with Another consideration is that I am running this on my i7 laptop, with a fast SSD hard drive. On other machines, I/O might be slower. The storage savings show about the same savings as in the first set of simulations - 1.016Gb vs. 1.6Gb, or about 63%. Timing (without ghost) Storage (without ghost) Timing (with ghost cells) Storage (with ghost cells) |
|
And as a final data point, here are the timing results using the The binary output is about 30x faster than the ascii, and there is about a 3:1 compression ratio using the binary (2.8GB vs 1.016GB). Timing (without ghost) Storage (without ghost cells) |
|
Leave |
|
Let's discuss further at SIAM CSE and figure out how best to handle. |
I've included an option in the 2d/3d valout.f90 files that strips ghost cells from binary output. Seems to be significantly faster than writing with ghost cells.