SharpClaw with TVD limiting gives different results in serial and parallel

[ketch]

@hadjimy and I have just verified that running the 2D acoustics example in serial and parallel gives different answers (relative error larger than 1e-3) when solver_type=sharpclaw and lim_type=1 (TVD limiting). When using WENO limiting (lim_type=2), the serial and parallel solutions agree to high accuracy.

We verified that running with MPI and PETSc but only one process gives results that match the serial results.

The differences seem also to depend on how many processors are used. My first guess is that somehow the TVD limiting is not behaving correctly in the ghost cells, though I don't see anything obvious in the code.

This bug exists already in master, but we don't have a test that triggers it. It is responsible for the failing test in #483.

[ketch]

h/t to @hadjimy for discovering this.

[ketch]

One additional observation: the results from parallel runs of SharpClaw with TVD limiting seem to be non-deterministic; the values vary by several percent from one run to the next.