Fix GlobAsyncRef

[devreal]

This PR is (yet another) attempt to get GlobAsyncRef into a stable state. As discussed at the F2F, value semantics are removed and no consistency guarantees are provided, i.e., a read does not necessarily yield a previously written value unless there has been a flush in between.

Asynchronous writes use a temporary value and a handle, which is used to ensure local completion upon destruction of the GlobAsyncRef object. Reads are never asynchronous as that would require speculatively starting a transfer even if the GlobAsyncRef would be used for writing only. With value semantics removed, this is now explicit by calling get().

The following use-case is covered:

dash::Array<int> arr(dash::size());

for (size_t i = 0; i < dash::size(); ++i) {
  arr.async[i] = i;
}
arr.flush(); // flush asynchronous transfers to all units
assert(arr[dash::myid()] == dash::myid()); // succeeds
assert(arr.async[dash::myid()].get() == dash::myid()); // succeeds

What does not work:

dash::Array<int> arr(dash::size());

for (size_t i = 0; i < dash::size(); ++i) {
  arr.async[i] = i;
}
assert(arr[dash::myid()] == dash::myid()); // might fail
assert(arr.async[dash::myid()].get() == dash::myid()); // might fail

The PR also fixes the semantics of dash::Matrix::flush* and dash::Array::flush* and removes push and fetch, which have not been used.

Related: #357 #358 #409
Fixes #365
Closes #371 #390

[devreal]

Update: I also removed the comparison operators to avoid hidden synchronous get() operations upon comparison. I also removed the reference-based comparison operators to avoid the following pitfall:

dash::Array<int> arr(...);
dash::fill(arr.begin(), arr.end(), 0);
if (arr.async[0] == arr.async[1]) { // always false 
  // ...
}

I noticed that GlobRef has the same behavior. IMO, reference-based comparison should be explicit on global references, i.e., arr.async[0].dart_gptr() == arr.async[1].dart_gptr(). Should we adapt the behavior of GlobRef as well?

[fmoessbauer]

This PR looks really good. The interface changes are all reasonable and required. Thanks for putting time into that.

Just one note: The move construction of GlobAsyncRef is defaulted, but there is no move-assignment operator. IMO we can default this as well. However I have not thought about possible impact, especially if the target GlobAsyncRef should be flushed before the assignment.

Regarding your question in the comments: Yes, I vote for changing the behavior of GlobRef as well.

And last but not least, we should check that all dash-apps still compile.

[fmoessbauer]

Does this renaming break anything? Should we provide an alias put()?

[devreal]

I will check. IMO, put() is closer to a C PGAS API.

[fmoessbauer]

Really cool feature. This fixes / answers #416.

[devreal]

Not entirely, unfortunately. We still need a beefed up dash::Future for bulk wait to not leak memory through the handles. It's on my list :)