Four miscellaneous suggestions

[brian-kelley]

After working with kokkos-remote-spaces a bit, there are a few little changes I would like to propose to make interfaces and behavior more intuitive, especially for people already comfortable with Kokkos core.

It's possible that some of these already exist and I just haven't found them in the code though.

• Make Kokkos::Experimental::getRange return a range with an exclusive upper bound (half-open interval), not inclusive. Kokkos Core consistently uses half-open intervals in RangePolicy constructors, and subview.
  For example,

auto myRange = Kokkos::Experimental::getRange(101, myRank);
std::cout << "Hello from rank " << myRank << ", range is " << myRange.first << "..." << myRange.second << std::endl;

produces this.

Hello from rank 0, range is 0...25
Hello from rank 1, range is 26...51
Hello from rank 2, range is 52...77
Hello from rank 3, range is 78...100

Instead, I'm proposing it should be

Hello from rank 0, range is 0...26
Hello from rank 1, range is 26...52
Hello from rank 2, range is 52...78
Hello from rank 3, range is 78...101

• Have a way to get the global (0th) extent of a RemoteSpace view. If you make a distributed view like this, with a non-partitioned layout:

    using RemoteSpace = Kokkos::Experimental::DefaultRemoteMemorySpace;
    Kokkos::View<double*, RemoteSpace> myView("myView", 100);

the global extent will be 100, but it seems like there isn't a way to get that back once you've created the view. myView.extent(0) will return the maximum local extent on any one rank (for example, with 3 ranks, the above example will have myView.extent(0) == 34 on all ranks). This means that even with an all-reduce, you can't recover the global extent if it wasn't evenly divisible by the number of ranks.

• Related to the last thing: if you have a distributed view, and add up its local extents on the ranks, that sum should be the same as the global extent you passed to the view constructor. So in this example, executed on 3 ranks:

    using RemoteSpace = Kokkos::Experimental::DefaultRemoteMemorySpace;
    Kokkos::View<double*, RemoteSpace> myView("myView", 100);
    std::cout << "On rank " << myRank << ", myView.extent(0) = " << myView.extent(0) << '\n';

right now this prints

On rank 0, myView.extent(0) = 34
On rank 1, myView.extent(0) = 34
On rank 2, myView.extent(0) = 34

I propose that this instead be

On rank 0, myView.extent(0) = 34
On rank 1, myView.extent(0) = 33
On rank 2, myView.extent(0) = 33

so that 34+33+33=100, the global extent.

• For remote space views, it would be nice to have a method to get the local subview. For example, suppose RemoteSpace == SHMEMSpace and myView is constructed like in the above snippet, myView.getLocalSubview() would return a Kokkos::View<double*, Kokkos::HostSpace> with extent 34 or 33 (depending on which rank it's called from). Right now, it seems that an unmanaged view (pointer & extent) is needed to create a view of the local data.

[janciesko]

Thanks for the comments.

1. Open upper bound in range API. This will be the case from Overhaul the project #82 on.
2. This could be a TODO. Historically, shmem requires symmetric memory allocations. We expose this through the allocation mode in the various remote back-ends enum RemoteSpaces_MemoryAllocationMode : int { Symmetric, Cached }; In the Symmetric allocation mode, we probably want to match the allocation size with the extents so the memory is accessible through the View operator. Currently the user would use the range API to get the actual logical range for that extent(0). Supporting Asymmetric should give the behavior you're describing.
3. Yes, we could add that. Currently, the programmer would use the Kokkos::subview API or View ctr with a range for that purpose.