MPI tag limit

[stgeke]

Currently we do not comply with the MPI standard (maximum tag value < 32768) as we tag messages with nid.

[pschlatt1]

yes, however the code checks the maximum tag ID with MPI_TAG_UB.

[stgeke]

Sure, but this check will not make us compliant. Cray typically support up to 2**21.

[stgeke]

You specify lp in SIZE, nid will run from 1 ... #MPI-ranks < lp. You are find if lp<MPI_TAG_UB but what if you want to run on more MPI ranks.

On 16.08.2016, at 19:52, Ron Rahaman notifications@github.com wrote:

Isn't NID declared in SIZE? If so, isn't it up to the user to make sure that the type of NID can accommodate however many ranks they want?

(Sorry for jumping in; I just saw this as a learning opportunity)

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or mute the thread.

[RonRahaman]

Then isn't that the fault of the MPI implementation? We get MPI_TAG_UB from mpi_get_attr. If MPI_TAG_UB doesn't accommodate the maximum ranks on the machine, then it's not the right implementation for that machine. Then you send a nasty email to your sysadmin.

[stgeke]

No! The MPI standard says MPI_TAG_UB has to be larger than 32768 (see above).

On 16.08.2016, at 19:59, Ron Rahaman notifications@github.com wrote:

Then isn't that the fault of the MPI implementation? We get MPI_TAG_UB from mpi_get_attr. If MPI_TAG_UB doesn't accommodate the maximum ranks on the machine, then it's not the right implementation for that machine. Then you send a nasty email to your sysadmin.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or mute the thread.

[pschlatt1]

Indeed, as @stgeke says we can only be sure to have 32768 tags, however most MPI implementations had many more in the past.
So I agree this is something one has to change at some point, but so far we could get around it as we still had 2**21=2M tags on the latest Cray. However, chances are that one gets a few bits more in the tag in the future. But sure, it is not standard compliant.

[stgeke]

How about your stuff @jlottes?
It looks like you are using the MPI rank to tag messages.

[jlottes]

Yea, the MPI code in jl/ could probably use a re-thinking in how it does tags. I don't think I gave this too much thought at the time, and probably just used the MPI rank because it seemed as good a thing to choose as anything else. In retrospect, I don't think using the rank buys anything. It shouldn't be too hard to fix things.