add macro to create custom Ops also on aarch64

[vchuravy]

No description provided.

[giordano]

Need to delete

MPI.jl/test/test_reduce.jl

Lines 7 to 10 in 609a8fe

	Sys.ARCH !== :powerpc64le &&
	Sys.ARCH !== :ppc64le &&
	Sys.ARCH !== :aarch64 &&
	!startswith(string(Sys.ARCH), "arm")

to actually test this?

[vchuravy]

No since I add it to this as well

MPI.jl/test/test_reduce.jl

Line 70 in 609a8fe

operators = [MPI.SUM, +, my_reduce]

[eschnett]

I only looked at the code and didn't try to run it.

[giordano]

Just to understand: this works around the closures limitations by creating a custom module on-the-fly and defining the closure "statically" inside it?

Side note, it's a bit unfortunate that this is introducing an alternative method to Op to define custom operators: generic libraries will have to pick either @Op or Op (presumably the former if they want to work on all platforms), but I guess we don't have more comprehensive solutions at the moment.

[giordano]

Also, should we add a reference to @Op in the error messages at

MPI.jl/src/operators.jl

Line 95 in 780aaa0

@static if MPI_LIBRARY == "MicrosoftMPI" && Sys.WORD_SIZE == 32

"[...] As an alternative, use the MPI.@Op macro...

[vchuravy]

, it's a bit unfortunate that this is introducing an alternative method to Op to define custom operators: generic libraries will have to pick either @op or Op (presumably the former if they want to work on all platforms), but I guess we don't have more comprehensive solutions at the moment.

The situation is even funnier. Op tries to handle all the things generally, so it takes a function f and doesn't know whether it is a closure or not. @Op "just" creates a method of Op that dispatches on the function f and tada...

So it doesn't introduce an alternative method, you still have to call Op it just "pre-registers" f.

[giordano]

I like the new name much better, it makes the difference clearer, thanks!

[giordano]

Should update documentation at https://juliaparallel.org/MPI.jl/stable/knownissues/#Custom-reduction-operators?

[giordano]

With

diff --git a/docs/examples/03-reduce.jl b/docs/examples/03-reduce.jl
index 86d22be1..26aa9658 100644
--- a/docs/examples/03-reduce.jl
+++ b/docs/examples/03-reduce.jl
@@ -33,8 +33,13 @@ end
 
 X = randn(10,3) .* [1,3,7]'
 
+# Register the custom reduction operator.  This is necessary only on platforms
+# where Julia doesn't support closures as cfunctions (e.g. ARM), but can be used
+# on all platforms for consistency.
+MPI.@RegisterOp(pool, SummaryStat)
+
 # Perform a scalar reduction
-summ = MPI.Reduce(SummaryStat(X), pool, root, comm)
+summ = MPI.Reduce(SummaryStat(X), pool, comm; root)
 
 if MPI.Comm_rank(comm) == root
     @show summ.var
@@ -42,7 +47,7 @@ end
 
 # Perform a vector reduction:
 # the reduction operator is applied elementwise
-col_summ = MPI.Reduce(mapslices(SummaryStat,X,dims=1), pool, root, comm)
+col_summ = MPI.Reduce(mapslices(SummaryStat,X,dims=1), pool, comm; root)
 
 if MPI.Comm_rank(comm) == root
     col_var = map(summ -> summ.var, col_summ)

I can run the custom reduction example in the documentation on aarch64-darwin, it errors in

MPI.jl/src/operators.jl

Line 98 in aac9688

error("User-defined reduction operators are currently not supported on non-Intel architectures.\nSee https://github.com/JuliaParallel/MPI.jl/issues/404 for more details.")

without this change. I think we may want to suggest users to look at MPI.@RegisterOp in that error message.

[giordano]

It turns out staticarrays is used? I'm missing where though 😕

[giordano]

Last bit: update the docsaabout known issues #871 (comment) and then we should be good to go for me!

[vchuravy]

@giordano done!

[giordano]

Thank you so much, this is great!

[simonbyrne]

Could we make use of the potential PerProcess construct here (JuliaLang/julia#55793)?

[vchuravy]

Could we make use of the potential PerProcess construct

Yeah, I think that could be used to generally simplify the implementation, and also our home-grown implementation for other globals.