A question about GASNet

[JsonZhangAA]

Hi,
I have a question about GASNet.If I have installed GASNet,I should run Lux like that:
./pagerank 1 -ll:gpu 2 -ll:fsize 6000 -ll:zsize 5000 -file ~/zy/hollywood.lux -ni 1.
And I also set the environment variable SSH_SERVERS.
But it has the following error：

*** GASNET WARNING: int AMUDP_SPMDStartup_AMUDP_NDEBUG(int*, char***, int, int, amudp_spawnfn_t, uint64_t*, amudp_eb**, amudp_ep**) returning an error code: AM_ERR_RESOURCE (Problem with requested resource)
  from function AMUDP_SPMDStartup
  at ./amudp_spmd.cpp:961
  reason: slave failed DNSLookup on master host name
GASNet initialization encountered an error: "slave AMUDP_SPMDStartup() failed"
  in gasnetc_init at /home/aim/zhangyang_workplace/GASNet-1.32.0/udp-conduit/gasnet_core.c:242
GASNet gasnetc_init returning an error code: GASNET_ERR_RESOURCE (Problem with requested resource)
  at /home/aim/zhangyang_workplace/GASNet-1.32.0/udp-conduit/gasnet_core.c:306
GASNet gasnet_init_GASNET_1320PARnopshmFASTnodebugnotracenostatsnodebugmallocnosrclines returning an error code: GASNET_ERR_RESOURCE (Problem with requested resource)
  at /home/aim/zhangyang_workplace/GASNet-1.32.0/udp-conduit/gasnet_core.c:312
GASNET: gasnet_init(argc, argv) = 3 (GASNET_ERR_RESOURCE, Problem with requested resource) 

Sorry, my English is not very good.

[jiazhihao]

Hi,

It seems you are running Lux on a single machine, in which case you don't need to build with GASNet enabled. In the case you would like to do distributed runs, you can use the following command line to start gasnet (wrap pagerank with mpirun)

mpirun -n NM ./pagerank -ni 10 -file /cstor/stanford/aaiken/users/zhihao/LuxGraphs/hollywood.lux  -ll:gpu 2 -ll:cpu 2 -ll:fsize 5000 -ll:zsize 5000

where NM is the number of machines. You can also replace mpirun with srun or gasnetrun.

[JsonZhangAA]

Hi,

I have built a distributed version of Lux,Like this,

make clean
USE_GASNET=1 make -j 4

And run Lux,like this,

mpirun -n 2 ./pagerank  -ni 1 -file ~/zy/hollywood.lux -ll:gpu 2 -ll:cpu 2 -ll:fsize 5000 -ll:zsize 5000
GASNet: Invalid number of nodes: -ni
GASNet: Usage './pagerank <num_nodes> {program arguments}'
GASNet: Invalid number of nodes: -ni
GASNet: Usage './pagerank <num_nodes> {program arguments}'
--------------------------------------------------------------------------
mpirun noticed that the job aborted, but has no info as to the process
that caused that situation.
--------------------------------------------------------------------------

[jiazhihao]

Hi,

I just realized that you are using GASNet UDP conduit, whose launch is slightly different than IBV convudit. Can you install another version of GASNet available at: https://github.com/StanfordLegion/gasnet. It includes all configuration parameters for building a correct version of GASNet. If you still encounter the issue in your first email, I will redirect this to the Legion/GASNet team to figure it out.

[JsonZhangAA]

Hi,

I install GASNet at https://github.com/StanfordLegion/gasnet.And I run it.
aim@aim-PowerEdge-R730:~/zy/Lux/pagerank$ mpirun -n 2 ./pagerank -ni 1 -file ~/zy/hollywood.lux -ll:gpu 1 -ll:cpu 1 -ll:fsize 5000 -ll:zsize 5000

GASNet gasnetc_init returning an error code: GASNET_ERR_RESOURCE (Problem with requested resource)
  at /home/aim/zy/gasnet-master/GASNet-1.30.0/ibv-conduit/gasnet_core.c:1625
  reason: unable to open any HCA ports
GASNet gasnet_init_GASNET_1300PARnopshmFASTnodebugnotracenostatsnodebugmallocnosrclines returning an error code: GASNET_ERR_RESOURCE (Problem with requested resource)
  at /home/aim/zy/gasnet-master/GASNet-1.30.0/ibv-conduit/gasnet_core.c:1911
GASNET: gasnet_init(argc, argv) = 10002 (GASNET_ERR_RESOURCE, Problem with requested resource)
GASNet gasnetc_init returning an error code: GASNET_ERR_RESOURCE (Problem with requested resource)
  at /home/aim/zy/gasnet-master/GASNet-1.30.0/ibv-conduit/gasnet_core.c:1625
  reason: unable to open any HCA ports
GASNet gasnet_init_GASNET_1300PARnopshmFASTnodebugnotracenostatsnodebugmallocnosrclines returning an error code: GASNET_ERR_RESOURCE (Problem with requested resource)
  at /home/aim/zy/gasnet-master/GASNet-1.30.0/ibv-conduit/gasnet_core.c:1911
GASNET: gasnet_init(argc, argv) = 10002 (GASNET_ERR_RESOURCE, Problem with requested resource)
--------------------------------------------------------------------------
mpirun has exited due to process rank 0 with PID 59579 on
node aim-PowerEdge-R730 exiting improperly. There are two reasons this could occur:

1. this process did not call "init" before exiting, but others in
the job did. This can cause a job to hang indefinitely while it waits
for all processes to call "init". By rule, if one process calls "init",
then ALL processes must call "init" prior to termination.

2. this process called "init", but exited without calling "finalize".
By rule, all processes that call "init" MUST call "finalize" prior to
exiting or it will be considered an "abnormal termination"

This may have caused other processes in the application to be
terminated by signals sent by mpirun (as reported here).
--------------------------------------------------------------------------

I think it may be my computer doesn't have the InfiniBand.Can I run Lux with mpi?

[elliottslaughter]

Can you try:

git clone https://github.com/StanfordLegion/gasnet.git
cd gasnet
make CONDUIT=mpi

And then when you build Lux:

make clean
USE_GASNET=1 CONDUIT=mpi make -j4

Please note: the mpi conduit of GASNet is not high-performance, it should be used for testing only. If you want high performance we'd generally recommend you use a high-performance network (Infiniband, Aries, etc.) or if you are forced to use Ethernet then the udp conduit of GASNet.

[elliottslaughter]

If you do need high performance on an Ethernet network, here would be my recommendation:

git clone https://github.com/StanfordLegion/gasnet.git
cd gasnet
make CONDUIT=udp

And then when you build Lux:

make clean
USE_GASNET=1 CONDUIT=udp make -j4

And then to run do something like:

SSH_SERVERS=localhost,localhost ../../gasnet/release/bin/amudprun -n 2 ./pagerank ...

Note the use of amudprun instead of mpirun.

[JsonZhangAA]

Hi,
Thanks for your reply.
I use udp according to your method, but it shows the following error.

aim@aim-PowerEdge-R730:~/zy/Lux/pagerank$ SSH_SERVERS=localhost,P4000 ~/zy/gasnet-master/release/bin/amudprun -n 2 ./pagerank  -ni 1 -file ~/zy/hollywood.lux -ll:gpu 1  -ll:fsize 5000 -ll:zsize 5000
aim@p4000's password: 
AMUDP int AMUDP_SPMDStartup_AMUDP_NDEBUG(int*, char***, int, int, amudp_spawnfn_t, uint64_t*, amudp_eb**, amudp_ep**) returning an error code: AM_ERR_RESOURCE (Problem with requested resource)
  from function AMUDP_SPMDStartup
  at /home/aim/zy/gasnet/GASNet-1.30.0/other/amudp/amudp_spmd.cpp:988
  reason: slave failed DNSLookup on master host name
GASNet initialization encountered an error: "slave AMUDP_SPMDStartup() failed"
  in gasnetc_init at /home/aim/zy/gasnet/GASNet-1.30.0/udp-conduit/gasnet_core.c:242
GASNet gasnetc_init returning an error code: GASNET_ERR_RESOURCE (Problem with requested resource)
  at /home/aim/zy/gasnet/GASNet-1.30.0/udp-conduit/gasnet_core.c:306
GASNet gasnet_init_GASNET_1300PARnopshmFASTnodebugnotracenostatsnodebugmallocnosrclines returning an error code: GASNET_ERR_RESOURCE (Problem with requested resource)
  at /home/aim/zy/gasnet/GASNet-1.30.0/udp-conduit/gasnet_core.c:312
GASNET: gasnet_init(argc, argv) = 3 (GASNET_ERR_RESOURCE, Problem with requested resource)

P4000 is another machine.What is the reason for this?
I have no idea.

[elliottslaughter]

The udp conduit in GASNet needs to use SSH to connect to the other machines (assuming you don't have some sort of job launcher installed on the machine). I believe in order for this to work you need to be able to SSH without a password. The easiest way to do this would be to create a passwordless SSH key, and copy the public key into ~/.ssh/authorized_keys.

So you should be able to manually run ssh p4000 and log in without a password prompt. If this doesn't work you'll have to fix this before trying GASNet again.

[elliottslaughter]

The other thing I see in your error trace is that it seems like DNS lookup is failing:

reason: slave failed DNSLookup on master host name

I don't know how your machine is configured, but if DNS isn't set up properly, you may need to use the direct IP address of the machine instead of its hostname.

Edit: and this probably goes for both machines (remember that the other machine will have to establish a connection back to your current machine as well, so localhost doesn't provide enough information to do this).

[JsonZhangAA]

Hi,
I solved the problem about DNS. And I can SSH without a password.But I have a question.
In Titan(node 0), I can run Lux on P4000(node 1) and I can also do that on Titan.But I can't run Lux on both nodes.

aim@aim-PowerEdge-R730:~/zy/Lux/pagerank$ SSH_SERVERS=Titan ../../gasnet-master/release/bin/amudprun -n 1 ./pagerank  -ni 1 -file ~/zy/hollywood.lux -ll:gpu 1  -ll:fsize 5000 -ll:zsize 5000
[0 - 7f3af6f8d700] {3}{pagerank}: PageRank settings: numPartitions(1) numIter(1) filename = /home/aim/zy/hollywood.lux
[0 - 7f3af6f8d700] {3}{graph}: Load graph: numNodes(1139905) numEdges(57515616)
[0 - 7f3af6f8d700] {3}{graph}: left_bound = 0 right_bound = 1139904
[Memory Setting] Set ll:fsize >= 470MB and ll:zsize >= 242MB
[0 - 7f3af6f8d700] {3}{graph}: Load task: file(/home/aim/zy/hollywood.lux) rowLeft(0) rowRight(1139904) colLeft(0) colRight(57515615)
[0 - 7f3af6f8d700] {3}{pagerank}: Start PageRank computation...
[0 - 7f3af6f8d700] {3}{pagerank}: Finish PageRank computation...
ELAPSED TIME = 0.0146050 s

aim@aim-PowerEdge-R730:~/zy/Lux/pagerank$ SSH_SERVERS=P4000 ../../gasnet-master/release/bin/amudprun -n 1 ./pagerank  -ni 1 -file ~/zy/hollywood.lux -ll:gpu 1  -ll:fsize 5000 -ll:zsize 5000
slave args: aim-PowerEdge-R730:57339
[0 - 7f63553ce700] {3}{pagerank}: PageRank settings: numPartitions(1) numIter(1) filename = /home/aim/zy/hollywood.lux
[0 - 7f63553ce700] {3}{graph}: Load graph: numNodes(1139905) numEdges(57515616)
[0 - 7f63553ce700] {3}{graph}: left_bound = 0 right_bound = 1139904
[Memory Setting] Set ll:fsize >= 470MB and ll:zsize >= 242MB
[0 - 7f63553ce700] {3}{graph}: Load task: file(/home/aim/zy/hollywood.lux) rowLeft(0) rowRight(1139904) colLeft(0) colRight(57515615)
[0 - 7f63553ce700] {3}{pagerank}: Start PageRank computation...
ELAPSED TIME = 0.0138280 s
[0 - 7f63553ce700] {3}{pagerank}: Finish PageRank computation..

The following situation has a problem

aim@aim-PowerEdge-R730:~/zy/Lux/pagerank$ SSH_SERVERS=Titan,P4000 ../../gasnet-master/release/bin/amudprun -n 2 ./pagerank  -ni 1 -file ~/zy/hollywood.lux -ll:gpu 1  -ll:fsize 5000 -ll:zsize 5000
slave args: aim-PowerEdge-R730:55492
[0 - 7fa17cdc4700] {3}{pagerank}: PageRank settings: numPartitions(1) numIter(1) filename = /home/aim/zy/hollywood.lux
[0 - 7fa17cdc4700] {3}{graph}: Load graph: numNodes(1139905) numEdges(57515616)
[0 - 7fa17cdc4700] {3}{graph}: left_bound = 0 right_bound = 600856
[0 - 7fa17cdc4700] {3}{graph}: left_bound = 600857 right_bound = 1139904
*** Caught a fatal signal: SIGSEGV(11) on node 1/2
NOTICE: Before reporting bugs, run with GASNET_BACKTRACE=1 in the environment to generate a backtrace. 
aim@aim-PowerEdge-R730:~/zy/Lux/pagerank$ bash: line 1: 21751 Segmentation fault      (core dumped) env 'AMUDP_SLAVE_ARGS=1,aim-PowerEdge-R730:55492,' './pagerank' '-ni' '1' '-file' '/home/aim/zy/hollywood.lux' '-ll:gpu' '1' '-ll:fsize' '5000' '-ll:zsize' '5000'

Do I still need a node to run P4000 and Titan?
Thank you for your prompt reply.

I found that the problem happend with the execution of the following statement:

Rect<1> r = runtime->get_index_space_domain(ctx,col_idx.get_index_space());

[elliottslaughter]

Can you get a backtrace? It's best to build with DEBUG=1 (note: make clean first) and then run with REALM_FREEZE_ON_ERROR=1 in the environment. You should get an error message like Process X has frozen on node N. and then you can attach with gdb -p X to dump a backtrace with thread apply all bt.

[JsonZhangAA]

Hi,
It prompts the following information：

aim@aim:~/zy/LuxOld/pagerank$ SSH_SERVERS=115.157.201.179,115.157.201.181 ~/zy/gasnet/release/bin/amudprun -n 2 ./pagerank  -ni 1 -file ~/zy/LuxData/indochina.lux -ll:gpu 1 -ll:fsize 5000 -ll:zsize 5000
*** GASNET WARNING: Both $GASNET_SSH_SERVERS and $SSH_SERVERS are set, to different values. Using the former.
slave args: 115,157,201,181,137,177
[0 - 7f0ad1104700] {3}{pagerank}: PageRank settings: numPartitions(1) numIter(1) filename = /home/aim/zy/LuxData/indochina.lux
[0 - 7f0ad1104700] {3}{graph}: Load graph: numNodes(7414866) numEdges(194109311)
[0 - 7f0ad1104700] {3}{graph}: left_bound = 0 right_bound = 5322925
[0 - 7f0ad1104700] {3}{graph}: left_bound = 5322926 right_bound = 7414865
*** Caught a fatal signal: SIGSEGV(11) on node 1/2
[1] /usr/bin/gdb -nx -batch -x /tmp/gasnet_at4eoA '/home/aim/zy/LuxOld/pagerank/./pagerank' 2918
[1] No threads.
aim@aim:~/zy/LuxOld/pagerank$ bash: line 1:  2918 Segmentation fault      (core dumped) env 'AMUDP_SLAVE_ARGS=1,115,157,201,181,137,177,' './pagerank' '-ni' '1' '-file' '/home/aim/zy/LuxData/indochina.lux' '-ll:gpu' '1' '-ll:fsize' '5000' '-ll:zsize' '5000'
^C

[elliottslaughter]

It's a bit hard to tell what's going on here since the backtrace appears to have failed, but here's what I'd recommend to debug it:

1. Rebuild with with DEBUG=1 (note: make clean first).
2. Run with REALM_FREEZE_ON_ERROR=1 and not GASNET_BACKTRACE=1. (Edit: I suspect the latter was set in your run above because it was trying to get a backtrace automatically. It should not have been doing this if only REALM_FREEZE_ON_ERROR=1 was set.)
3. Wait for message like Process 123 is frozen on node n0000.
4. SSH to n0000 and gdb -p 123, then thread apply all bt and copy-and-paste that into a file to be attached here.

[JsonZhangAA]

Hi,
I follow the method you recommended to execute the program.
After step 2, I don't receive the message like Process 123 is frozen on node n0000.
It shows the follow information:

aim@aim:~/zy/LuxOld/pagerank$ REALM_FREEZE_ON_ERROR=1 SSH_SERVERS=115.157.201.181,115.157.201.179 ~/zy/gasnet/release/bin/amudprun -n 2 ./pagerank  -ni 1 -file ~/zy/LuxData/hollywood.lux -ll:gpu 1 -ll:cpu 1 -ll:fsize 5000 -ll:zsize 5000
slave args: 115,157,201,181,187,105
[0 - 7f4d16a96700] {3}{pagerank}: PageRank settings: numPartitions(1) numIter(1) filename = /home/aim/zy/LuxData/hollywood.lux
[0 - 7f4d16a96700] {3}{graph}: Load graph: numNodes(1139905) numEdges(57515616)
[0 - 7f4d16a96700] {3}{graph}: left_bound = 0 right_bound = 600856
[0 - 7f4d16a96700] {3}{graph}: left_bound = 600857 right_bound = 1139904
*** Caught a fatal signal: SIGSEGV(11) on node 1/2
NOTICE: Before reporting bugs, run with GASNET_BACKTRACE=1 in the environment to generate a backtrace.
aim@aim:~/zy/LuxOld/pagerank$ bash: line 1: 18910 Segmentation fault      (core dumped) env 'AMUDP_SLAVE_ARGS=1,115,157,201,181,187,105,' './pagerank' '-ni' '1' '-file' '/home/aim/zy/LuxData/hollywood.lux' '-ll:gpu' '1' '-ll:cpu' '1' '-ll:fsize' '5000' '-ll:zsize' '5000'

This is the information of DEBUG=1 USE_GASNET=1 CONDUIT=udp make -j4

aim@aim:~/zy/LuxOld/pagerank$ DEBUG=1 USE_GASNET=1 CONDUIT=udp make -j4
g++ -o pagerank.cc.o -c pagerank.cc  -std=c++11 -DUSE_DISK  -march=native -DUSE_LIBDL -DREALM_USE_OPENMP -DREALM_OPENMP_GOMP_SUPPORT -DREALM_OPENMP_KMP_SUPPORT -DUSE_CUDA -DUSE_GASNET -DGASNETI_BUG1389_WORKAROUND=1 -DGASNET_CONDUIT_UDP -DUSE_ZLIB -DDEBUG_REALM -DDEBUG_LEGION -O0 -ggdb  -DCOMPILE_TIME_MIN_LEVEL=LEVEL_DEBUG-Wall -Wno-strict-overflow -I../cub -I. -I../legion/runtime -I../legion/runtime/mappers -I/usr/local/cuda/include -I../legion/runtime/realm/transfer -I/home/aim/zy/gasnet/release/include -I/home/aim/zy/gasnet/release/include/udp-conduit
g++ -o ../core/lux_mapper.cc.o -c ../core/lux_mapper.cc  -std=c++11 -DUSE_DISK  -march=native -DUSE_LIBDL -DREALM_USE_OPENMP -DREALM_OPENMP_GOMP_SUPPORT -DREALM_OPENMP_KMP_SUPPORT -DUSE_CUDA -DUSE_GASNET -DGASNETI_BUG1389_WORKAROUND=1 -DGASNET_CONDUIT_UDP -DUSE_ZLIB -DDEBUG_REALM -DDEBUG_LEGION -O0 -ggdb  -DCOMPILE_TIME_MIN_LEVEL=LEVEL_DEBUG         -Wall -Wno-strict-overflow -I../cub -I. -I../legion/runtime -I../legion/runtime/mappers -I/usr/local/cuda/include -I../legion/runtime/realm/transfer -I/home/aim/zy/gasnet/release/include -I/home/aim/zy/gasnet/release/include/udp-conduit
/usr/local/cuda/bin/nvcc -o pagerank_gpu.cu.o -c pagerank_gpu.cu  -std=c++11 -DUSE_CUDA -DDEBUG_REALM -DDEBUG_LEGION -g -O0 -arch=compute_52 -code=sm_52 -DMAXWELL_ARCH -Xptxas "-v"  -I../cub -I. -I../legion/runtime -I../legion/runtime/mappers -I/usr/local/cuda/include -I../legion/runtime/realm/transfer -I/home/aim/zy/gasnet/release/include -I/home/aim/zy/gasnet/release/include/udp-conduit
rm -f liblegion.a
ar rc liblegion.a ../legion/runtime/legion/legion.cc.o ../legion/runtime/legion/legion_c.cc.o ../legion/runtime/legion/legion_ops.cc.o ../legion/runtime/legion/legion_tasks.cc.o ../legion/runtime/legion/legion_context.cc.o ../legion/runtime/legion/legion_trace.cc.o ../legion/runtime/legion/legion_spy.cc.o ../legion/runtime/legion/legion_profiling.cc.o ../legion/runtime/legion/legion_profiling_serializer.cc.o ../legion/runtime/legion/legion_instances.cc.o ../legion/runtime/legion/legion_views.cc.o ../legion/runtime/legion/legion_analysis.cc.o ../legion/runtime/legion/legion_constraint.cc.o ../legion/runtime/legion/legion_mapping.cc.o ../legion/runtime/legion/region_tree.cc.o ../legion/runtime/legion/runtime.cc.o ../legion/runtime/legion/garbage_collection.cc.o ../legion/runtime/legion/mapper_manager.cc.o ../legion/runtime/mappers/default_mapper.cc.o ../legion/runtime/mappers/mapping_utilities.cc.o ../legion/runtime/mappers/shim_mapper.cc.o ../legion/runtime/mappers/test_mapper.cc.o ../legion/runtime/mappers/replay_mapper.cc.o ../legion/runtime/mappers/debug_mapper.cc.o ../legion/runtime/mappers/wrapper_mapper.cc.o
rm -f librealm.a
ar rc librealm.a ../legion/runtime/realm/runtime_impl.cc.o ../legion/runtime/realm/transfer/transfer.cc.o ../legion/runtime/realm/transfer/channel.cc.o ../legion/runtime/realm/transfer/channel_disk.cc.o ../legion/runtime/realm/transfer/lowlevel_dma.cc.o ../legion/runtime/realm/module.cc.o ../legion/runtime/realm/threads.cc.o../legion/runtime/realm/faults.cc.o ../legion/runtime/realm/operation.cc.o ../legion/runtime/realm/tasks.cc.o ../legion/runtime/realm/metadata.cc.o ../legion/runtime/realm/deppart/partitions.cc.o ../legion/runtime/realm/deppart/sparsity_impl.cc.o ../legion/runtime/realm/deppart/image.cc.o ../legion/runtime/realm/deppart/preimage.cc.o ../legion/runtime/realm/deppart/byfield.cc.o ../legion/runtime/realm/deppart/setops.cc.o ../legion/runtime/realm/event_impl.cc.o ../legion/runtime/realm/rsrv_impl.cc.o ../legion/runtime/realm/proc_impl.cc.o ../legion/runtime/realm/mem_impl.cc.o ../legion/runtime/realm/inst_impl.cc.o ../legion/runtime/realm/inst_layout.cc.o ../legion/runtime/realm/machine_impl.cc.o ../legion/runtime/realm/sampling_impl.cc.o ../legion/runtime/realm/transfer/lowlevel_disk.cc.o ../legion/runtime/realm/numa/numa_module.cc.o ../legion/runtime/realm/numa/numasysif.cc.o ../legion/runtime/realm/openmp/openmp_module.cc.o ../legion/runtime/realm/openmp/openmp_threadpool.cc.o ../legion/runtime/realm/openmp/openmp_api.cc.o ../legion/runtime/realm/procset/procset_module.cc.o ../legion/runtime/realm/cuda/cuda_module.cc.o ../legion/runtime/realm/cuda/cudart_hijack.cc.o ../legion/runtime/realm/activemsg.cc.o ../legion/runtime/realm/logging.cc.o ../legion/runtime/realm/cmdline.cc.o ../legion/runtime/realm/profiling.cc.o ../legion/runtime/realm/codedesc.cc.o ../legion/runtime/realm/timers.cc.o
ptxas info    : 1 bytes gmem
ptxas info    : Compiling entry function '_ZN3cub11EmptyKernelIvEEvv' for 'sm_52'
ptxas info    : Function properties for _ZN3cub11EmptyKernelIvEEvv
    0 bytes stack frame, 0 bytes spill stores, 0 bytes spill loads
ptxas info    : Used 2 registers, 320 bytes cmem[0]
ptxas info    : Compiling entry function '_Z11init_kerneljjmP10NodeStructP10EdgeStructPKmPKjS6_' for 'sm_52'
ptxas info    : Function properties for _Z11init_kerneljjmP10NodeStructP10EdgeStructPKmPKjS6_
    0 bytes stack frame, 0 bytes spill stores, 0 bytes spill loads
ptxas info    : Used 22 registers, 376 bytes cmem[0]
ptxas info    : Compiling entry function '_Z9pr_kerneljjmfPK10NodeStructPK10EdgeStructPfS5_' for 'sm_52'
ptxas info    : Function properties for _Z9pr_kerneljjmfPK10NodeStructPK10EdgeStructPfS5_
    0 bytes stack frame, 0 bytes spill stores, 0 bytes spill loads
ptxas info    : Used 48 registers, 4392 bytes smem, 376 bytes cmem[0], 12 bytes cmem[2]
ptxas info    : Compiling entry function '_Z11load_kerneljPKjPfPKf' for 'sm_52'
ptxas info    : Function properties for _Z11load_kerneljPKjPfPKf
    0 bytes stack frame, 0 bytes spill stores, 0 bytes spill loads
ptxas info    : Used 10 registers, 352 bytes cmem[0]
---> Linking objects into one binary: pagerank
g++ -o pagerank pagerank.cc.o ../core/lux_mapper.cc.o pagerank_gpu.cu.o  -L. -llegion -lrealm -lrt -lpthread-ldl -rdynamic -L/usr/local/cuda/lib64 -L/usr/local/cuda/lib64/stubs -lcuda  -Xlinker -rpath=/usr/local/cuda/lib64 -L/home/aim/zy/gasnet/release/lib -lrt -lm -lgasnet-udp-par -lamudp -lz

[jiazhihao]

Hi, I tried to reproduce your execution on our cluster with IBV conduit and it worked fine. This confirms that the segment fault you saw is not related to the application code. We still need a backtrack to figure out if the segfault is in the GASNet layer or the Legion runtime.