eisen@crusher012:~/proj> srun -n 1 ~/ADIOS2/crusher/bin/TestCommonRead sst tmp : -n 1 ~/ADIOS2/crusher/bin/TestCommonWrite sst tmp [==========] Running 1 test from 1 test suite. [----------] Global test environment set-up. [----------] 1 test from CommonReadTest [ RUN ] CommonReadTest.ADIOS2CommonRead1D8 Reader 0 (0x2e38d0): Sst set to use sockets as a Control Transport [==========] Running 1 test from 1 test suite. [----------] Global test environment set-up. [----------] 1 test from CommonWriteTest [ RUN ] CommonWriteTest.ADIOS2CommonWrite Nx is set to 10 on Rank 0 Writer 0 (0x2de050): Sst set to use sockets as a Control Transport Reader 0 (0x2e38d0): Looking for writer contact in file tmp.sst, with timeout 60 secs DP Writer 0 (0x2de050): Considering DataPlane "evpath" for possible use, priority is 1 DP Writer 0 (0x2de050): Considering DataPlane "ucx" for possible use, priority is 10 DP Writer 0 (0x2de050): Selecting DataPlane "ucx", priority 10 for use DP Writer 0 (0x2de050): UCX init Success Writer 0 (0x2de050): Opening Stream "tmp" Writer 0 (0x2de050): Writer stream params are: Param - RegistrationMethod=File Param - RendezvousReaderCount=1 Param - QueueLimit=0 (unlimited) Param - QueueFullPolicy=Block Param - StepDistributionMode=StepsAllToAll Param - DataTransport=ucx Param - ControlTransport=sockets Param - NetworkInterface=(default) Param - ControlInterface=(default to NetworkInterface if applicable) Param - DataInterface=(default to NetworkInterface if applicable) Param - CompressionMethod=None Param - CPCommPattern=Min Param - MarshalMethod=BP Param - FirstTimestepPrecious=False Param - IsRowMajor=1 (not user settable) Param - OpenTimeoutSecs=60 (seconds) Param - SpeculativePreloadMode=Auto Param - SpecAutoNodeThreshold=1 Param - ControlModule=select Writer 0 (0x2de050): Stream "tmp" waiting for 1 readers Reader 0 (0x2e38d0): Waiting for writer DPResponse message in SstReadOpen("tmp") Reader 0 (0x2e38d0): finished wait writer DPresponse message in read_open, WRITER is using "ucx" DataPlane DP Reader 0 (0x2e38d0): Prefered dataplane name is "ucx" DP Reader 0 (0x2e38d0): Considering DataPlane "evpath" for possible use, priority is 1 DP Reader 0 (0x2e38d0): Considering DataPlane "ucx" for possible use, priority is 10 DP Reader 0 (0x2e38d0): Selecting DataPlane "ucx" (preferred) for use DP Reader 0 (0x2e38d0): UCX init Success Reader 0 (0x2e38d0): Waiting for writer response message in SstReadOpen("tmp") Writer 0 (0x2de050): Beginning writer-side reader open protocol Writer 0 (0x2de050): Setting SpeculativePreload ON for new reader Writer 0 (0x2de050): My oldest timestep was 0, global oldest timestep was 0 Writer 0 (0x2de050): Finish writer-side reader open protocol for reader 0x36e1f0, reader ready response pending Writer 0 (0x2de050): (PID d8bf, TID 7fffed809780) Waiting for Reader ready on WSR 0x36e1f0. Reader 0 (0x2e38d0): finished wait writer response message in read_open Reader 0 (0x2e38d0): Opening Reader Stream. Writer stream params are: Param - RegistrationMethod=File Param - RendezvousReaderCount=1 Param - QueueLimit=0 (unlimited) Param - QueueFullPolicy=Block Param - StepDistributionMode=StepsAllToAll Param - DataTransport=ucx Param - ControlTransport=sockets Param - NetworkInterface=(default) Param - ControlInterface=(default to NetworkInterface if applicable) Param - DataInterface=(default to NetworkInterface if applicable) Param - CompressionMethod=None Param - CPCommPattern=Min Param - MarshalMethod=BP Param - FirstTimestepPrecious=False Param - IsRowMajor=1 (not user settable) Param - OpenTimeoutSecs=60 (seconds) Param - SpeculativePreloadMode=Auto Param - SpecAutoNodeThreshold=1 Param - ControlModule=select Reader 0 (0x2e38d0): Reader stream params are: Param - RegistrationMethod=File Param - DataTransport=ucx Param - ControlTransport=sockets Param - NetworkInterface=(default) Param - ControlInterface=(default to NetworkInterface if applicable) Param - DataInterface=(default to NetworkInterface if applicable) Param - AlwaysProvideLatestTimestep=False Param - OpenTimeoutSecs=60 (seconds) Param - SpeculativePreloadMode=Auto Param - SpecAutoNodeThreshold=1 Param - ControlModule=select Reader 0 (0x2e38d0): Writer is doing BP-based marshalling Reader 0 (0x2e38d0): Writer is using Minimum Connection Communication pattern (min) DP Reader 0 (0x2e38d0): Received contact info for WS_stream 0x300ed0, WSR Rank 0 Reader 0 (0x2e38d0): Sending Reader Activate messages to writer Reader 0 (0x2e38d0): Finish opening Stream "tmp", starting with Step number 0 Reader 0 (0x2e38d0): Wait for next metadata after last timestep -1 Reader 0 (0x2e38d0): Waiting for metadata for a Timestep later than TS -1 Reader 0 (0x2e38d0): (PID 5395, TID 7fffed809780) Stream status is Established Writer 0 (0x2de050): Reader Activate message received for Stream 0x36e1f0. Setting state to Established. Writer 0 (0x2de050): Parent stream reader count is now 1. Writer 0 (0x2de050): Reader ready on WSR 0x36e1f0, Stream established, Starting 0 LastProvided 0. Writer 0 (0x2de050): Finish opening Stream "tmp" DP Writer 0 (0x2de050): Providing timestep data with block 0x378340 and access key 3654640 Writer 0 (0x2de050): Reader 0 status Established has last released 4294967295, last sent 0 Writer 0 (0x2de050): QueueMaintenance, smallest last released = -1, count = 1 Writer 0 (0x2de050): Removing dead entries Writer 0 (0x2de050): QueueMaintenance complete Writer 0 (0x2de050): Sending TimestepMetadata for timestep 0 (ref count 1), one to each reader Writer 0 (0x2de050): Sent timestep 0 to reader cohort 0 Writer 0 (0x2de050): ADDING timestep 0 to sent list for reader cohort 0, READER 0x36e1f0, reference count is now 2 Writer 0 (0x2de050): PRELOADMODE for timestep 0 non-default for reader , active at timestep 0, mode 1 Writer 0 (0x2de050): Sending a message to reader 0 (0x2e38d0) Writer 0 (0x2de050): SubRef : Writer-side Timestep 0 now has reference count 1, expired 0, precious 0 Writer 0 (0x2de050): Reader 0 status Established has last released 4294967295, last sent 0 Writer 0 (0x2de050): QueueMaintenance, smallest last released = -1, count = 1 Writer 0 (0x2de050): Removing dead entries Writer 0 (0x2de050): QueueMaintenance complete DP Writer 0 (0x2de050): Providing timestep data with block 0x384e50 and access key -805293952 Writer 0 (0x2de050): Reader 0 status Established has last released 4294967295, last sent 0 Writer 0 (0x2de050): QueueMaintenance, smallest last released = -1, count = 2 Writer 0 (0x2de050): Removing dead entries Writer 0 (0x2de050): QueueMaintenance complete Writer 0 (0x2de050): Sending TimestepMetadata for timestep 1 (ref count 1), one to each reader Writer 0 (0x2de050): Sent timestep 1 to reader cohort 0 Writer 0 (0x2de050): ADDING timestep 1 to sent list for reader cohort 0, READER 0x36e1f0, reference count is now 2 Writer 0 (0x2de050): PRELOADMODE for timestep 1 non-default for reader , active at timestep 0, mode 1 Writer 0 (0x2de050): Sending a message to reader 0 (0x2e38d0) Writer 0 (0x2de050): SubRef : Writer-side Timestep 1 now has reference count 1, expired 0, precious 0 Writer 0 (0x2de050): Reader 0 status Established has last released 4294967295, last sent 1 Writer 0 (0x2de050): QueueMaintenance, smallest last released = -1, count = 2 Writer 0 (0x2de050): Removing dead entries Writer 0 (0x2de050): QueueMaintenance complete Reader 0 (0x2e38d0): Received a Timestep metadata message for timestep 0, signaling condition DP Writer 0 (0x2de050): Providing timestep data with block 0x38cdd0 and access key 3682960 Writer 0 (0x2de050): Reader 0 status Established has last released 4294967295, last sent 1 Reader 0 (0x2e38d0): Examining metadata for Timestep 0 Reader 0 (0x2e38d0): Returning metadata for Timestep 0 Reader 0 (0x2e38d0): Setting TSmsg to Rootentry value DP Reader 0 (0x2e38d0): UcxTimestepArrived with Timestep = 0, PreloadMode = 1 Reader 0 (0x2e38d0): SstAdvanceStep returning Success on timestep 0 Writer 0 (0x2de050): QueueMaintenance, smallest last released = -1, count = 3 Writer 0 (0x2de050): Removing dead entries Writer 0 (0x2de050): QueueMaintenance complete Writer 0 (0x2de050): Sending TimestepMetadata for timestep 2 (ref count 1), one to each reader Writer 0 (0x2de050): Sent timestep 2 to reader cohort 0 Writer 0 (0x2de050): ADDING timestep 2 to sent list for reader cohort 0, READER 0x36e1f0, reference count is now 2 Writer 0 (0x2de050): PRELOADMODE for timestep 2 non-default for reader , active at timestep 0, mode 1 Writer 0 (0x2de050): Sending a message to reader 0 (0x2e38d0) Writer 0 (0x2de050): SubRef : Writer-side Timestep 2 now has reference count 1, expired 0, precious 0 Writer 0 (0x2de050): Reader 0 status Established has last released 4294967295, last sent 2 Writer 0 (0x2de050): QueueMaintenance, smallest last released = -1, count = 3 Writer 0 (0x2de050): Removing dead entries Writer 0 (0x2de050): QueueMaintenance complete DP Writer 0 (0x2de050): Providing timestep data with block 0x394c60 and access key 3752416 Writer 0 (0x2de050): Reader 0 status Established has last released 4294967295, last sent 2 Writer 0 (0x2de050): QueueMaintenance, smallest last released = -1, count = 4 Writer 0 (0x2de050): Removing dead entries Writer 0 (0x2de050): QueueMaintenance complete Writer 0 (0x2de050): Sending TimestepMetadata for timestep 3 (ref count 1), one to each reader Writer 0 (0x2de050): Sent timestep 3 to reader cohort 0 Writer 0 (0x2de050): ADDING timestep 3 to sent list for reader cohort 0, READER 0x36e1f0, reference count is now 2 Writer 0 (0x2de050): PRELOADMODE for timestep 3 non-default for reader , active at timestep 0, mode 1 Writer 0 (0x2de050): Sending a message to reader 0 (0x2e38d0) Writer 0 (0x2de050): SubRef : Writer-side Timestep 3 now has reference count 1, expired 0, precious 0 Writer 0 (0x2de050): Reader 0 status Established has last released 4294967295, last sent 3 Writer 0 (0x2de050): QueueMaintenance, smallest last released = -1, count = 4 Writer 0 (0x2de050): Removing dead entries Writer 0 (0x2de050): QueueMaintenance complete DP Writer 0 (0x2de050): Providing timestep data with block 0x39cb30 and access key 3784944 Writer 0 (0x2de050): Reader 0 status Established has last released 4294967295, last sent 3 Writer 0 (0x2de050): QueueMaintenance, smallest last released = -1, count = 5 Writer 0 (0x2de050): Removing dead entries Writer 0 (0x2de050): QueueMaintenance complete Writer 0 (0x2de050): Sending TimestepMetadata for timestep 4 (ref count 1), one to each reader Writer 0 (0x2de050): Sent timestep 4 to reader cohort 0 Writer 0 (0x2de050): ADDING timestep 4 to sent list for reader cohort 0, READER 0x36e1f0, reference count is now 2 Writer 0 (0x2de050): PRELOADMODE for timestep 4 non-default for reader , active at timestep 0, mode 1 Writer 0 (0x2de050): Sending a message to reader 0 (0x2e38d0) Writer 0 (0x2de050): SubRef : Writer-side Timestep 4 now has reference count 1, expired 0, precious 0 Writer 0 (0x2de050): Reader 0 status Established has last released 4294967295, last sent 4 Writer 0 (0x2de050): QueueMaintenance, smallest last released = -1, count = 5 Writer 0 (0x2de050): Removing dead entries Writer 0 (0x2de050): QueueMaintenance complete DP Writer 0 (0x2de050): Providing timestep data with block 0x3a4a40 and access key 3817456 Writer 0 (0x2de050): Reader 0 status Established has last released 4294967295, last sent 4 Writer 0 (0x2de050): QueueMaintenance, smallest last released = -1, count = 6 Writer 0 (0x2de050): Removing dead entries Writer 0 (0x2de050): QueueMaintenance complete Writer 0 (0x2de050): Sending TimestepMetadata for timestep 5 (ref count 1), one to each reader Writer 0 (0x2de050): Sent timestep 5 to reader cohort 0 Writer 0 (0x2de050): ADDING timestep 5 to sent list for reader cohort 0, READER 0x36e1f0, reference count is now 2 Writer 0 (0x2de050): PRELOADMODE for timestep 5 non-default for reader , active at timestep 0, mode 1 Writer 0 (0x2de050): Sending a message to reader 0 (0x2e38d0) Writer 0 (0x2de050): SubRef : Writer-side Timestep 5 now has reference count 1, expired 0, precious 0 Writer 0 (0x2de050): Reader 0 status Established has last released 4294967295, last sent 5 Writer 0 (0x2de050): QueueMaintenance, smallest last released = -1, count = 6 Writer 0 (0x2de050): Removing dead entries Writer 0 (0x2de050): QueueMaintenance complete DP Writer 0 (0x2de050): Providing timestep data with block 0x3ac930 and access key 3849968 Writer 0 (0x2de050): Reader 0 status Established has last released 4294967295, last sent 5 Writer 0 (0x2de050): QueueMaintenance, smallest last released = -1, count = 7 Writer 0 (0x2de050): Removing dead entries Writer 0 (0x2de050): QueueMaintenance complete Writer 0 (0x2de050): Sending TimestepMetadata for timestep 6 (ref count 1), one to each reader Writer 0 (0x2de050): Sent timestep 6 to reader cohort 0 Writer 0 (0x2de050): ADDING timestep 6 to sent list for reader cohort 0, READER 0x36e1f0, reference count is now 2 Writer 0 (0x2de050): PRELOADMODE for timestep 6 non-default for reader , active at timestep 0, mode 1 Writer 0 (0x2de050): Sending a message to reader 0 (0x2e38d0) Writer 0 (0x2de050): SubRef : Writer-side Timestep 6 now has reference count 1, expired 0, precious 0 Writer 0 (0x2de050): Reader 0 status Established has last released 4294967295, last sent 6 Writer 0 (0x2de050): QueueMaintenance, smallest last released = -1, count = 7 Writer 0 (0x2de050): Removing dead entries Writer 0 (0x2de050): QueueMaintenance complete DP Writer 0 (0x2de050): Providing timestep data with block 0x3b4840 and access key 3882480 Writer 0 (0x2de050): Reader 0 status Established has last released 4294967295, last sent 6 Writer 0 (0x2de050): QueueMaintenance, smallest last released = -1, count = 8 Writer 0 (0x2de050): Removing dead entries Writer 0 (0x2de050): QueueMaintenance complete Writer 0 (0x2de050): Sending TimestepMetadata for timestep 7 (ref count 1), one to each reader Writer 0 (0x2de050): Sent timestep 7 to reader cohort 0 Writer 0 (0x2de050): ADDING timestep 7 to sent list for reader cohort 0, READER 0x36e1f0, reference count is now 2 Writer 0 (0x2de050): PRELOADMODE for timestep 7 non-default for reader , active at timestep 0, mode 1 Writer 0 (0x2de050): Sending a message to reader 0 (0x2e38d0) Writer 0 (0x2de050): SubRef : Writer-side Timestep 7 now has reference count 1, expired 0, precious 0 Writer 0 (0x2de050): Reader 0 status Established has last released 4294967295, last sent 7 Writer 0 (0x2de050): QueueMaintenance, smallest last released = -1, count = 8 Writer 0 (0x2de050): Removing dead entries Writer 0 (0x2de050): QueueMaintenance complete DP Reader 0 (0x2e38d0): Performing remote read of Writer Rank 0 at step 0 DP Reader 0 (0x2e38d0): Block address is 0x378340, with a key of l, and access key size 37. DP Reader 0 (0x2e38d0): Remote read target is Rank 0, EP = 0x7fffd43c0000 (Offset = 1046, Length = 80) DP Writer 0 (0x2de050): Providing timestep data with block 0x3bc730 and access key 3914992 Writer 0 (0x2de050): Reader 0 status Established has last released 4294967295, last sent 7 Writer 0 (0x2de050): QueueMaintenance, smallest last released = -1, count = 9 Writer 0 (0x2de050): Removing dead entries Writer 0 (0x2de050): QueueMaintenance complete Writer 0 (0x2de050): Sending TimestepMetadata for timestep 8 (ref count 1), one to each reader Writer 0 (0x2de050): Sent timestep 8 to reader cohort 0 Writer 0 (0x2de050): ADDING timestep 8 to sent list for reader cohort 0, READER 0x36e1f0, reference count is now 2 Writer 0 (0x2de050): PRELOADMODE for timestep 8 non-default for reader , active at timestep 0, mode 1 Writer 0 (0x2de050): Sending a message to reader 0 (0x2e38d0) DP Reader 0 (0x2e38d0): Posted RDMA get for Writer Rank 0 for handle 0x382380 Writer 0 (0x2de050): SubRef : Writer-side Timestep 8 now has reference count 1, expired 0, precious 0 DP Reader 0 (0x2e38d0): Performing remote read of Writer Rank 0 at step 0 DP Reader 0 (0x2e38d0): Block address is 0x378340, with a key of l, and access key size 37. DP Reader 0 (0x2e38d0): Remote read target is Rank 0, EP = 0x7fffd43c0000 (Offset = 1244, Length = 160) DP Reader 0 (0x2e38d0): Posted RDMA get for Writer Rank 0 for handle 0x3823b0 DP Reader 0 (0x2e38d0): Performing remote read of Writer Rank 0 at step 0 DP Reader 0 (0x2e38d0): Block address is 0x378340, with a key of l, and access key size 37. DP Reader 0 (0x2e38d0): Remote read target is Rank 0, EP = 0x7fffd43c0000 (Offset = 292, Length = 20) DP Reader 0 (0x2e38d0): Posted RDMA get for Writer Rank 0 for handle 0x3823e0 Writer 0 (0x2de050): Reader 0 status Established has last released 4294967295, last sent 8 Writer 0 (0x2de050): QueueMaintenance, smallest last released = -1, count = 9 Writer 0 (0x2de050): Removing dead entries Writer 0 (0x2de050): QueueMaintenance complete DP Reader 0 (0x2e38d0): Performing remote read of Writer Rank 0 at step 0 DP Reader 0 (0x2e38d0): Block address is 0x378340, with a key of l, and access key size 37. DP Reader 0 (0x2e38d0): Remote read target is Rank 0, EP = 0x7fffd43c0000 (Offset = 406, Length = 40) DP Reader 0 (0x2e38d0): Posted RDMA get for Writer Rank 0 for handle 0x391910 DP Writer 0 (0x2de050): Providing timestep data with block 0x3c4640 and access key 3947504 DP Reader 0 (0x2e38d0): Performing remote read of Writer Rank 0 at step 0 DP Reader 0 (0x2e38d0): Block address is 0x378340, with a key of l, and access key size 37. DP Reader 0 (0x2e38d0): Remote read target is Rank 0, EP = 0x7fffd43c0000 (Offset = 548, Length = 80) DP Reader 0 (0x2e38d0): Posted RDMA get for Writer Rank 0 for handle 0x391bc0 DP Reader 0 (0x2e38d0): Performing remote read of Writer Rank 0 at step 0 DP Reader 0 (0x2e38d0): Block address is 0x378340, with a key of l, and access key size 37. DP Reader 0 (0x2e38d0): Remote read target is Rank 0, EP = 0x7fffd43c0000 (Offset = 192, Length = 10) Writer 0 (0x2de050): Reader 0 status Established has last released 4294967295, last sent 8 Writer 0 (0x2de050): QueueMaintenance, smallest last released = -1, count = 10 Writer 0 (0x2de050): Removing dead entries Writer 0 (0x2de050): QueueMaintenance complete DP Reader 0 (0x2e38d0): Posted RDMA get for Writer Rank 0 for handle 0x391720 Writer 0 (0x2de050): Sending TimestepMetadata for timestep 9 (ref count 1), one to each reader DP Reader 0 (0x2e38d0): Performing remote read of Writer Rank 0 at step 0 Writer 0 (0x2de050): Sent timestep 9 to reader cohort 0 DP Reader 0 (0x2e38d0): Block address is 0x378340, with a key of l, and access key size 37. Writer 0 (0x2de050): ADDING timestep 9 to sent list for reader cohort 0, READER 0x36e1f0, reference count is now 2 Writer 0 (0x2de050): PRELOADMODE for timestep 9 non-default for reader , active at timestep 0, mode 1 Writer 0 (0x2de050): Sending a message to reader 0 (0x2e38d0) DP Reader 0 (0x2e38d0): Remote read target is Rank 0, EP = 0x7fffd43c0000 (Offset = 722, Length = 40) DP Reader 0 (0x2e38d0): Posted RDMA get for Writer Rank 0 for handle 0x392140 Writer 0 (0x2de050): SubRef : Writer-side Timestep 9 now has reference count 1, expired 0, precious 0 Writer 0 (0x2de050): Reader 0 status Established has last released 4294967295, last sent 9 Writer 0 (0x2de050): QueueMaintenance, smallest last released = -1, count = 10 Writer 0 (0x2de050): Removing dead entries Writer 0 (0x2de050): QueueMaintenance complete Writer 0 (0x2de050): SstWriterClose, Sending Close at Timestep 9, one to each reader Writer 0 (0x2de050): Working on reader cohort 0 Writer 0 (0x2de050): Sending a message to reader 0 (0x2e38d0) Writer 0 (0x2de050): Reader 0 status Established has last released 4294967295, last sent 9 Writer 0 (0x2de050): QueueMaintenance, smallest last released = -1, count = 10 Writer 0 (0x2de050): Removing dead entries Writer 0 (0x2de050): QueueMaintenance complete DP Reader 0 (0x2e38d0): Performing remote read of Writer Rank 0 at step 0 DP Reader 0 (0x2e38d0): Block address is 0x378340, with a key of l, and access key size 37. DP Reader 0 (0x2e38d0): Remote read target is Rank 0, EP = 0x7fffd43c0000 (Offset = 864, Length = 80) DP Reader 0 (0x2e38d0): Posted RDMA get for Writer Rank 0 for handle 0x3923f0 DP Reader 0 (0x2e38d0): Performing remote read of Writer Rank 0 at step 0 DP Reader 0 (0x2e38d0): Block address is 0x378340, with a key of l, and access key size 37. DP Reader 0 (0x2e38d0): Remote read target is Rank 0, EP = 0x7fffd43c0000 (Offset = 1560, Length = 160) DP Reader 0 (0x2e38d0): Posted RDMA get for Writer Rank 0 for handle 0x392590 DP Reader 0 (0x2e38d0): Performing remote read of Writer Rank 0 at step 0 DP Reader 0 (0x2e38d0): Block address is 0x378340, with a key of l, and access key size 37. DP Reader 0 (0x2e38d0): Remote read target is Rank 0, EP = 0x7fffd43c0000 (Offset = 1880, Length = 160) DP Reader 0 (0x2e38d0): Posted RDMA get for Writer Rank 0 for handle 0x3926e0 DP Reader 0 (0x2e38d0): Performing remote read of Writer Rank 0 at step 0 DP Reader 0 (0x2e38d0): Block address is 0x378340, with a key of l, and access key size 37. DP Reader 0 (0x2e38d0): Remote read target is Rank 0, EP = 0x7fffd43c0000 (Offset = 2143, Length = 8) DP Reader 0 (0x2e38d0): Posted RDMA get for Writer Rank 0 for handle 0x392880 Reader 0 (0x2e38d0): Received a Timestep metadata message for timestep 1, signaling condition Reader 0 (0x2e38d0): Received a Timestep metadata message for timestep 2, signaling condition Reader 0 (0x2e38d0): Received a Timestep metadata message for timestep 3, signaling condition Reader 0 (0x2e38d0): Received a Timestep metadata message for timestep 4, signaling condition Reader 0 (0x2e38d0): Received a Timestep metadata message for timestep 5, signaling condition Reader 0 (0x2e38d0): Received a Timestep metadata message for timestep 6, signaling condition Reader 0 (0x2e38d0): Received a Timestep metadata message for timestep 7, signaling condition Reader 0 (0x2e38d0): Received a Timestep metadata message for timestep 8, signaling condition Reader 0 (0x2e38d0): Received a Timestep metadata message for timestep 9, signaling condition Reader 0 (0x2e38d0): Received a writer close message. Timestep 9 was the final timestep. Writer 0 (0x2de050): Waiting for timesteps to be released in WriterClose Writer 0 (0x2de050): IN TS WAIT, ENTRIES are Timestep 9 (exp 0, Prec 0, Ref 1), Count now 10 Writer 0 (0x2de050): IN TS WAIT, ENTRIES are Timestep 8 (exp 0, Prec 0, Ref 1), Count now 10 Writer 0 (0x2de050): IN TS WAIT, ENTRIES are Timestep 7 (exp 0, Prec 0, Ref 1), Count now 10 Writer 0 (0x2de050): IN TS WAIT, ENTRIES are Timestep 6 (exp 0, Prec 0, Ref 1), Count now 10 Writer 0 (0x2de050): IN TS WAIT, ENTRIES are Timestep 5 (exp 0, Prec 0, Ref 1), Count now 10 Writer 0 (0x2de050): IN TS WAIT, ENTRIES are Timestep 4 (exp 0, Prec 0, Ref 1), Count now 10 Writer 0 (0x2de050): IN TS WAIT, ENTRIES are Timestep 3 (exp 0, Prec 0, Ref 1), Count now 10 Writer 0 (0x2de050): IN TS WAIT, ENTRIES are Timestep 2 (exp 0, Prec 0, Ref 1), Count now 10 Writer 0 (0x2de050): IN TS WAIT, ENTRIES are Timestep 1 (exp 0, Prec 0, Ref 1), Count now 10 Writer 0 (0x2de050): IN TS WAIT, ENTRIES are Timestep 0 (exp 0, Prec 0, Ref 1), Count now 10 Writer 0 (0x2de050): The timesteps still queued are: 9 8 7 6 5 4 3 2 1 0 Writer 0 (0x2de050): Reader Count is 1 Writer 0 (0x2de050): Reader [0] status is Established salloc: Job 250522 has exceeded its time limit and its allocation has been revoked. srun: Job step aborted: Waiting up to 32 seconds for job step to finish. srun: Job step aborted: Waiting up to 32 seconds for job step to finish. srun: Job step aborted: Waiting up to 32 seconds for job step to finish. slurmstepd: error: *** STEP 250522.4+0 ON crusher012 CANCELLED AT 2023-01-18T14:38:38 DUE TO TIME LIMIT *** slurmstepd: error: *** STEP 250522.4+1 ON crusher059 CANCELLED AT 2023-01-18T14:38:38 DUE TO TIME LIMIT *** slurmstepd: error: *** STEP 250522.interactive ON crusher012 CANCELLED AT 2023-01-18T14:38:38 DUE TO TIME LIMIT *** srun: error: crusher012: task 0: Terminated srun: launch/slurm: _step_signal: Terminating StepId=250522.4+0 srun: launch/slurm: _step_signal: Terminating StepId=250522.4+1 srun: error: crusher059: task 0: Terminated eisen@crusher012:~/proj> srun: error: crusher012: task 0: Killed srun: launch/slurm: _step_signal: Terminating StepId=250522.interactive [eisen@login2.crusher ~]$