[splash] need workaround to efficiently read job stdio output from large jobs

[garlick]

From conversation with @trws:

Scaling issues running large jobs wtih stdio redirected to a file (or interpreted by front end) have been discussed in #1406.

A workaround for this poor scaling is to avoid stdio redirection during the job (e.g. do not use -O file option to flux-submit), then copy the job output from the KVS to a file afterwards. Currently this would be done with something like:

#!/bin/bash -e

JOBID=$(flux submit "$@" | sed -e 's/Submitted jobid //')
# wait for job to complete
flux wreck attach ${JOBID} >output.$JOBID

This has the advantage of not competing with the big job for KVS and messaging resources, but it still does not scale well for a large number of tasks because flux wreck attach sets up the same set of parallel kz watchers that are used for in-job redirection.

One thought is to modify flux wreck attach so that it walks all existing data first, and only sets up the iowatchers if the job has not reached a terminal state.

[garlick]

I'm working on a PR to libkz that defers installation of the KVS watch until all the data already in the KVS has been read (with continuations). If EOF is encountered, then the watch is avoided entirely.

This should allow flux wreck attach to extract stdio from a completed job without internally creating any kvs watchers.

[garlick]

This is mostly done with the merge of #1424. however there was also a request by @trws for a flag to ignore EOF and terminate a read when the last block is read, so holding it open pending that.

[grondo]

This is mostly done with the merge of #1424. however there was also a request by @trws for a flag to ignore EOF and terminate a read when the last block is read, so holding it open pending that.

I ran into something similar with the job error log streams, which aren't typically terminated with EOF.
It would be nice if there was a way to do this from within a reactor loop, but there is probably no way to cleanly do that. I therefore resorted to plain kz_open and while kz_read != EAGAIN. Unfortunately this is completely anathema to the way flux-wreck attach works (and the ioplex stuff) so there might need to be a completely new command (or some sort of no-async mode for ioplex objects)

[garlick]

I was thinking of something like a KZ_FLAGS_NOFOLLOW or similar flag for kz_open that would cause kz_read to return EOF once the last "block" is consumed.

[grondo]

Ok, that sounds reasonable, I was initially not understanding how the latest flux_kvs_lookup_get had any context about whether it had fetched the last block in the fully asynchronous mode (before it went back to sleep in the reactor). However, looking again I see it is obvious in lookup_next so I can implement that instead of working around it in the client code.