[PR196 1/3+] Making stage reproduction more portable, making resultspace sourcing more robust

[jbohren]

• addresses issues:
  • clarify function name: [PR196 1/3] New asyncio-based execution engine #249 (comment)
  • don't use xargs: [PR196 1/3] New asyncio-based execution engine #249 (comment)
• adds env verb, which mimics the UNIX env command, but with the
  ability to read environment variables from stdin
• uses typeset builtin instead of cmake -E environment, which makes it robust to environment variables with line breaks
• defines repro commands as the piping of the two calls:

cd CWD; catkin build --get-env PKG | catkin env -si CMD; cd -

Full catkin env -h help:

usage: catkin env [-h] [-i] [-s]
                  [NAME=VALUE [NAME=VALUE ...]] [COMMAND] [ARG [ARG ...]]

Run an arbitrary command in a modified environment.

positional arguments:
  NAME=VALUE            Explicitly set environment variables for the
                        subcommand. These override variables given to stdin.

optional arguments:
  -h, --help            show this help message and exit
  -i, --ignore-environment
                        Start with an empty environment.
  -s, --stdin           Read environment variable definitions from stdin.
                        Variables should be given in NAME=VALUE format.

command:
  COMMAND               Command to run. If omitted, the environment is printed
                        to stdout.
  ARG                   Arguments to the command.

[wjwwood]

Should catkin build --get-env PKG return newline separated environment variables? Right now it seems to return whitespace separated entries.

[jbohren]

Should catkin build --get-env PKG return newline separated environment variables? Right now it seems to return whitespace separated entries.

Yeah, that's how it's designed, but hold off on merging this for the moment.

[jbohren]

@wjwwood This is good to merge, once you've tested it.

[jbohren]

@wjwwood You can get a human-readable printout like this:

catkin build --get-env PKGNAME | catkin env -si

We could add another option to get a human readable output without the pipes, but I don't know how important that is at the moment.

[jbohren]

@wjwwood This should be good to review. It should be robust to the PTY exhaustion issues on OS X, and we can re-enable emulate_tty once there's a better solution for that (in osrf_pycommon, I presume).

It also uses an asynchronous lock for the install lock, and treats it as a generic "locked resource" for a given stage.

Also, I ran into the same problem with execute_process which caused it prematurely terminate before all output had been received in the resultspace loading (note here). Because of that I switched it back to using subprocess.Popen. I think this indicates another bug in the execute_process of osrf_pycommon.

[jbohren]

Also, since catkin build --get-env emulates typeset -px, it prints out environment variables in a human-readable way.

[wjwwood]

Does this need to stay in? What callbacks do you expect might take longer?

[jbohren]

The default is 100ms. Many callbacks take upwards of 300ms, which causes Trollius to generate warnings.

[wjwwood]

Ok.