Test runner always returns 0, environment variables are not exported to parent shell

[nwhetsell]

Currently, the test runner always returns 0:

max-test/ruby/test.rb

Line 291 in 7097a11

exit 0 if !@noexit

This means that, for example, if max-test is used in GitHub Actions, the action always succeeds regardless of whether any tests fail.

A potential workaround might be to check the result of the MAXTEST_PASS environment variable:

max-test/ruby/test.rb

Lines 275 to 277 in 7097a11

	# export results so a caller of this script is able to access the summary for e.g. automated email delivery
	ENV['MAXTEST'] = estring # log
	ENV['MAXTEST_PASS'] = testpass # general pass/fail

However, environment variables set this way are not (and indeed can’t be) exported to the parent process, as can be seen by running:

ruby -e 'ENV["XXX_RUBY_ENV_TEST_XXX"] = "success"'
echo "XXX_RUBY_ENV_TEST_XXX is \"$XXX_RUBY_ENV_TEST_XXX\""

[benbrackenc74]

In the current state of things, the test runner always succeeds, and it is the expectation that the results will need to be parsed.

As far as the "environment variables" used here, in this case the ENV class is used to pass on the variable to a parent ruby process, nothing more. So if you had a parent ruby script that spawned test.rb, it should have access to these variables.

[nwhetsell]

@benbrackenc74 Thank you for your response.

So if you had a parent ruby script that spawned test.rb, it should have access to these variables.

I’m not sure I follow this. Whether I run (on macOS)

ruby -e 'system("ruby test.rb /Applications/Max.app")' \
     -e 'puts ENV.keys'

or

ruby -e 'require "open3"' \
     -e 'stdout, stderr = Open3.capture3("ruby test.rb /Applications/Max.app")' \
     -e 'puts stdout' \
     -e 'puts stderr' \
     -e 'puts ENV.keys'

or

ruby -e 'pid = spawn("ruby test.rb /Applications/Max.app")' \
     -e 'Process.wait pid' \
     -e 'puts ENV.keys'

I do not see MAXTEST or MAXTEST_PASS among ENV.keys. Is there some other way to spawn a Ruby process that I’m missing?

[benbrackenc74]

I've done something like this before using require:

in env_sub.rb:
ENV['AN_ENV_VARIABLE'] = 'cool!'

in env_main.rb:

require './env_sub.rb'
puts ENV['AN_ENV_VARIABLE']

[nwhetsell]

Thank you again for responding.

The problem with using require here is that test.rb ends with a call to exit:

max-test/ruby/test.rb

Line 291 in 7097a11

exit 0 if !@noexit

This means something like

ruby -e 'require("./test.rb")' \
     -e 'puts ENV.keys' \
     /Applications/Max.app

cannot work. Ruby exits immediately after the call to require, and the puts is never reached.

It seems like the most straightforward way to fix this (that is, to get a usable exit code) is to patch the last line of test.rb.