suppress Windows SHELL env error message - SPARK-53149

[philwalk]

Summary

This PR suppresses an error message when running in a Windows SHELL environment bash.exe session. It should not affect the execution path or behavior on Windows or any other OS.

This is a minor change that should have no effect except when running in Windows SHELL environments.

• MSYS2
• cygwin
• Git-for-Windows

Specifically, it should have no effect in normal usage in these environments:

• Linux
• OSX
• Windows WSL

The last part of script bin/load-spark-env.sh tests for whether running in foreground or background by calling ps -o stat= and checking output for +. The purpose is to configure SPARK_BEELINE_OPTS for non-interactive environments if running in the background and not reading from a pipe. See SPARK-53149.

This code causes Windows shell environments to print an error message to stdout, due to a less capable ps.exe tool without support for the ps -o stat= -p $$ expression.

AFAIK, there is no equivalent expression in any of these environments (MSYS2, cygwin64, Git-for-Windows). However, the error message can be suppressed by redirecting stderr to /dev/null, reducing the negative impact.

What changes were proposed in this pull request?

Currently, the last section of bin/load-spark-env.sh looks like this:

if [[ ( ! $(ps -o stat= -p $$) =~ "+" ) && ! ( -p /dev/stdin ) ]]; then
  export SPARK_BEELINE_OPTS="$SPARK_BEELINE_OPTS -Djline.terminal=jline.UnsupportedTerminal"
fi

This PR redirects STDERR to /dev/null, hiding the error message.

if [[ ( ! $(ps -o stat= -p $$ 2>/dev/null) =~ "+" ) && ! ( -p /dev/stdin ) ]]; then
  export SPARK_BEELINE_OPTS="$SPARK_BEELINE_OPTS -Djline.terminal=jline.UnsupportedTerminal"
fi

Does this PR introduce any user-facing change?

One possible side-effect: If running in Ubuntu or Windows WSL/Ubuntu and the procps package has not been installed, this has the effect of suppressing the error message bash: ps: command not found. The same effect is possible on other systems if not properly configured with required packages.

This is not an issue on macOS because ps is included by default.

There doesn't appear to be a way to reliably detect whether running in the foreground or background in a Windows SHELL environment, so the execution path is to always append to SPARK_BEELINE_OPTS except when stdin is a pipe. This has the effect of disabling command line editing and other niceties for Windows SHELL clients, but does provide support non-interactive clients. In any event, this is the current behavior for those clients, and this PR doesn't change that, but it seems like a reasonable if not ideal default behavior.

Why are the changes needed?

Because the error message indicates an unsupported feature in the SHELL environments (rather than a fixable error), it is misleading and unnecessary.

How was this patch tested?

Manualy verified the functional equivalence of the old and new conditional expressions, in each of the following environments:

• macOS
• Windows WSL/Ubuntu
• Linux Mint/Ubuntu
• MSYS2
• Cygwin64

The manual test was based on running the following test script in each environment. No error messages were produced.

#!/bin/bash
function doTest {
  local old new
  if [[ ( ! $(ps -o stat= -p $$) =~ "+" ) && ! ( -p /dev/stdin ) ]]; then
    old=true
  else
    old=false
  fi

  if [[ ( ! $(ps -o stat= -p $$ 2>/dev/null) =~ "+" ) && ! ( -p /dev/stdin ) ]]; then
    new=true
  else
    new=false
  fi
  if [ $old != $new ]; then
    echo "error: old[$old] != new[$new]" 1>&2
  fi
}
echo
doTest
echo
doTest &

Was this patch authored or co-authored using generative AI tooling?

No

[pan3793]

I verified the behavior on macOS zsh, it works as expected now.

[github-actions]

We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!