Command prompt escaping in Process.quote_windows

[HertzDevil]

From #13567 we know that Process.quote_windows, which only performs the opposite of CommandLineToArgvW, does not handle metacharacter escaping for the command prompt. We also need both kinds of escaping, and there isn't a one-size-fits-all solution. So I am proposing the following addition to the standard library:

class Process
  def self.quote_windows(args : Enumerable(String), *, c_runtime : Bool = true, shell : Bool = false)
    # if `c_runtime` is true, do what the method does right now; then
    # if `shell` is true, prepend a `^` to every metacharacter
    # (`(`, `)`, `%`, `!`, `^`, `"`, `<`, `>`, `&`, `|`), and also
    # raise if any `\n` appears
  end

  def self.quote_windows(args : String, *, c_runtime : Bool = true, shell : Bool = false)
    quote_windows({args}, c_runtime: c_runtime, shell: shell)
  end
end

Process.quote_windows(%q(foo <"bar">), c_runtime: false)              # => %q(foo <"bar">)
Process.quote_windows(%q(foo <"bar">), c_runtime: false, shell: true) # => %q(foo ^<^"bar^"^>)
Process.quote_windows(%q(foo <"bar">))                                # => %q("foo <\"bar\">")
Process.quote_windows(%q(foo <"bar">), shell: true)                   # => %q(^"foo ^<\^"bar\^"^>^")

Process.quote is not supposed to be used in platform-specific scenarios, so it doesn't have to expose those new parameters.

This still leaves some questions unanswered:

• Should we continue to expect that the backtick is portable?
• Should we continue to expect that Process.quote inside a backtick is portable?
• Do these parameters' defaults for .quote_windows also hold for .quote?

[oprypin]

And the big question that I keep insisting on...

• Should we assume that the "cmd" shell is the shell and deserves to be tied to a boolean shell: true

It might be cool to instead expand Process.run to accept an enum as the shell argument, which would then have implementations for sh, cmd, windows backed by corresponding quote implementations.

As for backticks, I don't have much hope of them ever doing anything sensible...

[HertzDevil]

Sometimes I've been thinking what the shell would be for a hypothetical MinGW-based toolchain MSYS2-based build environment. Crystal programs built with it can definitely expect the availability of sh.exe as a default POSIX shell. So maybe replacing shell: true with an enum isn't a bad idea after all...?

[HertzDevil]

#15111 establishes the use of cmd.exe as the default shell even for x86_64-windows-gnu programs built inside MSYS2's UCRT64 environment, because we are still targetting the Win32 API. The MSYS environment will use /bin/sh but then we won't be targetting Win32.

Reading https://stackoverflow.com/a/4095133 it seems environment variable substitution occurs before the special rules for ^, which explains why this won't be enough:

# only CRT escaping is applied at the moment
# `lpApplicationName` is nil
# `lpCommandLine` is `cmd.exe /c foo.cmd ^%cd^%`
# `foo.cmd` contains `echo %1`
Process.run("cmd.exe", ["/c", "foo.cmd", "^%cd^%"], output: :inherit, env: {"cd^" => "1 && calc.exe"})

Maybe this is why the workaround mentioned in #14536 is necessary?

[oprypin]

We don't need to spend any time thinking what happens in this scenario:

Process.run("cmd.exe", ["/c", ...])

(i.e. passing an array of args)

because it's complete nonsense in the first place. cmd.exe simply doesn't accept an array of arguments, it accepts a single command line.
There is no possible level of workarounds that would make this make any sense.

In #14557 we simply made it so that an equivalent scenario doesn't get created implicitly (in supported scenarios). But if someone spells that out explicitly, that's totally on them.

[oprypin]

#15111 establishes the use of cmd.exe as the default shell even for x86_64-windows-gnu programs

I didn't understand, in which way does it establish that?

[HertzDevil]

Then what about Process.run("cmd.exe /c foo.cmd %cd%", shell: true)? Is it Crystal::System::Process or the user's responsibility to do the correct shell escaping in place of %cd%?

[oprypin]

"cmd.exe /c foo.cmd %cd%" is a way to run a Batch script with the content foo.cmd %cd%. I'd say it's a user's responsibility to ensure they wrote a correct Batch script and whether they meant for the variable to be expanded or not. If there's a way to safely escape values to be inserted into Batch syntax, we could provide a helper for that.

[oprypin]

And it seems in your example you've shown that there actually isn't a way to ensure that an arbitrary string will not result in variable expansion in the Batch syntax.

Say we want to pass the literal string "%cd%" to a process.

Currently we can achieve it like this:

`some_process #{Process.quote("%cd%")}`

With cmd involved, there's no way to do that.

`cmd.exe /c some_process #{quote_batch("%cd%")}`

If the quote_batch function does nothing then this is an expansion of the variable %cd%
If it attempts to turn that string into ^%cd^% then this is an expansion of the variable %cd^% - do I understand correctly?

[oprypin]

But hold on - according to the same link, it seems to say that %% is a way to escape a single % character. Maybe that can still be achieved, then?