Problem parsing on single quotes in argument for Process.Start (dotnet Core, Linux console)

[MrM40]

(2019-07-21: rewrote this issues to simplify)

I'm executing console programs in Linux using

I'm executing programs on Linux with Process.Start.
Sometimes I need to provide single quotes in the arguments but I've not been able to succeed no matter how I escape the character.
A simplified example:

using System;
using System.Diagnostics;

namespace Execute_test
{
    class Program
    {
        static void Main(string[] args)
        {
            Process proc = new System.Diagnostics.Process();
            ProcessStartInfo pi = new ProcessStartInfo("ls");
            pi.Arguments = "-l '/tmp/'";
            proc.StartInfo = pi;
            proc.Start();
            do { System.Threading.Thread.Sleep(50); } while (proc.HasExited == false);
            Environment.Exit(0);
        }
    }
}

I do know I don't have to surround /tmp/ in single quote, but it's just to make a simple example (please don't suggest alternatives, that's not the issue!)
The err.out from this example is ls: cannot access "'/tmp/'": No such file or directory

I've tried to run this code in .net Core 2.2 and the newest .net Core 3.0.100-preview6-012264 in both C# and VB.

I also tried to use the ProcessStartInfo.ArgumentList it doesn't parse single quotes any better.

[EgorBo]

According to sources it follows https://docs.microsoft.com/en-us/cpp/cpp/parsing-cpp-command-line-arguments?view=vs-2017 rules

[tmds]

.NET Core added an ArgumentsList on ProcessStartInfo. You don't need to deal with escaping if you are using that.
You can Add the '-c' and 'ls -l /etc/ > /tmp/list.txt' to the ArgumentsList.

[MrM40]

thanks....I'll check it out :-)

[tmds]

@MrM40 Is this now working for you? Can the issue be closed?

[MrM40]

Well, kind of:-P
The ArgumentsList is not implemented in dotnet Standard, only in Core and framework (as fare I remember). Secondly I will claim it is a workaround and not the real fix. I still believe there is a problem/bug in the string-parsing.
So yes, it did find a way around it, but the bug it still there I assume.
But it sure is low-priority.

[tmds]

As mentioned by @EgorBo , this applies: https://docs.microsoft.com/en-us/cpp/cpp/parsing-cpp-command-line-arguments?view=vs-2017. Have you tried following those escape rules?

[MrM40]

Yes, doesn't work.
PI.FileName = "/bin/bash"
PI.Arguments = "-c \'ls -l /etc/ > /tmp/list.txt\'"

Result:
-l: 'ls: command not found

[tmds]

-l: 'ls: command not found

The single quotation is showing up here. Single quotation is also not mentioned in https://docs.microsoft.com/en-us/cpp/cpp/parsing-cpp-command-line-arguments?view=vs-2017. Try using \" instead of \'.

[MrM40]

The issue isn't really to get the command working, the issue is the string-parsing it supposed to parse whatever I tell it to parse! If I have a argument that need single quotation, I should be able to!

[tmds]

You want to start the executable /bin/bash and pass it two arguments: -c and ls -l /etc/ > /tmp/list.txt. Neither of these arguments contain a single quotation.

To do this with a shell, you type something like:

$ /bin/bash -c 'ls -l /etc/ > /tmp/list.txt'

The single quotations that show up here are meant for the shell. It uses them to figure out that ls -l /etc/ > /tmp/list.txt must be passed as a single argument.

When using ProcessStartInfo.Arguments you are also providing multiple arguments with a single string. How the Process class split that string into separate arguments is based on https://docs.microsoft.com/en-us/cpp/cpp/parsing-cpp-command-line-arguments?view=vs-2017.

You need to follow the rules of the thing that splits the string into separate arguments. ProcessStartInfo.Arguments is using a different scheme than a Linux shell.

[MrM40]

But I believe process should just parse -c 'ls -l /etc/ > /tmp/list.txt' as a single argument to /bin/bash, and then bash will see the string as two arguments, right?

I've tried this:
As one arguemnt:
PI.Arguments ="""-c 'ls -l /etc/ > /tmp/list.txt'"""
As two arguemnts:
PI.Arguments =""-c"" ""'ls -l /etc/ > /tmp/list.txt'""

It would help a lot if you could tell how the PI.Arguments =.... should look like if you believe it's working as expected?
Or am I still missing something.

[tmds]

But I believe process should just parse -c 'ls -l /etc/ > /tmp/list.txt' as a single argument to /bin/bash, and then bash will see the string as two arguments, right?

Process calls execve and needs to provide -c and ls -l /etc/ > /tmp/list.txt as separate items in the argv argument.

It would help a lot if you could tell how the PI.Arguments =.... should look like if you believe it's working as expected?

Try replacing the single quotes with double quotes: PI.Arguments = "-c \"ls -l /etc/ > /tmp/list.txt\""

[MrM40]

The ls.... command is just an example!
I have many different command that needs single quotes
You are again suggesting a work-around, using double quotes, again that not the issue
The issue is you should be able to put single-quotes in the argument.
And that doesn't seem to be the case. I just made the command as a simple example to make it clear that there seem to be something wrong with the parsting.
If you disagree, please tell how the argument should look like, you can use the ls.... example :-)

[tmds]

You want to start the application from .NET Core which you start on the command line as:

$ /bin/bash -c 'ls -l /etc/ > /tmp/list.txt'

Let's first figure out how many arguments are involved when starting the application. We'll use strace and trace for the execve call:

$ strace -e execve -qq /bin/bash -c 'ls -l /etc/ > /tmp/list.txt'
execve("/bin/bash", ["/bin/bash", "-c", "ls -l /etc/ > /tmp/list.txt"], 0x7ffe1e57c978 /* 71 vars */) = 0

As we can see in the output, two arguments are passed to bash: -c and ls -l /etc/ > /tmp/list.txt.

We need a way of passing the second argument so it gets treated as a whole (that is, not split at the spaces).

For bash, there are a number of options. One is to use single quotes, like /bin/bash -c 'ls -l /etc/ > /tmp/list.txt'.

For ProcessStartInfo.Arguments, we need to use double quotes.

static void Main(string[] args)
{
    Process.Start(
        new ProcessStartInfo
        {
            FileName = "/bin/bash",
            Arguments = "-c \"ls -l /etc/ > /tmp/list.txt\""
        }
    ).WaitForExit();

    if (File.Exists("/tmp/list.txt"))
    {
        System.Console.WriteLine("The file exists!");
    }
}

prints out:

The file exists!

[TSlivede]

I think this issue could be closed as duplicate of https://github.com/dotnet/corefx/issues/38483 because it essentially asks to make the behavior of ProcessStartInfo.Arguments platform dependent. Although https://github.com/dotnet/corefx/issues/38483 is newer, it has IMHO a much cleaner description.

@MrM40 Currently ' is not considered a quote in ProcessStartInfo.Arguments - it's just a normal character. (And this should IMO definitly stay this way)

[MrM40]

But the whole problem is, which is not covered by 38483, that the single quote ' is not parsed as a normal character. My simple example illustrate that I would say.

[TSlivede]

In https://github.com/dotnet/corefx/issues/23592#issuecomment-514215957 you wrote

require single quotes in the argument

I agree - and that is absolutely possible with the current behavior.

You also wrote:

single quotes are not parsed

I agree again, however they don't need to be parsed - they are just passed unchanged to the called program.

---

Anyway, I'd strongly recommend, that you use the new argument array API on linux - ProcessStartInfo.Arguments is just the Windows way of doing things. On Windows a called executable actually receives a single string, while on *nix it receives an array.

[TSlivede]

But the whole problem is, which is not covered by 38483, that the single quote ' is not parsed as a normal character. My simple example illustrate that I would say.

Which example do you mean, this one?

            ProcessStartInfo pi = new ProcessStartInfo("ls");
            pi.Arguments = "-l '/tmp/'";

What exactly do you mean by "parsed as a normal character"?

Can we agree, that

            ProcessStartInfo pi = new ProcessStartInfo("ls");
            pi.Arguments = "-l X/tmp/X";

searches for a file or folder X within a folder tmp within a folder X?

By the same logic your first example
searches for a file or folder ' within a folder tmp within a folder '?

I guess that is not what you want, so you actually don't want ' to be handled like any other char, do I understand you correctly?

[MrM40]

The command ls '/tmp/' just dir the /tmp folder.
You don't really need to use single quotes but I use it as an simple example, it was the simplest Linux command I could think of, and of the exact reason to not put focus on the command but the parsing issue (or whatever one would call it).
Bottom line, I've not found a way to execute a command with an argument containing '.

Are you takling about ProcessStartInfo.ArgumentList? I tried that and it didn't change much.
Or at least I'm doing it wrong.
How should I divide the argument '/tmp/'?
If you believe everything is working as it should how can I execute this very simple Linux command ls '/tmp/'from .Net Std. or Core.

[TSlivede]

    class Program
    {
        static void Main(string[] args)
        {
            Process proc = new System.Diagnostics.Process();
            ProcessStartInfo pi = new ProcessStartInfo("sh"){
                ArgumentList = {
                    "-c",
                    "ls '/tmp/'" 
                }
            };
            proc.StartInfo = pi;
            proc.Start();
            do { System.Threading.Thread.Sleep(50); } while (proc.HasExited == false);
            Environment.Exit(0);
        }
    }

works fine for me. If you want ' to be considered a quote, you need some executable, that considers it a quote - for example sh or bash.

[MrM40]

OMG!!!!
ProcessStartInfo pi = new ProcessStartInfo("ls"){ ArgumentList = {"'","/tmp/","'"} };
Pure beauty, thaaaaanks

[TSlivede]

No no no!
This is equivalent of executing ls "'" "/tmp/" "'" in bash. It works, but does not do what you want.
It lists the files of a folder named ' (twice) and lists the contents of /tmp/.

Please delete (or edit) your comment https://github.com/dotnet/corefx/issues/23592#issuecomment-514241549 as it could heavily confuse newcomers.

[TSlivede]

ProcessStartInfo pi = new ProcessStartInfo("ls"){ ArgumentList = {"/tmp/"} };
is enough - you don't need a ' in that case.

If you execute ls -l '/tmp/' '/path/with space/' in bash or sh, then ls never sees any ' - they are removed by the calling shell. ls just gets an array of

ls
-l
/tmp/
/path/with space/

[MrM40]

Hmmm.....not so easy :-(
Some of the real commands I need to run is:
dpkg-query -W -f=' ${db:Status-Status} ' mariadb*
and
virsh qemu-agent-command SRV01 '{"execute":"guest-ping"}'
I guess I could trough it around sh, it sure is the best workaround, but it would be more sexy to call the executable directly.

[jnm2]

@MrM40 Why aren't you doing ArgumentList = { "-W", "-f=' ${db:Status-Status} '", "mariadb*" }?

And ArgumentList = { "qemu-agent-command", "SRV01", "'{\"execute\":\"guest-ping\"}'" }?

[MrM40]

I'm working on it :-P (don't have sdk on the prod server where I run the commands)

[TSlivede]

@jnm2 I'm confused, shouldn't it be

ArgumentList = { "-W", "-f= ${db:Status-Status} ", "mariadb*" }

and

ArgumentList = { "qemu-agent-command", "SRV01", "{\"execute\":\"guest-ping\"}" }?

If you execute dpkg-query -W -f=' ${db:Status-Status} ' mariadb* in bash, the ' are not passed to the executable, as you can easily test by putting printf '%s\n' before the commands.
Running printf '%s\n' dpkg-query -W -f=' ${db:Status-Status} ' mariadb* in bash gives

dpkg-query
-W
-f= ${db:Status-Status}
mariadb*

as output, so ' is not contained in the arguments, if dpkg-query -W -f=' ${db:Status-Status} ' mariadb* is executed in bash.

[MrM40]

Yes. As it turn out when using the ArgumentList the qoutes are redundant.
This works for me:

In the Linux shell: dpkg-query -W -f=' ${db:Status-Status} ' mariadb*

ProcessStartInfo pi = new ProcessStartInfo("dpkg-query");
pi.ArgumentList.Add("-W");
pi.ArgumentList.Add("-f= ${db:Status-Status} ");
pi.ArgumentList.Add("mariadb*");`

In the Linux shell: virsh qemu-agent-command SRV04 '{"execute":"guest-ping"}'

ProcessStartInfo pi = new ProcessStartInfo("virsh");
pi.ArgumentList.Add("qemu-agent-command");
pi.ArgumentList.Add("SRV01");
pi.ArgumentList.Add("{\"execute\":\"guest-ping\"}");

[jnm2]

@TSlivede Yes, I wasn't thinking. Thanks!