Reading files from (piped) standard input, with correct cancellation support

[amoerie]

Hi, thanks for the great work so far!

I'm still struggling with one concept though. I want to create a CLI app that can receive input from Console.In, so it can be combined with other cli tools. It should also support proper cancellation. This combination (reading from standard input + cancellation) is where I'm stuck.

Main usage

Example preferred usage (from powershell, but that doesn't really matter)

gci *.txt | MyApp.exe

Fallback 1

If possible, I'd also like to provide an alternative so that the files can optionally be passed in as a positional argument, or if that's not possible, as a named argument. Something like this:

MyApp.exe C:\file1.txt C:\file2.txt C:\file3.txt

or perhaps

MyApp.exe --files=C:/file1.txt C:/file2.txt C:\file3.txt

Fallback 2

Furthermore, if MyApp.exe is called without any arguments, it should read the files from standard input with support for cancellation.

> MyApp.exe
> C:/file1.txt
> C:/file2.txt
> C:/file3.txt
> CTRL + C (app should immediately close)

Using just System.CommandLine (not Dragonfruit), what would MyApp look like? Can I use any of the existing Option infrastructure to implement this?

I've tried the following so far:

    public class Program
    {
        public static Task<int> Main(string[] args)
        {
            var filesArgument = new Argument("files")
            {
                Arity = ArgumentArity.ZeroOrMore,
                ArgumentType = typeof(IEnumerable<FileInfo>),
            };
            var filesOption = new Option(new[] {"-f", "--files"}, "Process these files")
            {
                Argument = filesArgument,
            };
            
            var rootCommand = new RootCommand("Process files")
            {
                // Make positional argument work
                filesArgument,
                
                // Make named argument work
                filesOption,
            };

            rootCommand.Handler =
                CommandHandler.Create<IEnumerable<FileInfo>?, CancellationToken>(ProcessFiles);

            return rootCommand.InvokeAsync(args);
        }

        private static void ProcessFiles(IEnumerable<FileInfo>? files, CancellationToken cancellationToken)
        {
            files ??= ReadFilesFromStandardInput(cancellationToken);

            foreach (var file in files)
            {
                cancellationToken.ThrowIfCancellationRequested();
                
                Console.WriteLine("Hello this is a file: " + file);
            }
        }

        private static IEnumerable<FileInfo> ReadFilesFromStandardInput(CancellationToken cancellationToken)
        {
            cancellationToken.ThrowIfCancellationRequested();
            
            string? line;
            while ((line = Console.In.ReadLine()) != null)
            {
                cancellationToken.ThrowIfCancellationRequested();

                if (File.Exists(line))
                    yield return new FileInfo(line);
            }
        }
    }

The above implementation gets very close to what I want, but I cannot seem to get cancellation support working correctly, because the process is blocked at "Console.In.ReadLine".

A simple summary:

• Support for piped input via standard input
• Support for manually typing file names one by one via standard input
• Support for named argument "files"
• Support for positional argument
• Support for proper cancellation when pressing CTRL + C

I've tried searching for issues or anything in the docs, but I couldn't find anything that answers my questions.

Thanks for your time.

[jonsequitur]

Related: #275.

[amoerie]

I did a little further digging and found that Console.ReadAsync or Console.ReadLineAsync always blocks the application. (ReadLineAsync does not even have an overload with a CancellationToken)
The CancellationToken from the model binding is properly immediately cancelled when pressing CTRL + C, it is up to the consumer code to immediately drop everything when this happens. It's not your fault that Console.ReadLineAsync is not cancellable.

The only solution that I've found is to wrap this in Task.Run. I'm on mobile right now but I can provide a sample later on.

If I may say so, I actually did expect that this library would provide something out of the box to model bind standard input to. For now, it seems that this is still a manual process.

[amoerie]

Update: this class seems to be working nicely, immediately respecting the cancellation token. Unfortunately the Console.ReadLineAsync process is still blocked, but that's okay because a little further down the line, the entire program exits anyway. I wouldn't use this outside CLI applications:

    public class LinesFromConsoleInputReader
    {
        public async IAsyncEnumerable<string> Read([EnumeratorCancellation] CancellationToken cancellationToken)
        {
            var cancellationTcs = new TaskCompletionSource<string?>(TaskCreationOptions.RunContinuationsAsynchronously);
            await using var registration = cancellationToken.Register(() => cancellationTcs.SetCanceled());
            Task<string?> cancellationTask = cancellationTcs.Task;
            
            while (!cancellationToken.IsCancellationRequested)
            {
                // A double 'await' is necessary here.
                // The first await is for the result of Task.WhenAny, which returns the Task that completes first
                // The second await extracts the result of the task returned by Task.WhenAny
                // In the normal case, this is the task returned from ReadLineAsync, so awaiting that returns a string?
                // In the cancelled case, this returns the task completion source which is set to canceled, so awaiting that throws an OperationCanceledException
                var line = await await Task.WhenAny(
                    // Task.Run is necessary here, because Console.In.ReadLineAsync blocks the thread
                    Task.Run(() => Console.In.ReadLineAsync(), cancellationToken), 
                    cancellationTask);
                
                if (line == null)
                    yield break;

                yield return line;
            }
        }
    }