Porting interactive functionality to Symfony

[wouterj]

I asked about this on Twitter last week, but I realize that is not the best place to discuss this. I'm really digging the interactive prompts (arrow-based choice lists & completion dropdowns) of this library! This is something I was trying to introduce to Symfony Console for many years (but lacking the stty skills needed for this).

Would it be an idea to see if this part of Prompts can be ported back to Symfony Console, a library you already depend on in the Laravel ecosystem and this library? That way, a larger part of the PHP ecosystem can have this much better terminal UX (including tooling used in the Laravel ecosystem like Composer and PHPstan).

The way I see it, this functionality is a perfect fit for the Question classes in Symfony Console. Laravel Prompts can then use the question classes, and add the custom "Laravel style" (plus any other extra integration, like the helper functions) on top of it.

Of course, given the code in this library is MIT, I can always try this proposal without your consent. But I'm looking forward to hear your ideas about this (e.g. did you try this and reached a major blocker already, is this just "not worth it" for your time, are you happy to try this out, or can I stalk a bit when experimenting with this myself 😁 )

[jessarcher]

Hey @wouterj! Sorry I missed your Twitter post - thank you for sharing it!

I would love to see something like this in Symfony. The challenge is supporting native Windows (i.e. php.exe without WSL) and potentially other environments. Currently, we're configuring the Prompts in Laravel to fall back to Symfony in Windows because I couldn't find a way to make the arrow keys work, amongst other things.

I've briefly explained the challenges I faced with Windows at laravel/docs#8924 (comment).

In Symfony, it's already possible to use the arrow keys as an alternative mode for selection. However, as the entries can still be typed manually, it's still useable without the arrow or tab keys (albeit more frustratingly). After trying to implement Laravel Prompts in Windows, I gained a new understanding and appreciation for why Symfony's prompts are the way they are. It appears they have made an admirable decision to sacrifice some usability in exchange for supporting more environments.

Unless we can figure out how to solve the problems mentioned in my linked comment, I think the only alternative would be exploring alternative UI approaches that allow for graceful degradation.

For what it's worth, this library was designed to work with any PHP project, not just Laravel. However, the Symfony fallback implementation is only configured out of the box when used with Laravel, as originally this package did not have any Symfony dependencies. We added Symfony Console as a dependency towards the end when we needed a shared output buffer to know how many newlines to output before and after each prompt when interleaved with other console output in Laravel.

It's also worth noting that the Node packages that inspired me seem to have similar problems in native Windows, as does ReactPHP's stdio package:

• https://github.com/natemoo-re/clack/tree/main/packages/prompts (this one was unusable on Windows)
• https://github.com/terkelg/prompts (I think this one was somewhat usable due to its UI)
• https://github.com/clue/reactphp-stdio#install

[driesvints]

Hey @wouterj. Was the fact that you could already use the package in Symfony projects enough or are you looking for some other (deeper) integration? Please let us know what more thought you have here 🙂

[ppastercik]

Hi, @jessarcher.

I don't know where to put this. But after seeing this issue, I took a close look at laravel/prompts. I work on Windows and it irritates me that the console use of PHP on the Windows system is limited, so I like the idea that it could work on Windows (even without WSL). I've tried some things like updating PHP source codes, and it works! See below for more detailed information.

You are correct that PHP on native Windows by default has no useful way to implement the correct behavior (reading arrow keys, etc). So I thought I'd rewrite some PHP source codes. I gave it a shot even though the last time I did C/C++ was many years ago at school. So after locally updating some PHP source code (8.2.9) and building it, then updating the laravel/prompts code to use the newly implemented PHP features, and it works. It works on CMD, Git Bash and PowerShell command lines (I haven't tried other Windows command lines yet). But as I wrote, if it's going to work, the PHP source code has to be updated, and I don't know if the PHP developers would agree to the necessary code update. But this is just a functionality extension, so it could work, just define the right way to implement these features.

For a better description of what the problems were, I list them here along with the possible solutions I came up with:

• The ability to not to echo the entered characters
  • Now this is not possible in PHP on Windows.
  • But by modifying the PHP source code it is possible to add this option. It is possible by calling the SetConsoleMode function, which will clear the ENABLE_ECHO_INPUT flag (it is set by default).
    • Therefore, I added a custom function to PHP to set/unset it for a specific console input stream.
• The ability to read every key, not just the line after Enter
  • Now this is not possible in PHP on Windows.
  • But by modifying the PHP source code it is possible to add this option. It is possible by calling the SetConsoleMode function, which will clear the ENABLE_LINE_INPUT flag (it is set by default).
    • Therefore, I added a custom function to PHP to set/unset it for a specific console input stream.
• The ability to read special keys, i.e. arrow keys and even special keys like Backspace etc.
  • Now this is not possible in PHP on Windows.
  • But by modifying the PHP source code it is possible to add this option. It is possible by calling the SetConsoleMode function, which sets the ENABLE_VIRTUAL_TERMINAL_INPUT flag (it is unset by default).
    • Therefore, I added a custom function to PHP to set/unset it for a specific console input stream.
• The ability to read Ctrl+C and not process it
  • Now this is not possible in PHP on Windows.
  • But by modifying the PHP source code it is possible to add this option. It is possible by calling the SetConsoleMode function, which will clear the ENABLE_PROCESSED_INPUT flag (it is set by default).
    • Therefore, I added a custom function to PHP to set/unset it for a specific console input stream.
• Working with multiple command line types in Windows
  • CMD command line
    • The input is a standard console stream, so I can use the functions added to PHP mentioned above.
  • PowerShell command line
    • The input is a standard console stream, so I can use the functions added to PHP mentioned above.
  • Git Bash command line (differs in how it is called)
    • Called with php artisan app:prompt
      • Input is a standard console stream, so I can use the functions added to PHP mentioned above.
    • Called with php.exe artisan app:prompt
      • The input is a pipe stream, so the functions mentioned above are unusable, but in this case I can use stty because Git bash has its own implementation of stty. However, I think this implementation also uses SetConsoleMode to set the console behavior, because when I tested my PHP functions to set the console behavior and then call stty, my settings were overwritten, so only one approach must be used. So it has to check if the input is console or pipe stream.
• Returned character for Enter key
  • On Linux it's the \n character.
  • On Windows it's the \r character, so I had to update laravel/prompts to handle both characters to submit values.

This would have to be properly tested for other command lines on the Windows platform to work. Furthermore, a suitable implementation of adding the functionality to PHP would also have to be resolved, because now it's only a proof of concept. And there is the question of whether to somehow merge it for Windows and Linux (wire up stty or something), or just add functionality for Windows.

Extending PHP to include the functionality may then take some time, if the PHP developers agree to it, and it is possible that even then it would only be from a new version of PHP.

So I'd like to ask about your opinion if it's worth implementing, or is it a waste of time?

Alternatively, do you have any insights on this? Or are there other things for Windows that are necessary for this, but I missed it?

PS:
I mistakenly thought it was necessary to add a non-blocking read of the console input, so the info below is not related to this issue, but it is interesting.

It's not possible in PHP on Windows now. But by updating PHP source code I managed to implement a workaround for non-blocking reading of console input in the PHP on the Windows platform. I use the PeekConsoleInput function to see if any character input exists, so we can safely use the read function without blocking. However, there is a problem with some keys when using console line input. For example, the F4 key, which brings up some modal window in CMD that I didn't know about and discovered it now. So for this to work 100%, it needs more work.

[jessarcher]

Hi @ppastercik,

Wow! Well done! This sounds very promising and exciting, and I think it's worth implementing.

It could be hard to get it into PHP, though. But I think we could make a convincing argument for it, given the current limitations.

Failing that, It sounds like this could be included in a PHP extension, but that isn't the greatest developer experience.

Alternatively, I wonder whether we could achieve this with the FFI extension, which might be a bit less annoying to install than a third-party extension.

[ppastercik]

Now I tried the same procedure as with stty, i.e. for example calling a special program that would set the console behavior. But when the program finishes, the console behaviour regarding echo and CTRL+C reverts to normal behaviour, but the arrow (and other special characters) settings persist, so it's useless unless all settings are preserved :(. But it's strange that something is preserved.

So the preferred way is to add this to core PHP if you can, otherwise as a PHP extension. Or an FFI extension as you suggest, but I don't know much about that. I've looked at the documentation, but haven't studied it in depth, so I don't know yet if it's possible to use PHP's internal structures, which would help. But I can give it a try if it doesn't work otherwise.

I tried searching about this issue on their bug tracker (https://bugs.php.net/) and github issues (https://github.com/php/php-src/issues), but I didn't find anything relevant. @jessarcher do you know of anything relevant where I should post my suggestion? Or should I start a new issue with this suggestion (I don't want to make a duplicate issue if there is one, so I'd rather ask if you know of anything)?

[jessarcher]

Now I tried the same procedure as with stty, i.e. for example calling a special program that would set the console behavior. But when the program finishes, the console behaviour regarding echo and CTRL+C reverts to normal behaviour, but the arrow (and other special characters) settings persist, so it's useless unless all settings are preserved :(. But it's strange that something is preserved.

Interesting idea! There is a precedent for this in Symfony too. On Windows, it calls a bundled hiddeninput.exe file to handle hidden input (i.e. password inputs).

https://github.com/symfony/console/blob/baad2fac42f37fef925482313291044143d3e00b/Helper/QuestionHelper.php#L417

With the stty implementation we need to manually restore the console behaviour afterwards - was that not possible with your approach?

prompts/src/Prompt.php

Lines 83 to 86 in a109cc5

	register_shutdown_function(function () {
	$this->restoreCursor();
	static::terminal()->restoreTty();
	});

[ppastercik]

With the stty implementation we need to manually restore the console behaviour afterwards - was that not possible with your approach?

@jessarcher I think there was a misunderstanding with my stty example. Maybe I write it wrong.

In Windows, let's name the program wtty.exe, which should behave like stty. In the sample code, I omit to preserve the previous console settings and restore the console settings, because it's not main problem. And this is my PHP test code:

<?php
    // "-line" parameter - means not to read by lines, but by characters
    // "-echo" parameter - means disabling echo of inputs
    // "-processed" parameter - means not to process "CTRL+C", etc. but read them as characters
    // "vt100" parameter - means to read also arrow keys, etc. as characters
    shell_exec("wtty.exe -line -echo -processed vt100");

    // But even if I call the previous program "wtty.exe", only the "vt100" setting is preserved. So next `fread` read whole line, echo inputs, not read "CTRL+C" as character, just only read arrow keys, etc. as characters. :(
    $read = fread(STDIN, 1024);

So it is not the right way if it is not possible to keep the settings after wtty.exe is ended.

Interesting idea! There is a precedent for this in Symfony too. On Windows, it calls a bundled hiddeninput.exe file to handle hidden input (i.e. password inputs).

Next, as you say, I looked at Symfony's approach with hiddeninput.exe. I think it is possible to create a program that returns read character, but I think this is not optimal due to having to call this program to read each character.

Therefore, I think it would be best to build this support into PHP or as an extension.

So if this will be supported within PHP, example of settings of console behaviours and reading keys could be (functions are inspired by an already implemented function to support vt100 settings for the console output stream):

<?php
    $defaultEcho = sapi_windows_input_echo(STDIN);
    $defaultLine = sapi_windows_input_by_line(STDIN);
    $defaultProcessed = sapi_windows_input_processed(STDIN);
    $defaultVt100 = sapi_windows_input_vt100_support(STDIN);

    sapi_windows_input_echo(STDIN, false);
    sapi_windows_input_by_line(STDIN, false);
    sapi_windows_input_processed(STDIN, false);
    sapi_windows_input_vt100_support(STDIN, true);

    // Now it's reading by char, not echo input, even read "CTRL+C" as characters, with arrow keys, etc. as characters.
    $readedChar = fread(STDIN, 1024);

    sapi_windows_input_echo(STDIN, $defaultEcho);
    sapi_windows_input_by_line(STDIN, $defaultLine);
    sapi_windows_input_processed(STDIN, $defaultProcessed);
    sapi_windows_input_vt100_support(STDIN, $defaultVt100);

Or can be implemented with using constants for settings console behaviour, so setting console and reading could be:

<?php
    $defaultMode = sapi_windows_input(STDIN);

    sapi_windows_input(STDIN, $defaultMode & ~SAPI_WINDOWS_INPUT_ECHO & ~SAPI_WINDOWS_INPUT_BY_LINE & ~SAPI_WINDOWS_INPUT_PROCESSED | SAPI_WINDOWS_INPUT_VT100_SUPPORT);

    // Now it's reading by char, not echo input, even read "CTRL+C" as characters, with arrow keys, etc. as characters.
    $readedChar = fread(STDIN, 1024);

    sapi_windows_input(STDIN, $defaultMode);

For the implementation there is another the question whether to implement all console modes supported by the SetConsoleMode function or only those that are needed.

[jessarcher]

Thanks for the clarification, @ppastercik.

I've read through the SetConsoleMode docs you posted, and it seems very strange that some settings persist and others don't, given they all operate on a console input buffer. I'd expect it to be all or nothing!

How are you determining which input handle to pass to the hConsoleHandle parameter? Is it possible it's not the same as STDIN in PHP?

In any case, support in PHP would be ideal. With your proposed changes, Windows support would be better than Linux and macOS, where stty is required.

To clarify my understanding, these are the flags we need to toggle, their default, and their purpose:

Mode	Operates on	Default	Purpose
ENABLE_ECHO_INPUT	Input	Enabled	Writes input to the screen buffer
ENABLE_LINE_INPUT	Input	Enabled	Waits for carriage return
ENABLE_VIRTUAL_TERMINAL_INPUT	Input	Disabled	Converts input into sequences (for arrow keys, backspace, etc.)
ENABLE_PROCESSED_INPUT	Input	Enabled	System processes Ctrl+C instead of input buffer

I don't have any preference for the PHP API. Exposing all console mode flags seems ideal, but perhaps there are good reasons not to.

I'm not sure who at PHP is responsible for this, but I think a good next step would be to talk to whoever that is so that we can figure out what would have the best chance of a successful RFC.

[ppastercik]

I've read through the SetConsoleMode docs you posted [...] I'd expect it to be all or nothing!

@jessarcher Yes, I think the same thing. But maybe there is some procedure in Windows OS that resets the console input stream mode when the process starts or ends. In the documentation of the SetConsoleMode function it says:

When a console is created, all input modes except ENABLE_WINDOW_INPUT and ENABLE_VIRTUAL_TERMINAL_INPUT are enabled by default.

The Console modes documentation also says:

A command-line application should expect that other command-line applications may change the console mode at any time and may not restore it to its original form before control is returned. Additionally, we recommend that all command-line applications should capture the initial console mode at startup and attempt to restore it when exiting to ensure minimal impact on other command-line applications attached to the same console.

So that's weird, because the console is not newly created (if we use current console input stream for nested process), and they even said that console mode changes can be made by other command line applications. And when I tested it, it doesn't preserve all or nothing, but only ENABLE_VIRTUAL_TERMINAL_INPUT setting.

How are you determining which input handle to pass to the hConsoleHandle parameter? Is it possible it's not the same as STDIN in PHP?

I implemented it the same way as the sapi_windows_vt100_support function, but with the difference that it allows you to set STDIN (and not STDOUT). So I used (HANDLE)_get_osfhandle(fileno), where fileno is obtained from the stream using the stream cast to fd. I also tried using GetStdHandle(STD_INPUT_HANDLE) to get this handle and it has the same value as my previous solution.

So I tried to test the console mode setup using a nested process and tried the console mode setup behavior. I wrote a test PHP script (with my PHP compilation with supporting methods), which I think proves that I am using the correct handle, because the console mode settings are set in the nested process for the parent process. And the settings are overwritten in the parent process. Here is the code of the test.php script:

<?php
    function echo_current_input_state($header) {
        echo "$header\n";

        var_dump(sapi_windows_input_echo(STDIN));
        var_dump(sapi_windows_input_by_line(STDIN));
        var_dump(sapi_windows_input_processed(STDIN));
        var_dump(sapi_windows_input_vt100_support(STDIN));
    }

    echo_current_input_state('default after start');

    // Set a custom settings to verify that it is passed to the nested process.
    sapi_windows_input_echo(STDIN, false);
    sapi_windows_input_by_line(STDIN, true);
    sapi_windows_input_processed(STDIN, false);
    sapi_windows_input_vt100_support(STDIN, false);

    echo_current_input_state('set custom settings (-echo, line, -processed, vt100)');

    // Process that prints the current console mode settings,
    // then sets own custom console settings (-echo, -line, -processed, vt100).
    $cmdProc = "C:\\php-sdk\\phpdev\\vs16\\x64\\php-8.2.9-src\\x64\\Release_TS\\php.exe -r \"var_dump(sapi_windows_input_echo(STDIN)); var_dump(sapi_windows_input_by_line(STDIN)); var_dump(sapi_windows_input_processed(STDIN)); var_dump(sapi_windows_input_vt100_support(STDIN)); sleep(2); sapi_windows_input_echo(STDIN, false); sapi_windows_input_by_line(STDIN, false); sapi_windows_input_processed(STDIN, false); sapi_windows_input_vt100_support(STDIN, true); sleep(2);\"";

    $proc = proc_open($cmdProc, [STDIN, ['pipe', 'w']], $pipes);
    
    sleep(1);

    echo "default settings readed from started nested process\n";

    echo fread($pipes[1], 1024);

    echo_current_input_state('settings after nested process started');

    sleep(2);

    echo_current_input_state('settings after nested process changed settings (-echo, -line, -processed, vt100)');

    sleep(4);

    proc_close($proc);

    echo_current_input_state('settings after end of nested process');

Here's how I ran it:

C:\php-sdk\phpdev\vs16\x64\php-8.2.9-src\x64\Release_TS\php.exe test.php

And it outputs:

default after start
bool(true)
bool(true)
bool(true)
bool(false)
set custom settings (-echo, line, -processed, vt100)
bool(false)
bool(true)
bool(false)
bool(false)
default settings readed from started nested process
bool(false)
bool(true)
bool(false)
bool(false)
settings after nested process started
bool(false)
bool(true)
bool(false)
bool(false)
settings after nested process changed settings (-echo, -line, -processed, vt100)
bool(false)
bool(false)
bool(false)
bool(true)
settings after end of nested process
bool(true)
bool(true)
bool(true)
bool(true)

So, as you can see, the console input stream setting is preserved after the internal process is started. But when the internal process ends, the console input stream setting will be changed back to the default (except for the ENABLE_VIRTUAL_TERMINAL_INPUT setting, which will be retained).

But there is another strange behavior regarding the ENABLE_VIRTUAL_TERMINAL_INPUT setting. When I run the same command again, even though it is ENABLE_VIRTUAL_TERMINAL_INPUT set after the previous command is finished (pressing the right arrow will print ^[[C in the console), the output in the default after start section shows ENABLE_VIRTUAL_TERMINAL_INPUT is not set again. So when the process is started directly from the command line, the ENABLE_VIRTUAL_TERMINAL_INPUT setting is also reset.

So, if any nested process is run in a PHP script that has an associated PHP script console input stream, the console input stream setting is reset after the process is finished (except for the ENABLE_VIRTUAL_TERMINAL_INPUT setting). So developers will need to be careful when using the mode settings together with nested process and possibly after nested process end set the console mode again.

To clarify my understanding, these are the flags we need to toggle, their default, and their purpose: [...]

Yes, you are right.

I'm not sure who at PHP is responsible for this, but I think a good next step would be to talk to whoever that is so that we can figure out what would have the best chance of a successful RFC.

I'll try some more ways to see if this behavior can be circumvented (but I'm afraid it can't). And then I will try to test other Windows command lines to see if they will work with the modified laravel/prompts (with using my PHP compilation).

I'll see what comes out of this, but if I don't discover any new information, I'll create an issue with the information found so far and the possibility of implementing it in the php-src repository.

[andyduke]

I encountered a similar problem and was able to solve it using FFI on Windows. For cross-platform I made a ConsoleKeyboard package, maybe it will help solve this problem. If there are any problems or suggestions, you can post them here.

[ppastercik]

Great job @andyduke. I tried it and it worked. But I found some bugs that I hope can be fixed. For example, pasting text from the clipboard doesn't work (not process each character pasted from clipboard). I'll write up the bugs in your package issues.

Your solution confirms that although my solution will not be implemented in PHP itself, we have a solution, but with a dependency on the FFI extension.

However, I prefer to implement it in PHP itself without dependency on the FFI extension, so that users can use the interactive console in Windows without having to enable the FFI extension.

Today, I finally finished testing my solution on different types of command lines in Windows (and it worked on all the ones I tried). So, in the next few days I will create an issue with my proposed solution in the php-src repository and see if the PHP developers agree to implement it in PHP.

If they don't agree, we can use your solution (after fixing the bugs). Or if they agree with my solution, maybe we can use your solution in the meantime until my solution is implemented in PHP itself.

[driesvints]

Hi all. Wanted to check in if it's still worth keeping this open or not? If there aren't moving parts I feel like it's best that we close this and move on.

[wouterj]

Over time, this issue has been completely hijacked by Windows support. I'm not sure about the state of that topic 🙂 (or the value the Laravel team sees in this)

I would still love to have seen some of the interactivity tools from Prompts as new Questions in the Symfony Console's question helper, to help the broader PHP community with these modern features. For what it's worth, we think it's OK to keep the current state as a fallback for Windows, and use the better interaction in terminals that have support.

[driesvints]

Hi @wouterj. Sounds good. We'll leave this open a bit longer. @ppastercik let's focus on Windows support in a different issue to keep this one on topic.

[driesvints]

@crazywhalecc heya, as mentioned above. Let's please focus this on the topic at hand and keep everything Windows related in a separate issue. Feel free to open one if needed.

[wouterj]

Let's close this issue. It's apparent that nobody at Laravel's side is interested in contributing part of this back to Symfony Console. So the most logical step is to wait for someone in the Symfony community really wanting this feature and contributing it to Symfony themselves.

Thanks for the discussion!

[driesvints]

It's apparent that nobody at Laravel's side is interested in contributing part of this back to Symfony Console.

Hey @wouterj. That's a bit harsh as that's definitely not true. We'd definitely want to open up the prompts package to the larger PHP community. We just struggle to find time to look into it more deeply as we have a lot of other things going on at Laravel.

In fact, as soon as I find the time I'm going to look into getting rid of the illuminate/collection dependency so the package is even less dependant on Laravel. I really want to get that in before we tag a v1.0

Of course, this package is open-source and we'd very much welcome PR's that help open up this package further to other frameworks as long as we also make sure it doesn't breaks existing (Laravel) apps.

[wouterj]

Hi @driesvints. Sorry if my message came across harsh, this was not my intention. There was no plan or update in this issue for almost a year, so I expected there was no intention to act here. I also realize that it's much more likely that someone from the non-Laravel users of the Console component will be motivated to work on this because they want slick CLI interfaces as well.

For me, that's what open source is about: someone that wants something, should put in the work. So I didn't want to leave this open on the Laravel side (were I don't expect someone to be motivated about this, because this works just fine already for Laravel users).

Anyway, cool to hear you have plans to one day decouple this package more from Laravel!

[driesvints]

Thanks @wouterj. I appreciate the feedback 😄