Binary over sys.stdin question, sys.stdin is type TextIOWrapper, not FileIO?

[stephanelsmith]

Hello all,

Brief, I'm writing a CLI tool with micropython where I need to pipe in binary over stdin. On unix port, I'type of sys.stdin is a TextIOWrapper. Conversely, on ESP32, sys.stdin is a FileIO with sys.stdin.buffer available for use.

Why not FileIO which would provide text and buffered versions?

I have a naive workaround, this feels like a total hack, BUT, if I change is_text from true to false in extmod/vfs_posix_file.c, I no longer get unicode error, reading every raw byte value from stdin as expected.

STATIC const mp_stream_p_t vfs_posix_textio_stream_p = {
    .read = vfs_posix_file_read,
    .write = vfs_posix_file_write,
    .ioctl = vfs_posix_file_ioctl,
    .is_text = true,     <<<<============ change this from true to **false**
};

Naturally this breaks my compatibility and portability. Any other options (or even better, something completely obvious that I don't know yet! 😉 ) for binary of stdin on unix port?

Micropython rocks! 🚀

[bidrohini]

[stephanelsmith]

@bidrohini Are you an ai bot?

[jimmo]

@bidrohini This answer and the suggested code is not correct, as on unix sys.stdin does not have the buffer attribute and is not a binary stream (as you correctly point out, it has is_text set to true). It also does not answer the original question which had no mention of needing to work on ESP32.

As others have pointed out, your answers are suspiciously similar to the sort out output we see from LLMs like ChatGPT. We appreciate that you are trying to help but these responses are not actually answering people's questions and just adding noise to the discussion. Please stop doing this or I will need to block your account.

[stephanelsmith]

@jimmo I think this episode relates a bit to, https://github.com/orgs/micropython/discussions/11895
One thought, you mentioned the possibility of blocking/banning this account. From running my own organization, I would think it's a good idea to provide a clear path to disciplinary action in the code of conduct. Maybe not even need be more than 'All AI generated must clearly be identified as AI generated content or be blocked.'
https://github.com/micropython/micropython/blob/master/CODEOFCONDUCT.md

[jimmo]

@stephanelsmith I did update the welcome post yesterday https://github.com/orgs/micropython/discussions/9111 (As you can see, my preference is that AI content is just not posted at all).

I'm hesitant to outline an exact procedure because some situations require execptions, and in general (as per our code of conduct) just issuing a warning (with stated consequences of disregarding that warning) has been sufficient to resolve most problems. (And thank you @bidrohini for removing the post).

[jimmo]

@stephanelsmith

On unix port, I'type of sys.stdin is a TextIOWrapper.

This is the correct behavior. However what's missing is that it should have a .buffer attribute to access the underlying binary stream -- this is the CPython-compatible way to do this (see the note at the end of the sys.stdin docs -- https://docs.python.org/3/library/sys.html#sys.stderr ).

Conversely, on ESP32, sys.stdin is a FileIO with sys.stdin.buffer available for use.

Confusingly, despite being named FileIO, this object (on all the embedded ports) is actually a TextIOWrapper-like implementation. It is a text stream, and supports the .buffer attribute to get the underlying binary stream.

I don't know the history, but I'm guessing it's called "FileIO" just to save the additional code size for "TextIOWrapper", but if you look at shared/runtime/sys_stdio_mphal.c you can see it's implemented with is_text set to true.

Why not FileIO which would provide text and buffered versions?

I think the solution is just to provide the buffer attribute on the unix version (in vfs_posix_file.c), which should be do-able using basically the same approach as sys_stdio_mphal.c.

[stephanelsmith]

Exactly, the .buffer is what I want on the unix port; sounds like there's a learning and hopefully a small contribution opportunity for me here. I'll take this up and see if I can come up with something PR-able.