Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix Windows Terminal output for non-UTF-8 byte sequences #292

Closed
lepapareil opened this issue Oct 12, 2021 · 3 comments · Fixed by #302
Closed

Fix Windows Terminal output for non-UTF-8 byte sequences #292

lepapareil opened this issue Oct 12, 2021 · 3 comments · Fixed by #302
Assignees
@fabricereix
Labels
bug Something isn't working windows
Milestone
1.4.0

Comments

@lepapareil
Copy link
Collaborator

I create a file named test.hurl containing the sample described by https://hurl.dev/docs/samples.html#getting-data:

GET https://example.net/news
User-Agent: Mozilla/5.0 
Accept: */*
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
Connection: keep-alive

I use hurl on my windows over cmd or powershell and i get:

> hurl test.hurl

thread 'main' panicked at 'writing bytes to console: Error { kind: InvalidData, message: "Windows stdio in console mode does not support writing non-UTF-8 byte sequences" }', packages\hurl\src\main.rs:448:18
stack backtrace:
   0:     0x7ff6dd3e4def - <unknown>
   1:     0x7ff6dd3fb87a - <unknown>
   2:     0x7ff6dd3dfcf8 - <unknown>
   3:     0x7ff6dd3e7846 - <unknown>
   4:     0x7ff6dd3e7329 - <unknown>
   5:     0x7ff6dd3e7ea5 - <unknown>
   6:     0x7ff6dd3e7a91 - <unknown>
   7:     0x7ff6dd3e570f - <unknown>
   8:     0x7ff6dd3e79e9 - <unknown>
   9:     0x7ff6dd448ec0 - <unknown>
  10:     0x7ff6dd448fd3 - <unknown>
  11:     0x7ff6dd189313 - <unknown>
  12:     0x7ff6dd1b6bf6 - <unknown>
  13:     0x7ff6dd1b6c8c - <unknown>
  14:     0x7ff6dd3e8406 - <unknown>
  15:     0x7ff6dd190067 - <unknown>
  16:     0x7ff6dd43fcf0 - <unknown>
  17:     0x7ff989e67c24 - BaseThreadInitThunk
  18:     0x7ff98a50d721 - RtlUserThreadStart
@lepapareil lepapareil added the bug Something isn't working label Oct 12, 2021
@tbolon
Copy link

tbolon commented Oct 12, 2021

Same error when requesting against https://www.google.com:

GET https://www.google.com/

HTTP/1.1 200

PS C:\...> hurl .\GoogleTests.hurl
<!doctype html><html itemscope="" itemtype="http://schema.org/WebPage" lang="fr"><head><meta content="text/html; charset=UTF-8" http-equiv="Content-Type"><meta content="/logos/doodles/2021/paulette-nardals-125th-birthday-6753651837109209.4-l.png" itemprop="image"><meta content="Il y a 125thread 'main' panicked at 'writing bytes to console: Error { kind: InvalidData, message: "Windows stdio in console mode does not support writing non-UTF-8 byte sequences" }', packages\hurl\src\main.rs:447:18
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

@fabricereix
Copy link
Collaborator

This is something to do with the Terminal.
It works when hurl output is redirected to a file.
We are going to look into it.

@fabricereix
Copy link
Collaborator

fabricereix commented Oct 16, 2021 •
edited
Loading

While this bug can be easily reproduced locally, it is more difficult to automatically validate it in the integration tests.
This is probably due to the way Hurl is run, through the python subprocess module.

This bug occurs for the following combined conditions:

  • Output in terminal (not to a file/pipe)
  • Windows
  • Invalid utf8 sequence

Related to the fix (#302) , I have added a specific case for Windows + Output in terminal,
in which a string is created from the API String::from_utf8_lossy
and printed to the output.

echo "GET https://www.google.com" | hurl is now printing to the terminal.

Relating to the https://example.net/news example, there is also no error when printed to the terminal.

The output is readable when running
echo "GET https://example.net/news" | hurl

But not from the previous example

GET https://example.net/news
User-Agent: Mozilla/5.0 
Accept: */*
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
Connection: keep-alive

This is due to the Accept-Encoding header.
The returned output is gziped and printed as such in the terminal.

 hurl --verbose .\test.hurl
* fail fast: true
* insecure: false
* follow redirect: false
* max redirect: 50
* ------------------------------------------------------------------------------
* executing entry 1
*
* Cookie store:
*
* Request
* GET https://example.net/news
* User-Agent: Mozilla/5.0
* Accept: */*
* Accept-Language: en-US,en;q=0.5
* Accept-Encoding: gzip, deflate, br
* Connection: keep-alive
*
* request can be run with the following curl command:
* curl 'https://example.net/news' -H 'User-Agent: Mozilla/5.0' -H 'Accept: */*' -H 'Accept-Language: en-US,en;q=0.5' -H 'Accept-Encoding: gzip, deflate, br' -H 'Connection: keep-alive'
*
> GET /news HTTP/1.1
> Host: example.net
> User-Agent: Mozilla/5.0
> Accept: */*
> Accept-Language: en-US,en;q=0.5
> Accept-Encoding: gzip, deflate, br
> Connection: keep-alive
>
< HTTP/1.1 404 Not Found
< Content-Encoding: gzip
< Age: 460751
< Cache-Control: max-age=604800
< Content-Type: text/html; charset=UTF-8
< Date: Sat, 16 Oct 2021 17:48:23 GMT
< Expires: Sat, 23 Oct 2021 17:48:23 GMT
< Last-Modified: Mon, 11 Oct 2021 09:49:12 GMT
< Server: ECS (nyb/1D20)
< Vary: Accept-Encoding
< X-Cache: 404-HIT
< Content-Length: 648
<
*
a#(�Y�y���:@�����<�%nZc�d�Fi��i�i�i�=���P���t�߻B�#7����]x�ﱐ���������)f�c�\�3�_�o����m�>�ť���:(��ʚ[�>�:_�E�i1q�޷�w�ϣ�F{Ԟ
�\�a�����,9|���r0����w¦�̡��S.a�����'t�pKx;����/Q��y��9���E��&���+�2˅���\���-s5�fC���.転o����?�ǡ%���Մ_������I�bB�1��ś�b��p�L`i,���Km4NA�)��É�[���j��t�~��QU���D�c��$d��
τ[��f�I$�<1v��!]�O3*n��H��v�Qe,t��Ht|m:�����4t�R1����S���-Z%�#G���!ό�U�
A���x���4���{���A��(�|o��

Using the option --compressed automatically decompress the content, and makes the output readable in the terminal.
We may have to change this default behavior.

@fabricereix fabricereix linked a pull request Oct 17, 2021 that will close this issue
Fix Windows Terminal output for non-UTF-8 byte sequences #302
Merged
@fabricereix fabricereix added this to the 1.4.0 milestone Oct 17, 2021
@fabricereix fabricereix changed the title thread 'main' panicked at 'writing bytes to windows terminal console (cmd and powershell) Fix Windows Terminal output for non-UTF-8 byte sequences Oct 17, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@fabricereix fabricereix

Labels
bug Something isn't working windows
Projects
None yet
Milestone
1.4.0
Development

Successfully merging a pull request may close this issue.

Fix Windows Terminal output for non-UTF-8 byte sequences Orange-OpenSource/hurl
3 participants
@tbolon @fabricereix @lepapareil