[>=2.0.0] ValueError: Data must not be a string.

[AxelVoitier]

Hello,

Since the release of 2.0.0, all my CI builds have been failing on the same error from httpie :-(.

I use httpie to post test results to AppVeyor API:

after_test:
    # Upload test results to AppVeyor
    - http -f POST https://ci.appveyor.com/api/testresults/junit/${APPVEYOR_JOB_ID} file@test-reports/pytest/junit.xml

Now it fails with:

Running "after_test" scripts
http -f POST https://ci.appveyor.com/api/testresults/junit/${APPVEYOR_JOB_ID} file@test-reports/pytest/junit.xml
http: error: ValueError: Data must not be a string.
Command exited with code 1
Build failed

As AppVeyor has a way to block a build execution and log on the machine by ssh, I tried to repeat the command manually. And there, quite strangely, it works...:

appveyor@appveyor-vm:~/projects/etka-core$ http -f POST https://ci.appveyor.com/api/testresults/junit/${APPVEYOR_JOB_ID} file@test-reports/pytest/junit.xml

HTTP/1.1 204 No Content
Cache-Control: no-cache
Date: Mon, 20 Jan 2020 08:57:47 GMT
Expires: -1
Pragma: no-cache
Strict-Transport-Security: max-age=31536000
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block

More strangely, if I then unblock the build and let the original http command execute, then it fails in the same way...

I first though it would be related to a coincident change in AppVeyor environment. But then retried, blocking the build again but this time downgrading to httpie 1.0.3:

appveyor@appveyor-vm:~$ pip uninstall httpie
Uninstalling httpie-2.0.0:
  Would remove:
    /home/appveyor/venv3.7.1/bin/http
    /home/appveyor/venv3.7.1/bin/https
    /home/appveyor/venv3.7.1/lib/python3.7/site-packages/httpie-2.0.0.dist-info/*
    /home/appveyor/venv3.7.1/lib/python3.7/site-packages/httpie/*
Proceed (y/n)? y
  Successfully uninstalled httpie-2.0.0
appveyor@appveyor-vm:~$ pip install "httpie<2.0.0"
Collecting httpie<2.0.0
  Downloading https://files.pythonhosted.org/packages/da/08/22487f04aa2cb831c5e02c9286f1d145f81b06bf9bbf1e73b70d2dc15d7f/httpie-1.0.3-py2.py3-none-any.whl (59kB)
     |████████████████████████████████| 61kB 2.3MB/s 
Requirement already satisfied: Pygments>=2.3.1 in ./venv3.7.1/lib/python3.7/site-packages (from httpie<2.0.0) (2.5.2)
Requirement already satisfied: requests>=2.21.0 in ./venv3.7.1/lib/python3.7/site-packages (from httpie<2.0.0) (2.22.0)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in ./venv3.7.1/lib/python3.7/site-packages (from requests>=2.21.0->httpie<2.0.0) (3.0.4)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in ./venv3.7.1/lib/python3.7/site-packages (from requests>=2.21.0->httpie<2.0.0) (1.25.7)
Requirement already satisfied: idna<2.9,>=2.5 in ./venv3.7.1/lib/python3.7/site-packages (from requests>=2.21.0->httpie<2.0.0) (2.8)
Requirement already satisfied: certifi>=2017.4.17 in ./venv3.7.1/lib/python3.7/site-packages (from requests>=2.21.0->httpie<2.0.0) (2019.11.28)
Installing collected packages: httpie
Successfully installed httpie-1.0.3

And then unblock the script and let the original command execute, and it worked:

Build lock file has been deleted. Resuming build.
http -f POST https://ci.appveyor.com/api/testresults/junit/${APPVEYOR_JOB_ID} file@test-reports/pytest/junit.xml

HTTP/1.1 204 No Content
Cache-Control: no-cache
Date: Mon, 20 Jan 2020 09:10:03 GMT
Expires: -1
Pragma: no-cache
Strict-Transport-Security: max-age=31536000
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block

It seems the error "ValueError: Data must not be a string." is comming from requests as this old thread indicate: https://stackoverflow.com/questions/27553082/valueerror-data-must-not-be-a-string
But as you can see in the logs above, when downgrading httpie, requests stays the same.

So I guess the issue comes from a mix between how httpie now invokes requests, and some special conditions in AppVeyor build environment (which make things harder to debug...).

Any though?

Cheers,
Axel

[jkbrzt]

@AxelVoitier thanks for the report.

The reason why it fails on AppVeyor is that when you invoke HTTPie from that script, then HTTPie’s stdin is redirected. And redirected stdin is not compatible with extra request data specified on the CLI. It’s easy to reproduce it with:

echo 'file data'  > file.txt
echo 'stdin data' | http --form httpbin.org/post foo@file.txt

http: error: ValueError: Data must not be a string.

---

The solution is to include --ignore-stdin in the command (see https://httpie.org/doc#scripting for more details):

$ echo 'ignored stdin data' | http --ignore-stdin --form httpbin.org/post foo@file.txt

---

The bug here is that HTTPie doesn't fail with the following error as it does when you try to send some non-form-file data with redirected stdin on top of that:

$ echo 'stdin data' | http --form httpbin.org/post foo@file.txt data=data

http: error: Request body (from stdin or a file) and request data (key=value) cannot be mixed. 
             Pass --ignore-stdin to let key/value take priority.

[AxelVoitier]

Hello,

I can confirm it works with --ignore-stdin. Thanks :).

Cheers,
Axel