python-msgpack test fails when building on my machine due to encoding

[nh2]

Issue description

When I recompile python3Packages.msgpack (based on 18.03) on my NixOS 18.03 server, as opposed to fetching it from the binary cache, I get:

	Testing 16-bit string: object 'ccccccccccccccccccccccc...
	Testing 32-bit string: object 'bbbbbbbbbbbbbbbbbbbbbbb...
E
======================================================================
ERROR: test_pack_single (test_umsgpack.TestUmsgpack)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/nix-build-python3.6-u-msgpack-python-2.4.1.drv-0/u-msgpack-python-2.4.1/test_umsgpack.py", line 355, in test_pack_single
    print("nh2 err Testing: object %s" % (obj_repr, ), file=sys.stdout )
UnicodeEncodeError: 'ascii' codec can't encode character '\xe9' in position 31: ordinal not in range(128)

======================================================================
ERROR: test_unpack_single (test_umsgpack.TestUmsgpack)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/nix-build-python3.6-u-msgpack-python-2.4.1.drv-0/u-msgpack-python-2.4.1/test_umsgpack.py", line 382, in test_unpack_single
    (name, obj_repr if len(obj_repr) < 24 else obj_repr[0:24] + "..."))
UnicodeEncodeError: 'ascii' codec can't encode character '\xe9' in position 41: ordinal not in range(128)

----------------------------------------------------------------------
Ran 17 tests in 0.404s

FAILED (errors=2)

This is due to some terminal encoding weirdness.

The line that fails is this printing of the name of the test currently being run:

https://github.com/vsergeev/u-msgpack-python/blob/38c712520092128caa795762bb2c2fdf30275a06/test_umsgpack.py#L346-L347

It happens when the printout contains the letter é:

https://github.com/vsergeev/u-msgpack-python/blob/38c712520092128caa795762bb2c2fdf30275a06/test_umsgpack.py#L84

Interestingly, the issue happens only on stdout, not on stderr: If I insert prints like this:

            print(sys.stdout.encoding)
            print(sys.stderr.encoding)
            print("nh2 out Testing: object %s" % (obj_repr, ), file=sys.stderr )
            print("nh2 err Testing: object %s" % (obj_repr, ), file=sys.stdout )

then the out print succeeds but the err one crashes:

ANSI_X3.4-1968
ANSI_X3.4-1968
nh2 out Testing: object 'Allagb\xe9'
Traceback (most recent call last):
  File "/tmp/nix-build-python3.6-u-msgpack-python-2.4.1.drv-0/u-msgpack-python-2.4.1/test_umsgpack.py", line 355, in test_pack_single
    print("nh2 err Testing: object %s" % (obj_repr, ), file=sys.stdout )
UnicodeEncodeError: 'ascii' codec can't encode character '\xe9' in position 31: ordinal not in range(128)

Note how it succeeds on stderr but fails on stdout even though the encoding is apparently the same.

Why does this happen when I build on my server but not on Hydra?

Might this be some configuration of the outer terminal is this run from?

[nh2]

For reference on my server:

# echo $LANG
en_US.UTF-8

# locale
locale: Cannot set LC_CTYPE to default locale: No such file or directory
locale: Cannot set LC_MESSAGES to default locale: No such file or directory
locale: Cannot set LC_ALL to default locale: No such file or directory
LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=en_US.UTF-8

[nh2]

Looks like I can make the test pass by setting PYTHONIOENCODING=utf-8 (got it from here before running the test.

[FRidh]

Nixpkgs manual:

Unicode issues can typically be fixed by including glibcLocales in buildInputs and exporting LC_ALL=en_US.utf-8.

[nh2]

Perhaps relevant, another python package has this:

nixpkgs/pkgs/applications/misc/buku/default.nix

Lines 30 to 32 in 5453997

	preCheck = ''
	# Fixes two tests for wrong encoding
	export PYTHONIOENCODING=utf-8

Added here with comment "Another issue was due to locale issues".

@infinisil Do you remember if this was necessary only on your machine, or also for Hydra?

[nh2]

Unicode issues can typically be fixed by including glibcLocales in buildInputs and exporting LC_ALL=en_US.utf-8

@FRidh I don't get what the issue is though: Why should this issue arise when nix-building on my NixOS server, but not on Hydra?

[nh2]

Maybe it's a sandboxing thing that changes things about stdout encoding?

I tried

# nix-store --realise /nix/store/hv3rf9ypck00nhy65dd1jwc272ycqhcb-python3.6-u-msgpack-python-2.4.1.drv --option build-use-sandbox true

but it fails just as much as leaving --option build-use-sandbox true away.

[nh2]

@FRidh Also the package in question already has this:

nixpkgs/pkgs/development/python-modules/u-msgpack-python/default.nix

Lines 18 to 20 in 5453997

	LC_ALL="en_US.UTF-8";
	buildInputs = [ glibcLocales ];

since your original commit in bec4794#diff-7530dfc8d434ca1b15be565c346a4a98R18

[nh2]

This patch correspondingly makes it work for me, but I don't like it because I don't understand why it's necessary:

Author: Niklas Hambüchen <mail@nh2.me>
Date:   Sun Nov 11 14:39:48 2018 +0100

    u-msgpack-python: Workaround for #50231 encoding issue

diff --git a/pkgs/development/python-modules/u-msgpack-python/default.nix b/pkgs/development/python-modules/u-msgpack-python/default.nix
index bab0bc28850..4a88c99ac4d 100644
--- a/pkgs/development/python-modules/u-msgpack-python/default.nix
+++ b/pkgs/development/python-modules/u-msgpack-python/default.nix
@@ -20,6 +20,9 @@ buildPythonPackage rec {
   buildInputs = [ glibcLocales ];
 
   checkPhase = ''
+    # See https://github.com/NixOS/nixpkgs/issues/50231
+    export PYTHONIOENCODING=utf-8
+
     ${python.interpreter} -m unittest discover
   '';
 

[nh2]

Hmm, compiling futher, I'm getting this with more python packages:

______________________ TestConnection.test_wantWriteError ______________________

self = <tests.test_ssl.TestConnection object at 0x7ffff07b6790>

    def test_wantWriteError(self):
        """
            `Connection` methods which generate output raise
            `OpenSSL.SSL.WantWriteError` if writing to the connection's BIO
            fail indicating a should-write state.
            """
        client_socket, server_socket = socket_pair()
        # Fill up the client's send buffer so Connection won't be able to write
        # anything.  Only write a single byte at a time so we can be sure we
        # completely fill the buffer.  Even though the socket API is allowed to
        # signal a short write via its return value it seems this doesn't
        # always happen on all platforms (FreeBSD and OS X particular) for the
        # very last bit of available buffer space.
        msg = b"x"
        for i in range(1024 * 1024 * 4):
            try:
                client_socket.send(msg)
            except error as e:
                if e.errno == EWOULDBLOCK:
                    break
                raise
        else:
            pytest.fail(
>               "Failed to fill socket buffer, cannot test BIO want write")
E           Failed: Failed to fill socket buffer, cannot test BIO want write

tests/test_ssl.py:2593: Failed
==================== 13 failed, 478 passed in 7.57 seconds =====================
builder for '/nix/store/5riaz9x6cm60vmngi93hcnyfyn7z2d8w-python2.7-pyOpenSSL-17.5.0.drv' failed with exit code 1

[nh2]

Perhaps what's happening is that this issue arises because some dependent package wasn't built with the sandbox on (sneaking in an impurity), and enabling the sandbox for these Python packages is already too late?

To test this hypothesis, I'm now recompiling everything up from glibc with the sandbox on, and will report when that's done.

[nh2]

To test this hypothesis, I'm now recompiling everything up from glibc with the sandbox on, and will report when that's done.

DIdn't work :( still getting the same error.

#32656 also suggests the sandbox as the issue, and at some point the problem just went away for them. So odd ...

[nh2]

OK, here I have a thing for people to try.

NIX_PATH=nixpkgs=https://github.com/NixOS/nixpkgs/archive/19fc6daa537.tar.gz nix-build -E 'with import <nixpkgs> {}; python27Packages.pyopenssl' --check

fails for me, but leaving away --check downloads it from cache.nixos.org.

Can anybody try whether they can reproduce that?

[nh2]

@srhb tried the above for me but could not reproduce it.

But I fooled myself: That is because for PyOpenSSL, there isn't only the above unicode error, but also pyca/pyopenssl#768 on my machine (which I still haven't figured out why).

And I think my issue here in general was that I fooled myself another time:

I was recompiling my system with a glibc that had 1 patch added that I wrote myself (for malloc_info, so it should really have no effect on anything). But I added that patch in a patches = [...] section here in the glibc/default.nix file. I did not spot that the above did callPackage ./common.nix with my patches = as argument in args, which overrode the long list of patches = [...] in common.nix because there is this line below:

nixpkgs/pkgs/development/libraries/glibc/common.nix

Line 168 in 5453997

// (removeAttrs args [ "withLinuxHeaders" "withGd" ]) //

So I accidentally removed all nixpkgs-specific patches from glibc, which I finally figured out with an invocation of nix-diff:

      • The input named `glibc-2.26-131` differs
        - /nix/store/6jccsmli15ydlivh2x4yidxipvx4jf80-glibc-2.26-131.drv:{bin,dev,out}
        + /nix/store/pgh78jldvx8fgwb8bz00l6a14izjmq50-glibc-2.26-131.drv:{bin,dev,out}
        • The set of input sources do not match:
            - /nix/store/59pddys5h0581y68jzl0zf26lqyb5zm2-2.26-75to115.diff.gz
            - /nix/store/5mmhd582d9yav5vhmch237my1gfyb5d2-2.26-115to131.diff.gz
            - /nix/store/5vbd382jzryfwsa30fmaci5z1x6g4sgj-dont-use-system-ld-so-cache.patch
            - /nix/store/7kw224hdyxd7115lrqh9a4dv2x8msq2s-fix-x64-abi.patch
            - /nix/store/khnabr5dkjmz30fklh99ag0ak65nb7wz-2.26-75.patch.gz
            - /nix/store/mnglr8rr7nl444h7p50ysyq8qd0fm1lm-dont-use-system-ld-so-preload.patch
            - /nix/store/r0rc2prwi70hgw4jray5a8sd41fhvjm7-rpcgen-path.patch
            - /nix/store/svkayyzppy2ncmax2m4klgqkya61ln83-nix-locale-archive.patch
            - /nix/store/xkd50xxii6k7l1kmw4l5x6xzbhamcs87-allow-kernel-2.6.32.patch
            - /nix/store/za0pg7fmysrcwrqcal26fnmzw6vycgdn-fix_path_attribute_in_getconf.patch
            + /nix/store/jr56l9wwdfpk0kri2qm1yvcyj3max7m6-malloc_info-Fix-missing-accounting-of-top-chunk.patch

Still confirming whether this is the solution with a rebuild, but I'm quite sure that'll be it.

[nh2]

Yes, looks like that was it.

No bug, my mistake. I made PR #50291 so that others are less likely to make the same mistake.