Tests failing on Bazel CI #67

katre · 2018-02-21T13:57:53Z

I am seeing failures in Bazel CI for both Bazel HEAD and Bazel 0.10.1: https://ci.bazel.build/blue/organizations/jenkins/Global%2Frules_python/detail/rules_python/276/tests

I can reproduce this locally and get the same error messages.

katre · 2018-02-22T20:41:30Z

This seems to be due to 3175797, @mattmoor can you comment?

mattmoor · 2018-02-26T15:23:37Z

@duggelz made PAR construction deterministic, so without probing deeper I would guess that you changed something (transitively) about how the PAR is built (the same __init__.py change that broke rules_docker seems plausible?).

buchgr · 2018-02-27T13:59:29Z

How can we resolve this issue? Who would be the person to talk to?

duggelz · 2018-02-27T23:46:00Z

Bazel has apparently just switched to a new CI system:

https://groups.google.com/d/msg/bazel-discuss/8spcVZMv9yE/nz-ya0tpAQAJ

buchgr · 2018-03-23T10:16:22Z

Heya,

we are still seeing this error on our CI. //tools:par_test is failing with Bazel from HEAD but it works with the latest released Bazel.

exec ${PAGER:-/usr/bin/less} "$0" || exit 1
-----------------------------------------------------------------------------
FF
======================================================================
FAIL: test_piptool_matches (__main__.WheelTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/3dd5dfaab0d9fd8c7dd47297b1d500e7/sandbox/5266514015113038112/execroot/io_bazel_rules_python/bazel-out/k8-fastbuild/bin/tools/par_test.runfiles/io_bazel_rules_python/tools/par_test.py", line 33, in test_piptool_matches
    'The checked in tools/piptool.par does not match the latest build.')
AssertionError: The checked in tools/piptool.par does not match the latest build.

======================================================================
FAIL: test_whltool_matches (__main__.WheelTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/3dd5dfaab0d9fd8c7dd47297b1d500e7/sandbox/5266514015113038112/execroot/io_bazel_rules_python/bazel-out/k8-fastbuild/bin/tools/par_test.runfiles/io_bazel_rules_python/tools/par_test.py", line 42, in test_whltool_matches
    'The checked in tools/whltool.par does not match the latest build.')
AssertionError: The checked in tools/whltool.par does not match the latest build.

----------------------------------------------------------------------
Ran 2 tests in 0.050s

FAILED (failures=2)

duggelz · 2018-03-24T00:05:22Z

I am pretty curious what the actual difference is. I don't have access to the new CI, so I guess I'll see if I can repo it locally.

buchgr · 2018-03-25T18:14:18Z

@duggelz I have sent you an invite to your @google.com address.

lfpino · 2018-03-26T16:09:29Z

Any updates on this issue @duggelz ? Thanks!

lfpino · 2018-03-28T11:07:22Z

Friendly ping @duggelz, we're still seeing the failures in Bazel CI

duggelz · 2018-03-29T00:49:13Z

Ok, so here's my diagnosis: .par file construction is not deterministic between different Python interpreter versions. In particular, python 2.7.13+ and python 2.7.12- produce zip files with slightly different header flags, because 2.7.12 was out of spec (but the zip files still work - it's a loose spec). That seems to be what's happening here.

It's not really fixable by subpar. We could relax the test to check that the contents of the individual zip members are the same. But a malicious adversary could easily handcraft a zipfile header that passes that relaxed check. And I'm actually not comfortable with this as a "security" boundary anyway, because I certainly didn't write the code as such.

So, unless we have some kind of process where .par files are regenerated on a "blessed" image with a known Python interpreter version, and/or by a "blessed" release person, we should probably disable this test.

And we really shouldn't let untrusted contributors submit updated .par files.

duggelz · 2018-04-02T22:33:42Z

As per this thread: https://bugs.python.org/issue29094 , it appears that Python 2.7.13, 3.5.3, 3.6.0, and 3.6.1 (only) produce these subtly broken .par files. Unfortunately, those are the versions that are installed with the widely used Debian 9 (not to mention my personal workstation!). By chance, it appears that none of the BuildKite builder machines have those "bad" versions, so we should be able to rebuild the .par files, and turn the build green.

duggelz · 2018-04-05T01:15:19Z

Regenerating the files on Ubuntu 16.04 turned the build green: #83

Going forward, it seems like we can keep this test, at the cost of a bit of annoyance for people with "broken" Python versions (basically just Debian 9). Google's internal workstations will be updating from Python 2.7.13 to 2.7.14 "soon", and thus get the fix, in the future (We also have Python 3.6! Which is now entirely supported and recommended inside Google! Use Python 3 everywhere!)

vladmos · 2018-04-06T13:16:48Z

Can #83 be merged? We'd like to cut a release for Bazel 0.13 and failing rules_python builds block that.

duggelz · 2018-04-07T06:50:05Z

Merged.

philwo · 2018-04-09T21:49:18Z

@duggelz Not sure what's going on, but this still seems to fail (tested with Bazel 0.12.0rc3 just a few minutes ago):

exec ${PAGER:-/usr/bin/less} "$0" || exit 1
-----------------------------------------------------------------------------
FF
======================================================================
FAIL: test_piptool_matches (__main__.WheelTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/b5c94007a378b85b2836f44d796f101b/bazel-sandbox/3529782167307750595/execroot/io_bazel_rules_python/bazel-out/k8-fastbuild/bin/tools/par_test.runfiles/io_bazel_rules_python/tools/par_test.py", line 86, in test_piptool_matches
    TestData('tools/piptool.par'))
  File "/var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/b5c94007a378b85b2836f44d796f101b/bazel-sandbox/3529782167307750595/execroot/io_bazel_rules_python/bazel-out/k8-fastbuild/bin/tools/par_test.runfiles/io_bazel_rules_python/tools/par_test.py", line 82, in _diff_zip
    self.assertEquals(contents1, contents2, message)
AssertionError: Files do not match.

************************************************************************
File 1: /var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/b5c94007a378b85b2836f44d796f101b/bazel-sandbox/3529782167307750595/execroot/io_bazel_rules_python/bazel-out/k8-fastbuild/bin/tools/par_test.runfiles/io_bazel_rules_python/rules_python/piptool.par
Length in bytes: 5879466
SHA256: 51519a9c2c8b0917c82158bdd07bd36f9db5b0e82519cb34b1ce66cc09e0a1d1
************************************************************************
File 2: /var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/b5c94007a378b85b2836f44d796f101b/bazel-sandbox/3529782167307750595/execroot/io_bazel_rules_python/bazel-out/k8-fastbuild/bin/tools/par_test.runfiles/io_bazel_rules_python/tools/piptool.par
Length in Bytes: 5878341
SHA256: 5deae79fc45c6b9f3cbb9943e48b24ef63ebad563b002c453bf97a2f56efaba5
************************************************************************
Zip Content Diff:
--- 

+++ 

@@ -2,7 +2,7 @@

 __init__.py                                    1980-01-01 00:00:00            0
 __main__.py                                    1980-01-01 00:00:00         7673
 io_bazel_rules_python/rules_python/__init__.py 1980-01-01 00:00:00            0
-io_bazel_rules_python/rules_python/piptool     1980-01-01 00:00:00         6847
+io_bazel_rules_python/rules_python/piptool     1980-01-01 00:00:00         5722
 io_bazel_rules_python/rules_python/piptool.py  1980-01-01 00:00:00         7342
 io_bazel_rules_python/rules_python/whl.py      1980-01-01 00:00:00         5606
 pypi__pip_9_0_3/__init__.py                    1980-01-01 00:00:00            0
************************************************************************


======================================================================
FAIL: test_whltool_matches (__main__.WheelTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/b5c94007a378b85b2836f44d796f101b/bazel-sandbox/3529782167307750595/execroot/io_bazel_rules_python/bazel-out/k8-fastbuild/bin/tools/par_test.runfiles/io_bazel_rules_python/tools/par_test.py", line 90, in test_whltool_matches
    TestData('tools/whltool.par'))
  File "/var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/b5c94007a378b85b2836f44d796f101b/bazel-sandbox/3529782167307750595/execroot/io_bazel_rules_python/bazel-out/k8-fastbuild/bin/tools/par_test.runfiles/io_bazel_rules_python/tools/par_test.py", line 82, in _diff_zip
    self.assertEquals(contents1, contents2, message)
AssertionError: Files do not match.

************************************************************************
File 1: /var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/b5c94007a378b85b2836f44d796f101b/bazel-sandbox/3529782167307750595/execroot/io_bazel_rules_python/bazel-out/k8-fastbuild/bin/tools/par_test.runfiles/io_bazel_rules_python/rules_python/whltool.par
Length in bytes: 1386426
SHA256: acbf53c155c70fe36d3c1a0e0fc70c18776a99566b3a9ff3063e592445a58c5c
************************************************************************
File 2: /var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/b5c94007a378b85b2836f44d796f101b/bazel-sandbox/3529782167307750595/execroot/io_bazel_rules_python/bazel-out/k8-fastbuild/bin/tools/par_test.runfiles/io_bazel_rules_python/tools/whltool.par
Length in Bytes: 1385301
SHA256: f56e8128d2bd575859e483eaa9a356ec05442765a076d757fc9067cb654de833
************************************************************************
Zip Content Diff:
--- 

+++ 

@@ -3,7 +3,7 @@

 __main__.py                                    1980-01-01 00:00:00         5847
 io_bazel_rules_python/rules_python/__init__.py 1980-01-01 00:00:00            0
 io_bazel_rules_python/rules_python/whl.py      1980-01-01 00:00:00         5606
-io_bazel_rules_python/rules_python/whltool     1980-01-01 00:00:00         6806
+io_bazel_rules_python/rules_python/whltool     1980-01-01 00:00:00         5681
 pypi__setuptools_38_2_4/__init__.py            1980-01-01 00:00:00            0
 pypi__setuptools_38_2_4/easy_install.py        1980-01-01 00:00:00          126
 pypi__setuptools_38_2_4/pkg_resources/__init__.py 1980-01-01 00:00:00       105991
************************************************************************


----------------------------------------------------------------------
Ran 2 tests in 0.231s

FAILED (failures=2)

https://buildkite.com/bazel/bazel-with-downstream-projects-bazel/builds/193#143a5613-592d-4f0e-a442-ca9a9c21fc86

buchgr mentioned this issue Mar 23, 2018

rules_python red on bazel ci #80

Closed

duggelz mentioned this issue Mar 24, 2018

Switch from using .par files to pure python for piptool and whltool #81

Closed

vladmos mentioned this issue Apr 6, 2018

Release - April 2018 - Target RC date: 2018-04-02 - name: 0.13.0 bazelbuild/bazel#4583

Closed

duggelz closed this as completed Apr 7, 2018

philwo mentioned this issue Apr 9, 2018

Release - March 2018 - Target RC date: 2018-03-01 - name: 0.12.0 bazelbuild/bazel#4582

Closed

hlopko mentioned this issue Sep 5, 2018

//tools:par_test red on bazel ci again #117

Closed

brandjon mentioned this issue Apr 23, 2019

Don't use .par files for piptool/whltool #184

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Tests failing on Bazel CI #67

Tests failing on Bazel CI #67

katre commented Feb 21, 2018

katre commented Feb 22, 2018

mattmoor commented Feb 26, 2018

buchgr commented Feb 27, 2018

duggelz commented Feb 27, 2018

buchgr commented Mar 23, 2018

duggelz commented Mar 24, 2018

buchgr commented Mar 25, 2018

lfpino commented Mar 26, 2018

lfpino commented Mar 28, 2018

duggelz commented Mar 29, 2018 •

edited

Loading

duggelz commented Apr 2, 2018

duggelz commented Apr 5, 2018

vladmos commented Apr 6, 2018

duggelz commented Apr 7, 2018

philwo commented Apr 9, 2018

Tests failing on Bazel CI #67

Tests failing on Bazel CI #67

Comments

katre commented Feb 21, 2018

katre commented Feb 22, 2018

mattmoor commented Feb 26, 2018

buchgr commented Feb 27, 2018

duggelz commented Feb 27, 2018

buchgr commented Mar 23, 2018

duggelz commented Mar 24, 2018

buchgr commented Mar 25, 2018

lfpino commented Mar 26, 2018

lfpino commented Mar 28, 2018

duggelz commented Mar 29, 2018 • edited Loading

duggelz commented Apr 2, 2018

duggelz commented Apr 5, 2018

vladmos commented Apr 6, 2018

duggelz commented Apr 7, 2018

philwo commented Apr 9, 2018

duggelz commented Mar 29, 2018 •

edited

Loading