From 93083d38668b59804fb753b9578ac96cc2c7634b Mon Sep 17 00:00:00 2001 From: Alex Hornby Date: Mon, 25 Nov 2024 00:16:50 +0000 Subject: [PATCH] fix hangs in watchman oss tests on ubuntu Summary: watchman oss tests on ubuntu are [hanging for over 2 hours on github](https://github.com/facebook/watchman/actions/runs/11989130985/job/33425158244), and locally show errors. One clue was that tests worked locally on centos stream 9 and fedora 40, pointing to one or more of the ubuntu system packages being the trigger. Turns out that having the ubuntu xz aka liblzma-dev system packages linked is triggering the issue on both ubuntu 22.04 and 24.04. Disabled system packages for xz and for glog and libunwind that also bring in the xz system packages on ubuntu. Test Plan: Local run: ``` /build/fbcode_builder/getdeps.py --allow-system-packages test watchman --src-dir=. --filter watchman.integration.test_bulkstat.TestBulkStatCliJson --retry 0 ``` Before, fails ``` Test project /home/alex/local/tmp/ubuntu-24.04/fbcode_builder_getdeps-ZhomeZalexZlocalZwatchmanZbuildZfbcode_builder/build/watchman Start 110: test_py::watchman.integration.test_bulkstat.TestBulkStatCliJson.test_bulkstat_off Start 111: test_py::watchman.integration.test_bulkstat.TestBulkStatCliJson.test_bulkstat_on 1/2 Test #111: test_py::watchman.integration.test_bulkstat.TestBulkStatCliJson.test_bulkstat_on ....***Failed 60.82 sec test_bulkstat_on (watchman.integration.test_bulkstat.TestBulkStatCliJson.test_bulkstat_on) ... 2024-11-24T18:33:28,230: [cli] failed to identify PDU: fill_buffer: Connection reset by peer ERROR Watchman logs: CLI logs Server logs 2024-11-24T18:33:28,401: [listener] Watchman 0.0.0 starting up on ubuntu-24.04.local by command /usr/bin/python3 /home/alex/local/tmp/ubuntu-24.04/fbcode_builder_getdeps-ZhomeZalexZlocalZwatchmanZbuildZfbcode_builder/build/watchman/watchman/integration/test_py watchman.integration.test_bulkstat.TestBulkStatCliJson.test_bulkstat_on 2024-11-24T18:33:28,403: [sanitychecks] starting sanityCheckThread 2024-11-24T18:33:28,439: [unix-listener] accepted client:stm=0x7fd388000ce0 2024-11-24T18:33:28,439: [client=1:stm=140546496531680:pid=539033] dispatch_command: version 2024-11-24T18:33:28,439: [client=1:stm=140546496531680:pid=539033] dispatch_command: version (completed) 2024-11-24T18:33:28,439: [client=1:stm=140546496531680:pid=539033] dispatch_command: version 2024-11-24T18:33:28,439: [client=1:stm=140546496531680:pid=539033] dispatch_command: version (completed) 2024-11-24T18:33:28,440: [client=1:stm=140546496531680:pid=539033] dispatch_command: get-pid 2024-11-24T18:33:28,440: [client=1:stm=140546496531680:pid=539033] dispatch_command: get-pid (completed) 2024-11-24T18:33:28,440: [NOT_CONN:client=1:stm=140546496531680:pid=539033] client_delete 1 ====================================================================== ERROR: test_bulkstat_on (watchman.integration.test_bulkstat.TestBulkStatCliJson.test_bulkstat_on) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/alex/local/tmp/ubuntu-24.04/fbcode_builder_getdeps-ZhomeZalexZlocalZwatchmanZbuildZfbcode_builder/build/watchman/watchman/integration/test_py/watchman/integration/test_bulkstat.py", line 26, in test_bulkstat_on self.assertFileList(root, ["foo", "bar"]) File "/home/alex/local/tmp/ubuntu-24.04/fbcode_builder_getdeps-ZhomeZalexZlocalZwatchmanZbuildZfbcode_builder/build/watchman/watchman/integration/test_py/watchman/integration/lib/WatchmanTestCase.py", line 389, in assertFileList st, res = self.waitFor( ^^^^^^^^^^^^^ File "/home/alex/local/tmp/ubuntu-24.04/fbcode_builder_getdeps-ZhomeZalexZlocalZwatchmanZbuildZfbcode_builder/build/watchman/watchman/integration/test_py/watchman/integration/lib/WatchmanTestCase.py", line 301, in waitFor return self._waitForCheck(cond, lambda res: res, timeout) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/alex/local/tmp/ubuntu-24.04/fbcode_builder_getdeps-ZhomeZalexZlocalZwatchmanZbuildZfbcode_builder/build/watchman/watchman/integration/test_py/watchman/integration/lib/WatchmanTestCase.py", line 289, in _waitForCheck res = cond() ^^^^^^ File "/home/alex/local/tmp/ubuntu-24.04/fbcode_builder_getdeps-ZhomeZalexZlocalZwatchmanZbuildZfbcode_builder/build/watchman/watchman/integration/test_py/watchman/integration/lib/WatchmanTestCase.py", line 391, in self.getFileList(root, cursor=cursor, relativeRoot=relativeRoot), ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/alex/local/tmp/ubuntu-24.04/fbcode_builder_getdeps-ZhomeZalexZlocalZwatchmanZbuildZfbcode_builder/build/watchman/watchman/integration/test_py/watchman/integration/lib/WatchmanTestCase.py", line 339, in getFileList res = self.watchmanCommand("query", root, expr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/alex/local/tmp/ubuntu-24.04/fbcode_builder_getdeps-ZhomeZalexZlocalZwatchmanZbuildZfbcode_builder/build/watchman/watchman/integration/test_py/watchman/integration/lib/WatchmanTestCase.py", line 271, in watchmanCommand return client.query(*args) ^^^^^^^^^^^^^^^^^^^ File "/home/alex/local/tmp/ubuntu-24.04/fbcode_builder_getdeps-ZhomeZalexZlocalZwatchmanZbuildZfbcode_builder/build/watchman/watchman/integration/test_py/pywatchman/__init__.py", line 1143, in query res = self.receive() ^^^^^^^^^^^^^^ File "/home/alex/local/tmp/ubuntu-24.04/fbcode_builder_getdeps-ZhomeZalexZlocalZwatchmanZbuildZfbcode_builder/build/watchman/watchman/integration/test_py/pywatchman/__init__.py", line 1051, in receive result = self.recvConn.receive() ^^^^^^^^^^^^^^^^^^^^^^^ File "/home/alex/local/tmp/ubuntu-24.04/fbcode_builder_getdeps-ZhomeZalexZlocalZwatchmanZbuildZfbcode_builder/build/watchman/watchman/integration/test_py/pywatchman/__init__.py", line 804, in receive line = self.transport.readLine() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/alex/local/tmp/ubuntu-24.04/fbcode_builder_getdeps-ZhomeZalexZlocalZwatchmanZbuildZfbcode_builder/build/watchman/watchman/integration/test_py/pywatchman/__init__.py", line 244, in readLine b = self.readBytes(4096) ^^^^^^^^^^^^^^^^^^^^ File "/home/alex/local/tmp/ubuntu-24.04/fbcode_builder_getdeps-ZhomeZalexZlocalZwatchmanZbuildZfbcode_builder/build/watchman/watchman/integration/test_py/pywatchman/__init__.py", line 657, in readBytes raise WatchmanError("EOF on CLI process transport") pywatchman.WatchmanError: EOF on CLI process transport, while executing ('query', '/home/alex/local/tmp/ubuntu-24.04/watchmantestu1xdmt7d/watchman.integration.test_bulkstat.TestBulkStatCliJson.test_bulkstat_on.cli.json.p/tmpmlimtbp5', {'expression': ['exists'], 'fields': ['name']}) ---------------------------------------------------------------------- Ran 1 test in 60.590s ``` After, works ``` 1/2 Test #111: test_py::watchman.integration.test_bulkstat.TestBulkStatCliJson.test_bulkstat_on .... Passed 0.52 sec 2/2 Test #110: test_py::watchman.integration.test_bulkstat.TestBulkStatCliJson.test_bulkstat_off ... Passed 0.52 sec 100% tests passed, 0 tests failed out of 2 ``` After, all tests work locally on ubuntu 22.04 (although 2 intermittently fail): ``` 441/441 Test #337: test_py::watchman.integration.test_sock_perms.TestSockPerms.test_invalid_sock_access ......................................... Passed 120.20 sec 100% tests passed, 0 tests failed out of 441 Total Test time (real) = 123.80 sec ``` Example remaining intermittent fails: ``` 24-11-24T20:13:54,749: [client=4:stm=140224373984480:pid=718930] dispatch_command: query (completed) 2024-11-24T20:13:54,750: [NOT_CONN:client=4:stm=140224373984480:pid=718930] client_delete 4 tcache_thread_shutdown(): unaligned tcache chunk detected *** Aborted at 1732479234 (Unix time, try 'date -d @1732479234') *** *** Signal 6 (SIGABRT) (0x3e8000af610) received by PID 718352 (pthread TID 0x7f88db7fe640) (linux TID 718939) (maybe from PID 718352, UID 1000) (code: -6), stack trace: *** @ 000000000082b9d2 _ZN5folly10symbolizer12_GLOBAL__N_113signalHandlerEiP9siginfo_tPv /home/alex/local/tmp/ubuntu-22.04/fbcode_builder_getdeps-ZhomeZalexZlocalZwatchmanZbuildZfbcode_builder/repos/github.com-facebook-folly.git/folly/debugging/symbolizer/SignalHandler.cpp:453 @ 000000000004251f (unknown) @ 00000000000969fc pthread_kill @ 0000000000042475 raise @ 00000000000287f2 abort @ 0000000000089675 (unknown) @ 00000000000a0cfb (unknown) @ 00000000000a56c3 (unknown) @ 000000000009494e (unknown) @ 000000000012684f (unknown) ERROR ... 99% tests passed, 2 tests failed out of 441 Total Test time (real) = 120.17 sec The following tests FAILED: 140 - test_py::watchman.integration.test_content_hash.TestContentHashCliJson.test_cacheLimit (Failed) 141 - test_py::watchman.integration.test_content_hash.TestContentHashCliJson.test_contentHash (Failed) Errors while running CTest Command '['/usr/bin/ctest', '--output-on-failure', '-j', '31', '--rerun-failed']' returned non-zero exit status 8. !! Failed ``` --- build/fbcode_builder/manifests/glog | 3 ++- build/fbcode_builder/manifests/libunwind | 3 ++- build/fbcode_builder/manifests/xz | 3 ++- 3 files changed, 6 insertions(+), 3 deletions(-) diff --git a/build/fbcode_builder/manifests/glog b/build/fbcode_builder/manifests/glog index b5d5fa814cc2..2649eaad1805 100644 --- a/build/fbcode_builder/manifests/glog +++ b/build/fbcode_builder/manifests/glog @@ -24,7 +24,8 @@ HAVE_TR1_UNORDERED_SET=OFF [homebrew] glog -[debs] +# on ubuntu glog brings in liblzma-dev, which in turn breaks watchman tests +[debs.not(distro=ubuntu)] libgoogle-glog-dev [rpms.distro=fedora] diff --git a/build/fbcode_builder/manifests/libunwind b/build/fbcode_builder/manifests/libunwind index 0a4f03bc8fef..fda19209ad32 100644 --- a/build/fbcode_builder/manifests/libunwind +++ b/build/fbcode_builder/manifests/libunwind @@ -5,7 +5,8 @@ name = libunwind libunwind-devel libunwind -[debs] +# on ubuntu this brings in liblzma-dev, which in turn breaks watchman tests +[debs.not(distro=ubuntu)] libunwind-dev [download] diff --git a/build/fbcode_builder/manifests/xz b/build/fbcode_builder/manifests/xz index 0b27ad63cc91..e6c0808ff01f 100644 --- a/build/fbcode_builder/manifests/xz +++ b/build/fbcode_builder/manifests/xz @@ -1,7 +1,8 @@ [manifest] name = xz -[debs] +# ubuntu's package causes watchman's tests to hang +[debs.not(distro=ubuntu)] liblzma-dev [homebrew]