restore mononoke getdeps integration tests

Summary:
Bring back mononoke getdeps integration tests.  This is a refesh of the previously working export-D34186407 branch from 2022

Main changes since:
 * depend on restored sapling manifest from previous commit
 * bring back selected manifests needed by mononoke that were deleted in D51869247. I added the sqlite binary packages from this to the sqlite3 manifest rather than bring back the sqllite3-bin manifest
 * add manifests for new tools used: ripgrep, git-lfs, and zstd cli (we already had the zstd libs)
 * fix a few test expecations that where too closely tied to git cli or TLS version etc (ubuntu 22.04 is on older version)
 * getdeps MakefileBuilder.run_tests() improvements
   * fix error status reporting, was not failing if tests failed
   * pass of --num-jobs to tests to stop it OOMing my machine
   * pass of --filter to tests so can iterate on one test more easily

Can to iterate local execution for one test with:
```
python3 ./build/fbcode_builder/getdeps.py test --num-jobs 4 --allow-system-packages --no-facebook-internal --src-dir=. mononoke_integration --retry 0 --filter server/test-gettreepack.t
```

Test Plan:

Setup a local ubuntu 22.04 toolbox with:
```
toolbox create --distro ubuntu --release 22.04
toolbox enter ubuntu-toolbox-22.04
sudo apt update
sudo apt install g++ libssl-dev pkg-config
sudo locale-gen en_US.UTF-8
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
```

Build dependencies:
```
python3 ./build/fbcode_builder/getdeps.py install-system-deps --recursive mononoke_integration
python3 ./build/fbcode_builder/getdeps.py build --allow-system-packages --no-facebook-internal --src-dir=. --no-tests sapling
python3 ./build/fbcode_builder/getdeps.py build --allow-system-packages --no-facebook-internal --src-dir=. --no-tests mononoke
```

then test locally:
```
./build/fbcode_builder/getdeps.py build --no-deps --num-jobs 16 --allow-system-packages --no-facebook-internal --src-dir=. mononoke_integration
./build/fbcode_builder/getdeps.py test --num-jobs 6 --allow-system-packages --no-facebook-internal --src-dir=. mononoke_integration
```

Generate the github actions yaml
```
./build/fbcode_builder/getdeps.py generate-github-actions --allow-system-packages --test --os-type linux --job-file-prefix mononoke-integration_ --job-name-prefix "Mononoke Integration " --output-dir .github/workflows mononoke_integration --num-jobs 4
```

.github/workflows/mononoke-integration_linux.yml

@@ -0,0 +1,214 @@
+# This file was @generated by getdeps.py
+
+name: Mononoke Integration Linux
+
+on:
+ push:
+ branches:
+ - main
+ pull_request:
+ branches:
+ - main
+
+permissions:
+ contents: read # to fetch code (actions/checkout)
+
+jobs:
+ build:
+ runs-on: ubuntu-22.04
+ steps:
+ - uses: actions/checkout@v4
+ - name: Update system package info
+ run: sudo apt-get update
+ - name: Install system deps
+ run: sudo python3 build/fbcode_builder/getdeps.py --allow-system-packages install-system-deps --recursive mononoke_integration
+ - name: Install packaging system deps
+ run: sudo python3 build/fbcode_builder/getdeps.py --allow-system-packages install-system-deps --recursive patchelf
+ - name: Install Rust Stable
+ uses: dtolnay/rust-toolchain@stable
+ - name: Fetch git-lfs
+ run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests git-lfs
+ - name: Fetch ripgrep
+ run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests ripgrep
+ - name: Fetch tree
+ run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests tree
+ - name: Fetch bz2
+ run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests bz2
+ - name: Fetch ninja
+ run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests ninja
+ - name: Fetch cmake
+ run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests cmake
+ - name: Fetch zstd
+ run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests zstd
+ - name: Fetch python-click
+ run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests python-click
+ - name: Fetch sqlite3
+ run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests sqlite3
+ - name: Fetch gflags
+ run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests gflags
+ - name: Fetch glog
+ run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests glog
+ - name: Fetch fmt
+ run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests fmt
+ - name: Fetch googletest
+ run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests googletest
+ - name: Fetch python-six
+ run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests python-six
+ - name: Fetch autoconf
+ run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests autoconf
+ - name: Fetch automake
+ run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests automake
+ - name: Fetch libtool
+ run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests libtool
+ - name: Fetch jq
+ run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests jq
+ - name: Fetch libsodium
+ run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests libsodium
+ - name: Fetch boost
+ run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests boost
+ - name: Fetch double-conversion
+ run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests double-conversion
+ - name: Fetch fast_float
+ run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests fast_float
+ - name: Fetch libdwarf
+ run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests libdwarf
+ - name: Fetch libevent
+ run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests libevent
+ - name: Fetch libunwind
+ run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests libunwind
+ - name: Fetch lz4
+ run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests lz4
+ - name: Fetch snappy
+ run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests snappy
+ - name: Fetch xz
+ run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests xz
+ - name: Fetch zlib
+ run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests zlib
+ - name: Fetch libiberty
+ run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests libiberty
+ - name: Fetch folly
+ run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests folly
+ - name: Fetch libffi
+ run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests libffi
+ - name: Fetch ncurses
+ run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests ncurses
+ - name: Fetch python
+ run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests python
+ - name: Fetch nmap
+ run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests nmap
+ - name: Fetch openssl
+ run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests openssl
+ - name: Fetch liboqs
+ run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests liboqs
+ - name: Fetch fizz
+ run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests fizz
+ - name: Fetch mvfst
+ run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests mvfst
+ - name: Fetch wangle
+ run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests wangle
+ - name: Fetch fbthrift
+ run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests fbthrift
+ - name: Fetch fb303
+ run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests fb303
+ - name: Fetch rust-shed
+ run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests rust-shed
+ - name: Build git-lfs
+ run: python3 build/fbcode_builder/getdeps.py --allow-system-packages build --no-tests git-lfs
+ - name: Build ripgrep
+ run: python3 build/fbcode_builder/getdeps.py --allow-system-packages build --no-tests ripgrep
+ - name: Build tree
+ run: python3 build/fbcode_builder/getdeps.py --allow-system-packages build --no-tests tree
+ - name: Build bz2
+ run: python3 build/fbcode_builder/getdeps.py --allow-system-packages build --no-tests bz2
+ - name: Build ninja
+ run: python3 build/fbcode_builder/getdeps.py --allow-system-packages build --no-tests ninja
+ - name: Build cmake
+ run: python3 build/fbcode_builder/getdeps.py --allow-system-packages build --no-tests cmake
+ - name: Build zstd
+ run: python3 build/fbcode_builder/getdeps.py --allow-system-packages build --no-tests zstd
+ - name: Build python-click
+ run: python3 build/fbcode_builder/getdeps.py --allow-system-packages build --no-tests python-click
+ - name: Build sqlite3
+ run: python3 build/fbcode_builder/getdeps.py --allow-system-packages build --no-tests sqlite3
+ - name: Build gflags
+ run: python3 build/fbcode_builder/getdeps.py --allow-system-packages build --no-tests gflags
+ - name: Build glog
+ run: python3 build/fbcode_builder/getdeps.py --allow-system-packages build --no-tests glog
+ - name: Build fmt
+ run: python3 build/fbcode_builder/getdeps.py --allow-system-packages build --no-tests fmt
+ - name: Build googletest
+ run: python3 build/fbcode_builder/getdeps.py --allow-system-packages build --no-tests googletest
+ - name: Build python-six
+ run: python3 build/fbcode_builder/getdeps.py --allow-system-packages build --no-tests python-six
+ - name: Build autoconf
+ run: python3 build/fbcode_builder/getdeps.py --allow-system-packages build --no-tests autoconf
+ - name: Build automake
+ run: python3 build/fbcode_builder/getdeps.py --allow-system-packages build --no-tests automake
+ - name: Build libtool
+ run: python3 build/fbcode_builder/getdeps.py --allow-system-packages build --no-tests libtool
+ - name: Build jq
+ run: python3 build/fbcode_builder/getdeps.py --allow-system-packages build --no-tests jq
+ - name: Build libsodium
+ run: python3 build/fbcode_builder/getdeps.py --allow-system-packages build --no-tests libsodium
+ - name: Build boost
+ run: python3 build/fbcode_builder/getdeps.py --allow-system-packages build --no-tests boost
+ - name: Build double-conversion
+ run: python3 build/fbcode_builder/getdeps.py --allow-system-packages build --no-tests double-conversion
+ - name: Build fast_float
+ run: python3 build/fbcode_builder/getdeps.py --allow-system-packages build --no-tests fast_float
+ - name: Build libdwarf
+ run: python3 build/fbcode_builder/getdeps.py --allow-system-packages build --no-tests libdwarf
+ - name: Build libevent
+ run: python3 build/fbcode_builder/getdeps.py --allow-system-packages build --no-tests libevent
+ - name: Build libunwind
+ run: python3 build/fbcode_builder/getdeps.py --allow-system-packages build --no-tests libunwind
+ - name: Build lz4
+ run: python3 build/fbcode_builder/getdeps.py --allow-system-packages build --no-tests lz4
+ - name: Build snappy
+ run: python3 build/fbcode_builder/getdeps.py --allow-system-packages build --no-tests snappy
+ - name: Build xz
+ run: python3 build/fbcode_builder/getdeps.py --allow-system-packages build --no-tests xz
+ - name: Build zlib
+ run: python3 build/fbcode_builder/getdeps.py --allow-system-packages build --no-tests zlib
+ - name: Build libiberty
+ run: python3 build/fbcode_builder/getdeps.py --allow-system-packages build --no-tests libiberty
+ - name: Build folly
+ run: python3 build/fbcode_builder/getdeps.py --allow-system-packages build --no-tests folly
+ - name: Build libffi
+ run: python3 build/fbcode_builder/getdeps.py --allow-system-packages build --no-tests libffi
+ - name: Build ncurses
+ run: python3 build/fbcode_builder/getdeps.py --allow-system-packages build --no-tests ncurses
+ - name: Build python
+ run: python3 build/fbcode_builder/getdeps.py --allow-system-packages build --no-tests python
+ - name: Build nmap
+ run: python3 build/fbcode_builder/getdeps.py --allow-system-packages build --no-tests nmap
+ - name: Build openssl
+ run: python3 build/fbcode_builder/getdeps.py --allow-system-packages build --no-tests openssl
+ - name: Build liboqs
+ run: python3 build/fbcode_builder/getdeps.py --allow-system-packages build --no-tests liboqs
+ - name: Build fizz
+ run: python3 build/fbcode_builder/getdeps.py --allow-system-packages build --no-tests fizz
+ - name: Build mvfst
+ run: python3 build/fbcode_builder/getdeps.py --allow-system-packages build --no-tests mvfst
+ - name: Build wangle
+ run: python3 build/fbcode_builder/getdeps.py --allow-system-packages build --no-tests wangle
+ - name: Build fbthrift
+ run: python3 build/fbcode_builder/getdeps.py --allow-system-packages build --no-tests fbthrift
+ - name: Build fb303
+ run: python3 build/fbcode_builder/getdeps.py --allow-system-packages build --no-tests fb303
+ - name: Build rust-shed
+ run: python3 build/fbcode_builder/getdeps.py --allow-system-packages build --no-tests rust-shed
+ - name: Build mononoke
+ run: python3 build/fbcode_builder/getdeps.py --allow-system-packages build --src-dir=. --no-tests mononoke
+ - name: Build sapling
+ run: python3 build/fbcode_builder/getdeps.py --allow-system-packages build --src-dir=. --no-tests sapling
+ - name: Build mononoke_integration
+ run: python3 build/fbcode_builder/getdeps.py --allow-system-packages build --no-deps --src-dir=. mononoke_integration --project-install-prefix mononoke_integration:/usr/local
+ - name: Copy artifacts
+ run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fixup-dyn-deps --strip --src-dir=. mononoke_integration _artifacts/linux --project-install-prefix mononoke_integration:/usr/local --final-install-prefix /usr/local
+ - uses: actions/upload-artifact@v4
+ with:
+ name: mononoke_integration
+ path: _artifacts
+ - name: Test mononoke_integration
+ run: python3 build/fbcode_builder/getdeps.py --allow-system-packages test --num-jobs 4 --src-dir=. mononoke_integration --project-install-prefix mononoke_integration:/usr/local

.gitignore

@@ -11,3 +11,6 @@
 # Rust libraries and extensions
 Cargo.lock
 target/
+
+# Getdeps
+/eden/mononoke/tests/integration/getdeps_build.log

build/fbcode_builder/getdeps.py

@@ -1190,9 +1190,13 @@ def write_job_for_platform(self, platform, args): # noqa: C901
  and manifest.get("github.actions", "run_tests", ctx=manifest_ctx)
  != "off"
  ):
+ num_jobs_arg = ""
+ if args.num_jobs:
+ num_jobs_arg = f"--num-jobs {args.num_jobs} "
+
  out.write(" - name: Test %s\n" % manifest.name)
  out.write(
- f" run: {getdepscmd}{allow_sys_arg} test --src-dir=. {manifest.name} {project_prefix}\n"
+ f" run: {getdepscmd}{allow_sys_arg} test {num_jobs_arg}--src-dir=. {manifest.name} {project_prefix}\n"
  )
  if build_opts.free_up_disk and not build_opts.is_windows():
  out.write(" - name: Show disk space at end\n")

build/fbcode_builder/getdeps/builder.py

@@ -315,9 +315,22 @@ def run_tests(self, schedule_type, owner, test_filter, retry, no_testpilot) -> N
  return
 
  env = self._compute_env()
+ if test_filter:
+ env["GETDEPS_TEST_FILTER"] = test_filter
+ else:
+ env["GETDEPS_TEST_FILTER"] = ""
 
- cmd = [self._make_binary] + self.test_args + self._get_prefix()
- self._run_cmd(cmd, env=env)
+ if retry:
+ env["GETDEPS_TEST_RETRY"] = retry
+ else:
+ env["GETDEPS_TEST_RETRY"] = 0
+
+ cmd = (
+ [self._make_binary, "-j%s" % self.num_jobs]
+ + self.test_args
+ + self._get_prefix()
+ )
+ self._run_cmd(cmd, allow_fail=False, env=env)
 
 
 class CMakeBootStrapBuilder(MakeBuilder):

build/fbcode_builder/manifests/git-lfs

@@ -0,0 +1,15 @@
+[manifest]
+name = git-lfs
+
+[rpms]
+git-lfs
+
+[debs]
+git-lfs
+
+[homebrew]
+git-lfs
+
+# only used from system packages currently
+[build]
+builder = nop

build/fbcode_builder/manifests/jq

@@ -0,0 +1,25 @@
+[manifest]
+name = jq
+
+[rpms.distro=fedora]
+jq
+
+[homebrew]
+jq
+
+[download.not(os=windows)]
+# we use jq-1.7+ to get fix for number truncation https://github.com/jqlang/jq/pull/1752
+url = https://github.com/jqlang/jq/releases/download/jq-1.7.1/jq-1.7.1.tar.gz
+sha256 = 478c9ca129fd2e3443fe27314b455e211e0d8c60bc8ff7df703873deeee580c2
+
+[build.not(os=windows)]
+builder = autoconf
+subdir = jq-1.7.1
+
+[build.os=windows]
+builder = nop
+
+[autoconf.args]
+# This argument turns off some developers tool and it is recommended in jq's
+# README
+--disable-maintainer-mode

build/fbcode_builder/manifests/mononoke_integration

@@ -0,0 +1,53 @@
+[manifest]
+name = mononoke_integration
+fbsource_path = fbcode/eden
+shipit_project = eden
+shipit_fbcode_builder = true
+
+[git]
+repo_url = https://github.com/facebook/sapling.git
+
+[build.not(os=windows)]
+builder = make
+subdir = eden/mononoke/tests/integration
+
+[build.os=windows]
+# building Mononoke on windows is not supported
+builder = nop
+
+[make.build_args]
+build-getdeps
+
+[make.install_args]
+install-getdeps
+
+[make.test_args]
+test-getdeps
+
+[shipit.pathmap]
+fbcode/eden/mononoke/tests/integration = eden/mononoke/tests/integration
+
+[shipit.strip]
+^.*/facebook/.*$
+^.*/fb/.*$
+
+[dependencies]
+git-lfs
+jq
+mononoke
+nmap
+python
+python-click
+ripgrep
+sapling
+tree
+zstd
+
+[dependencies.os=linux]
+sqlite3
+
+[dependencies.os=darwin]
+gnu-bash
+gnu-coreutils
+gnu-grep
+gnu-sed

build/fbcode_builder/manifests/nmap

@@ -0,0 +1,30 @@
+[manifest]
+name = nmap
+
+[rpms]
+nmap
+nmap-ncat
+
+[debs]
+nmap
+
+# 18.04 combines ncat into the nmap package, newer need the separate one
+[debs.not(all(distro=ubuntu,distro_vers="18.04"))]
+ncat
+
+[download.not(os=windows)]
+url = https://api.github.com/repos/nmap/nmap/tarball/ef8213a36c2e89233c806753a57b5cd473605408
+sha256 = eda39e5a8ef4964fac7db16abf91cc11ff568eac0fa2d680b0bfa33b0ed71f4a
+
+[build.not(os=windows)]
+builder = autoconf
+subdir = nmap-nmap-ef8213a
+build_in_src_dir = true
+
+[build.os=windows]
+builder = nop
+
+[autoconf.args]
+# Without this option the build was filing to find some third party libraries
+# that we don't need
+enable_rdma=no

build/fbcode_builder/manifests/python-click

@@ -0,0 +1,9 @@
+[manifest]
+name = python-click
+
+[download]
+url = https://files.pythonhosted.org/packages/d2/3d/fa76db83bf75c4f8d338c2fd15c8d33fdd7ad23a9b5e57eb6c5de26b430e/click-7.1.2-py2.py3-none-any.whl
+sha256 = dacca89f4bfadd5de3d7489b7c8a566eee0d3676333fbb50030263894c38c0dc
+
+[build]
+builder = python-wheel