-
Notifications
You must be signed in to change notification settings - Fork 4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Disable git fsmonitor in git_worker to fix hangs when fetching repositories #23626
Disable git fsmonitor in git_worker to fix hangs when fetching repositories #23626
Conversation
…tories This commit adds the `-c core.fsmonitor=false` arguments to all git commands (excluding `git init`) in `git_worker.bzl`. The purpose of this is to disable git from spawning a file system monitor, when many of these are spawned on a machine that is already under load or out of file descriptors, the git commands will hang indefinitely. Fixes #21438
I performed tests by spawning many file watchers using the I confirmed that building #!/bin/bash
# Number of monitors to create
NUM_MONITORS=100 # Adjust based on your stress test needs
# Directory to monitor (can be the same or different for each monitor)
MONITOR_DIR="/tmp/fsmonitor-test"
# Create the directory if it doesn't exist
mkdir -p "$MONITOR_DIR"
# Array to store process IDs of background fswatch processes
declare -a PIDS
# Function to create and run a file system monitor
create_monitor() {
local dir=$1
fswatch "$dir" >/dev/null &
PIDS+=($!) # Store the process ID
}
# Create the specified number of monitors
for i in $(seq 1 $NUM_MONITORS); do
create_monitor "$MONITOR_DIR"
# Optionally, introduce a slight delay to avoid overwhelming the system too quickly
sleep 0.1
done
echo "Created $NUM_MONITORS file system monitors."
# Function to clean up by killing all background processes
cleanup() {
echo "Cleaning up monitors..."
for pid in "${PIDS[@]}"; do
kill "$pid" 2>/dev/null
done
echo "All monitors terminated."
}
# Trap script exit (Ctrl+C or normal exit) to ensure cleanup happens
trap cleanup EXIT
# Keep the script running to simulate long-running monitors
echo "Press Ctrl+C to stop the script and clean up the monitors."
while true; do
sleep 10 # Keep the script alive indefinitely
done |
@@ -196,8 +196,11 @@ def _git_maybe_shallow(ctx, git_repo, command, *args): | |||
return _execute(ctx, git_repo, start + args_list) | |||
|
|||
def _execute(ctx, git_repo, args): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I moved the passing of git
to be implied in this function as its intended to run git
commands anyway. This allows me to add -c
which must come directly after git
without searching through the args
we are given
@bazel-io fork 7.4.0 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nice! thanks for providing lots of context too, that really helps.
…tories This commit adds the `-c core.fsmonitor=false` arguments to all git commands (excluding `git init`) in `git_worker.bzl`. The purpose of this is to disable git from spawning a file system monitor, when many of these are spawned on a machine that is already under load or out of file descriptors, the git commands will hang indefinitely. Fixes bazelbuild#21438 Closes bazelbuild#23626. PiperOrigin-RevId: 675472544 Change-Id: I79e0b6d1e406a9076cdb810747c382de52face1a
…g repositories (#23647) This commit adds the `-c core.fsmonitor=false` arguments to all git commands (excluding `git init`) in `git_worker.bzl`. The purpose of this is to disable git from spawning a file system monitor, when many of these are spawned on a machine that is already under load or out of file descriptors, the git commands will hang indefinitely. Fixes #21438 Closes #23626. PiperOrigin-RevId: 675472544 Change-Id: I79e0b6d1e406a9076cdb810747c382de52face1a Commit bb577b0 Co-authored-by: Luis Padron <lpadron@squareup.com>
This commit adds the
-c core.fsmonitor=false
arguments to all git commands (excludinggit init
) ingit_worker.bzl
. The purpose of this is to disable git from spawning a file system monitor, when many of these are spawned on a machine that is already under load or out of file descriptors, the git commands will hang indefinitely.Fixes #21438