Buildkite Agent PTY randomly failed with fork/exec... operation not permitted

[leakingtapan]

The problem

The buildkite agent could randomly fail the job with following error message:

2024-05-20 11:15:52 PDT $ git clone -v -- ssh...
...
...
2024-05-20 11:15:52 PDT 🚨 Error: Failed to checkout plugin docker: error running "/usr/bin/git checkout -f e9efccb": error starting pty: fork/exec /usr/bin/git: operation not permitted
2024-05-20 11:15:52 PDT Running global pre-exit hook

2024-05-21 09:28:12 PDT ⚠️ Warning: Checkout failed! cleaning repository post-checkout: error running "/usr/bin/git clean -ffxdq": error starting pty: fork/exec /usr/bin/git: operation not permitted (Attempt 1/3 Retrying in 2s)

2024-05-20 10:37:40 PDT 🚨 Error: error running "/bin/bash -e -c \"trap 'kill -- $$' INT TERM QUIT; if [ -n \\\"$(git diff HEAD~1 --exit-code 'cruise/ai_platform/search')\\\" ]; then\\n  cat <<- ...\"": error starting pty: fork/exec /bin/bash: operation not permitted

I worked with BK support on this issue for long time, and we are able to pin down the issue being the github.com/creack/pty PTY package that's used. Although v1.1.21 reverted a bug v1.1.20 that's mostly related to this bug. The issue is not fully resolved in v1.1.21.

I tested following BK agent version:

• v3.59.0 uses pty v1.1.20
• v3.63.1 uses pty v1.1.21
  all has the same issue, just v3.3.60 has a lower frequency.

I tested v3.63.1 with pty v1.1.19 and the problem no longer happens. So there should be something something wrong with pty v1.1.21 still.

Proposed solution

pin agent's pty package version to v1.1.19

cc @DrJosh9000 @steveh