fork/exec <command> operation not permitted
error in Workflows with 'tty' option enabled on v3.5.2+
#12829
Closed
3 of 4 tasks
Labels
P3
Low priority
type/bug
type/dependencies
PRs and issues specific to updating dependencies
type/regression
Regression from previous behavior (a specific type of bug)
Milestone
Pre-requisites
:latest
What happened/what did you expect to happen?
We use Argo Worklfows in our production environment. Last month, we upgradeded Argo Workflows from v3.4.6 to v3.5.4 and encountered a strange error at a low frequency. The affected step in the Workflow was marked as a failed phase.
To identify the version causing the error, we gradually increased the version of Argo Workflows and determined that it was caused by Argo Workflows v3.5.2. After reverting the suspicious changes and determining the cause of the error, we found that the error was caused by a change in argoproj/argo-workflows#12139.
In PR argoproj/argo-workflows#12139, the dependant creack/pty library was was upgraded to v1.1.20, affecting workflows that enable the tty option. (source code)
Our Workflow had the
tty
option set to true. The following is a simplified version of our workflow illustrates this setup:We investigated the releases in creack/pty and discovered that creack/pty v1.1.21 reverted the change introduced in v1.1.20 due to race conditions on Linux. For more information, refer to the revert PR. The main branch of Argo Workflows has already incorporated PR argoproj/argo-workflows#12312 to upgrade creack/pty to v1.1.21.
After reverting the changes made in argoproj/argo-workflows#12139, we confirmed that the error no longer occurred in Argo Workflows v3.5.2. Additionally, we upgraded creack/pty to v1.1.21 in Argo Workflows v3.5.2 and confirmed the same result.
Currently, it seems that the PR argoproj/argo-workflows#12312 hasn't been cherry-picked into the v3.5 release branch. Would you consider cherry-picking it to the v3.5 release branch?
Version
v3.5.2
Paste a small workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.
listed above
Logs from the workflow controller
Logs from in your workflow's wait container
The text was updated successfully, but these errors were encountered: