-
Notifications
You must be signed in to change notification settings - Fork 242
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix timeout regex for spark submit command #4017
Fix timeout regex for spark submit command #4017
Conversation
…3674_fix_timeout_regex_for_spark_submit_command
Could you add an unit test and also add test run details (i.e. local paasta spark-run) in the PR description? Example local spark-run cmd:
|
I looked for existing unit tests to update them but didn't find any, is it ok to add new one? will update the description with local spark-run command |
Sure, feel free to create one, thanks! |
Added unit test for `--` option
Problem
We check if there is already a timeout command in the user's spark command before auto adding it, but the check doesn't work with the case that an CLI argument is used, e.g.
timeout -v spark-submit
Solution
Fix the regex which checks if timeout is already present
Testing
I test ran the
spark-submit
locally on dev-box withtimeout -v 1m
, as expected it timed out at1m
command and output logs can be found here