Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

slurm tweaks #802

Merged
merged 66 commits into from
Apr 15, 2020
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
66 commits
Select commit Hold shift + click to select a range
191c8f9
Shows stacktrace only when --debug is given
ivotron Mar 18, 2020
da03369
WIP kubernetes runner
ivotron Mar 18, 2020
a666826
Minor tweaks to resource manager dynamic loader
ivotron Mar 18, 2020
cb61940
wip
ivotron Mar 30, 2020
dae7fed
wip - volume creation and build/push
ivotron Mar 30, 2020
42700ca
Renames install-deps script
ivotron Apr 1, 2020
b0c1c00
Adds tests for kubernetes
ivotron Apr 1, 2020
7b106ec
tweaks to config.py module and tests
ivotron Apr 6, 2020
bed6d1f
changes to runner
ivotron Apr 6, 2020
da44088
Removes dotmap
ivotron Apr 6, 2020
4935bf4
Changes to runner_host and tests
ivotron Apr 6, 2020
9aa3ba0
WIP
ivotron Apr 6, 2020
da5a240
wip
ivotron Apr 8, 2020
48bf3c7
wip
ivotron Apr 8, 2020
7ecef02
wippy
ivotron Apr 8, 2020
5ab47e3
wip tweak slurm runner
ivotron Apr 8, 2020
e802abf
Fixes passing arguments up the class hierarchy
ivotron Apr 8, 2020
68cc78b
brings back setup_base_cache tests
ivotron Apr 8, 2020
66d2a13
revert change
ivotron Apr 8, 2020
1e6ce6c
change _spawned_containers list to set
JayjeetAtGithub Apr 8, 2020
dd66f42
Fix wrong name
JayjeetAtGithub Apr 8, 2020
3327e0b
Bugfix
JayjeetAtGithub Apr 8, 2020
54a2e99
Bugfix
JayjeetAtGithub Apr 8, 2020
22e5038
Bugfix
JayjeetAtGithub Apr 9, 2020
3b2ef89
Bugfix
JayjeetAtGithub Apr 9, 2020
fda1b54
Bugfix
JayjeetAtGithub Apr 9, 2020
6335179
Fixes
JayjeetAtGithub Apr 9, 2020
7e70eaf
Fixes
JayjeetAtGithub Apr 9, 2020
739e1fa
Completes test_runner_host and fixes
JayjeetAtGithub Apr 9, 2020
56a62c6
Fix permissions on install_scripts.sh
JayjeetAtGithub Apr 9, 2020
34cf97a
Add psutil dependency to setup.py
JayjeetAtGithub Apr 9, 2020
1e9bf16
Change repository name to lower case
JayjeetAtGithub Apr 9, 2020
5d00e5b
Fix config parsing
JayjeetAtGithub Apr 10, 2020
039f04e
Add pu.decode back temporarily
JayjeetAtGithub Apr 10, 2020
8c41dc5
Fix stop_running_tasks in DockerRunner
JayjeetAtGithub Apr 10, 2020
bb453cb
Complete tests
JayjeetAtGithub Apr 10, 2020
0780272
Add test_run() to SlurmDockerRunnerTests
JayjeetAtGithub Apr 10, 2020
90ff39c
Fix test_run()
JayjeetAtGithub Apr 10, 2020
22c7408
Seperate test dependencies
JayjeetAtGithub Apr 11, 2020
d2b654f
Complete test_runner_slurm.py
JayjeetAtGithub Apr 11, 2020
021ab04
Removes TODO
JayjeetAtGithub Apr 11, 2020
2ba4564
Change to using rstrip()
JayjeetAtGithub Apr 11, 2020
f50f9da
Check thread status
JayjeetAtGithub Apr 11, 2020
eaec5a2
Fix tests
JayjeetAtGithub Apr 11, 2020
6368021
Add tail process status check
JayjeetAtGithub Apr 11, 2020
61d573c
Add delay after thread start and use default decode
JayjeetAtGithub Apr 11, 2020
ace8d4d
PEP8 issue fixes
JayjeetAtGithub Apr 11, 2020
91bf0e3
Fix test
JayjeetAtGithub Apr 11, 2020
afed13b
Indentation fixes
JayjeetAtGithub Apr 12, 2020
2b8f184
Test calls in MockPopen
JayjeetAtGithub Apr 14, 2020
fbdd346
Add precedence information in cmd_run docstring
JayjeetAtGithub Apr 14, 2020
93a083f
Change info statements
JayjeetAtGithub Apr 14, 2020
4ea812c
Changes in host runner
JayjeetAtGithub Apr 14, 2020
cae4ac4
Tests updated
JayjeetAtGithub Apr 14, 2020
e2ef512
Fix the failing test_submit_job_failure
JayjeetAtGithub Apr 14, 2020
5eb8a95
test_runner_slurm updates
JayjeetAtGithub Apr 14, 2020
d2b434b
Use context managers in test_runner_host.py
JayjeetAtGithub Apr 14, 2020
bfd842b
Use context managers in test_runner_slurm
JayjeetAtGithub Apr 14, 2020
61e5de3
Add instructions for running unittests locally
JayjeetAtGithub Apr 15, 2020
6ed6a8e
Check logs from exec_cmd
JayjeetAtGithub Apr 15, 2020
efff84b
Add section on resource manager in docs
JayjeetAtGithub Apr 15, 2020
d938530
Change levels in CONTRIBUTING.md
JayjeetAtGithub Apr 15, 2020
cfdbe90
Fix the description in cmd_run.py
JayjeetAtGithub Apr 15, 2020
093fa46
Show container args in docker create step info
JayjeetAtGithub Apr 15, 2020
f5f974e
PEP8 issues addressed
JayjeetAtGithub Apr 15, 2020
aac79fa
Remove psutil dependency and fix resource leak
JayjeetAtGithub Apr 15, 2020
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion cli/setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@
'docker'
],
extras_require={
'dev': ['psutil', 'testfixtures']
'dev': ['testfixtures']
ivotron marked this conversation as resolved.
Show resolved Hide resolved
},
entry_points='''
[console_scripts]
Expand Down
11 changes: 5 additions & 6 deletions cli/test/test_runner_host.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,6 @@

from subprocess import Popen

import psutil
import docker

import utils as testutils
Expand Down Expand Up @@ -89,13 +88,13 @@ def test_exec_cmd(self):
self.assertEqual(len(_pids), 1)

def test_stop_running_tasks(self):
pid = Popen(["sleep", "2000"]).pid
with HostRunner() as hr:
hr._spawned_pids.add(pid)
hr.stop_running_tasks()
with Popen(["sleep", "2000"]) as p:
pid = p.pid
hr._spawned_pids.add(pid)
hr.stop_running_tasks()
time.sleep(2)
proc = psutil.Process(pid)
self.assertEqual(proc.status(), psutil.STATUS_ZOMBIE)
self.assertRaises(ProcessLookupError, os.kill, pid, 0)
ivotron marked this conversation as resolved.
Show resolved Hide resolved


class TestHostDockerRunner(unittest.TestCase):
Expand Down