Skip to content

Commit

Permalink
FIX: Fixed some old job ids instead of references.
Browse files Browse the repository at this point in the history
  • Loading branch information
d-krupke committed Aug 3, 2024
1 parent 14c5b60 commit 657c4e1
Show file tree
Hide file tree
Showing 3 changed files with 18 additions and 13 deletions.
1 change: 1 addition & 0 deletions README.rst
Original file line number Diff line number Diff line change
Expand Up @@ -359,6 +359,7 @@ The project is reasonably easy:
Changes
-------

- 1.1.2: Fixing some return types for job bundling. Still not perfect. Be aware of potentially breaking changes if you have been using the job ids.
- 1.1.1: Fixing bug when there is some output to stdout when loading the code, such as deprecation warnings.
- 1.1.0: Slurminade can now be called from iPython, too! `exec` has been renamed `shell` to prevent confusion with the Python call `exec` which will evaluate a string as Python code.
- 1.0.1: Dispatcher now return jobs references instead of job ids. This allows to do some fancier stuff in the future, when the jobs infos are only available a short time after the job has been submitted.
Expand Down
2 changes: 1 addition & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ where = ["src"]

[project]
name = "slurminade"
version = "1.1.1"
version = "1.1.2"
authors = [
{ name = "TU Braunschweig, IBR, Algorithms Group (Dominik Krupke)", email = "krupke@ibr.cs.tu-bs.de" },
]
Expand Down
28 changes: 16 additions & 12 deletions src/slurminade/bundling.py
Original file line number Diff line number Diff line change
Expand Up @@ -76,33 +76,37 @@ def __init__(self, max_size: int):
self.max_size = max_size
self.subdispatcher = get_dispatcher()
self._tasks = TaskBuffer()
self._all_job_ids = []
self._all_job_refs = []

def flush(self) -> typing.List[int]:
def flush(self) -> typing.List[JobReference]:
"""
Distribute all buffered tasks. Return the job ids used.
Distribute all buffered tasks. Return the jobs used.
This method is called automatically when the context is exited.
However, you may want to call it manually to get the job ids,
However, you may want to call it manually to get the job references,
for example to use them for dependency management with ``wait_for``.
:param options: Only flush tasks with specific options.
:return: A list of job ids.
:return: A list of job references.
"""
job_ids = []
job_refs = []
for entry_point, opt, tasks_ in self._tasks.items():
tasks = tasks_
while tasks:
job_id = self.subdispatcher(tasks[: self.max_size], opt, entry_point)
job_ids.append(job_id)
job_ref = self.subdispatcher(tasks[: self.max_size], opt, entry_point)
job_refs.append(job_ref)
tasks = tasks[self.max_size :]
self._tasks.clear()
self._all_job_ids.extend(job_ids)
return job_ids
self._all_job_refs.extend(job_refs)
return job_refs

def get_all_job_ids(self):
def get_all_job_ids(self) -> typing.List[int]:
"""
Return all job ids that have been used.
"""
return list(self._all_job_ids)
job_ids = [job_ref.get_job_id() for job_ref in self._all_job_refs]
return [jid for jid in job_ids if jid is not None]

def get_all_jobs(self) -> typing.List[JobReference]:
return list(self._all_job_refs)

def add(self, func: SlurmFunction, *args, **kwargs):
"""
Expand Down

0 comments on commit 657c4e1

Please sign in to comment.