Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] Use StorageManager in execute_DAG #258

Draft
wants to merge 95 commits into
base: main
Choose a base branch
from
Draft
Changes from 1 commit
Commits
Show all changes
95 commits
Select commit Hold shift + click to select a range
ffdb901
PathLike wrapper/cache for ExternalStorage
dwhswenson Apr 26, 2023
4ad7fb1
Merge branch 'main' into shared-object-v2
dwhswenson Apr 26, 2023
eb19e0f
mypy
dwhswenson Apr 26, 2023
8e6a78c
Merge branch 'shared-object-v2' of github.com:OpenFreeEnergy/gufe int…
dwhswenson Apr 26, 2023
3be13c0
docstrings
dwhswenson Apr 27, 2023
dda02e2
Merge branch 'main' of github.com:OpenFreeEnergy/gufe into shared-obj…
dwhswenson May 17, 2023
472151e
Add StorageManager code
dwhswenson May 30, 2023
b692c2f
rename to Stagine
dwhswenson May 30, 2023
7432ff4
Add tests for _delete_empty_dirs
dwhswenson May 30, 2023
319d0d0
Storage docs
dwhswenson May 31, 2023
5650609
outline of storage manager tests
dwhswenson May 31, 2023
ddbbd19
minor improvements on staging directory
dwhswenson May 31, 2023
c5ce48a
first storage lifecycle test works
dwhswenson May 31, 2023
236b263
Merge branch 'main' of github.com:OpenFreeEnergy/gufe into shared-obj…
dwhswenson May 31, 2023
b805aac
cleanup mypy
dwhswenson May 31, 2023
ed5e83c
change to unit taking in the label
dwhswenson May 31, 2023
6181039
lots of updates; switched to harness for tests
dwhswenson Jun 5, 2023
1880f73
Big reorg for shared overlapping staging
dwhswenson Jun 6, 2023
1e4ca2c
remove _storage_path_conflict
dwhswenson Jun 6, 2023
a6d26f3
docs & types
dwhswenson Jun 6, 2023
aabbc33
docs, types, logging
dwhswenson Jun 6, 2023
b4d73b3
finish TestHoldingOverlapsPermanentStorageManager
dwhswenson Jun 6, 2023
7af006e
mypy
dwhswenson Jun 6, 2023
58a58bc
test_repr
dwhswenson Jun 6, 2023
8e429f5
renaming around DAGContextManager
dwhswenson Jun 6, 2023
b70df48
holding => staging
dwhswenson Jun 7, 2023
08e3ac2
finish docs (I think?)
dwhswenson Jun 7, 2023
ca7871b
remove completed TODO
dwhswenson Jun 7, 2023
2aa0616
start to testing edge case logging
dwhswenson Jun 9, 2023
7c03dcd
Update stagingdirectory.py
richardjgowers Jun 12, 2023
383075e
tests for single_file_transfer logging
dwhswenson Jun 12, 2023
6365398
tests for read-only transfers
dwhswenson Jun 16, 2023
d35bd60
fix repr and cleanup tests
dwhswenson Jun 16, 2023
7cc10f9
test for permanent transfer to external
dwhswenson Jun 16, 2023
ab025f1
test for Permanent delete staging
dwhswenson Jun 17, 2023
cd70ab2
Add test for missing file on cleanup
dwhswenson Jun 17, 2023
ac1b1d0
Merge branch 'shared-object-v2' of github.com:OpenFreeEnergy/gufe int…
dwhswenson Jun 17, 2023
ea054be
Merge branch 'main' of github.com:OpenFreeEnergy/gufe into shared-obj…
dwhswenson Jun 22, 2023
90f2597
get_other_shared to private
dwhswenson Jun 22, 2023
a2e05b2
Merge branch 'main' into shared-object-v2
dwhswenson Jul 6, 2023
80eccc4
Merge branch 'main' into shared-object-v2
dwhswenson Aug 28, 2023
e9ed7a8
Merge branch 'main' of github.com:OpenFreeEnergy/gufe into shared-obj…
dwhswenson Sep 8, 2023
b4e1d42
Merge branch 'main' into shared-object-v2
dotsdl Sep 8, 2023
2b070ca
Merge branch 'shared-object-v2' of github.com:OpenFreeEnergy/gufe int…
dwhswenson Sep 9, 2023
4fd4a66
Merge branch 'main' into shared-object-v2
dotsdl Sep 12, 2023
5e56461
Merge branch 'main' into shared-object-v2
dotsdl Sep 19, 2023
73d3a1e
Merge branch 'main' into shared-object-v2
dwhswenson Nov 3, 2023
524cc6e
Merge branch 'main' of github.com:OpenFreeEnergy/gufe into shared-obj…
dwhswenson Dec 1, 2023
dd1b6dc
updates from other branch
dwhswenson Dec 1, 2023
a575dd3
make mypy happy
dwhswenson Dec 1, 2023
24d3820
Use StorageManager/StagingPath in execute_DAG
dwhswenson Dec 1, 2023
0c973b7
fix up protocol tests
dwhswenson Dec 1, 2023
9260a2d
Add __eq__ for ext resources
dwhswenson Dec 1, 2023
d97fca4
Test: FileStorage.store_path for nested target
dwhswenson Dec 1, 2023
a01feaf
add tests for ext resource __eq__
dwhswenson Dec 1, 2023
ba6fcff
Merge branch 'main' into shared-object-v2
dwhswenson Dec 1, 2023
11810eb
make mypy happy
dwhswenson Dec 1, 2023
908329c
some docstrings
dwhswenson Dec 1, 2023
d8507d4
Merge branch 'ext-resource-cleanup' into staging-execute-dag
dwhswenson Dec 1, 2023
a659afa
staging serialization
dwhswenson Dec 1, 2023
32b1fe4
tests for staging serialization
dwhswenson Dec 1, 2023
5d0df5f
pep8
dwhswenson Dec 1, 2023
1418aee
Merge branch 'shared-object-v2' of github.com:OpenFreeEnergy/gufe int…
dwhswenson Dec 1, 2023
e057332
pep8
dwhswenson Dec 1, 2023
bdb37bb
pep8
dwhswenson Dec 1, 2023
e39a790
pep8
dwhswenson Dec 1, 2023
1cfe910
StagingDirectory -> StagingRegistry
dwhswenson Dec 4, 2023
78e003b
remove prefix; remove get_other_shared
dwhswenson Dec 7, 2023
265e786
delete_empty_dirs => keep_empty_dirs
dwhswenson Dec 7, 2023
006d787
Merge branch 'main' into shared-object-v2
dwhswenson Dec 11, 2023
ce12326
Add logging to not clean up registered directory
dwhswenson Dec 11, 2023
9afeb2f
Merge branch 'shared-object-v2' of github.com:OpenFreeEnergy/gufe int…
dwhswenson Dec 11, 2023
b3fea47
Merge branch 'shared-object-v2' into staging-execute-dag
dwhswenson Dec 11, 2023
2420f6c
remove unneeded comment
dwhswenson Dec 11, 2023
fe64baa
yield Context instead of tuple
dwhswenson Dec 11, 2023
288e6cc
ReproduceOldBehaviorStorageManager.from_old_args
dwhswenson Dec 11, 2023
bc1f36f
tests for ReproduceOldBehaviorStorageManager.from_old_args
dwhswenson Dec 11, 2023
aaa2aab
StagingPath.fspath => StagingPath.as_path()
dwhswenson Dec 13, 2023
cf60d1b
pep8
dwhswenson Dec 13, 2023
d0af5cf
Merge branch 'shared-object-v2' into staging-execute-dag
dwhswenson Dec 14, 2023
4861b3e
Add tests for nested files in shared
dwhswenson Dec 14, 2023
c0a37b3
tests for empty directories
dwhswenson Dec 14, 2023
3d7ca7b
tests for the directories in the overlap example
dwhswenson Dec 14, 2023
529b8a6
Merge branch 'staging-execute-dag' into staging-serialization
dwhswenson Dec 14, 2023
c412611
update for changes in preceding PRs
dwhswenson Dec 14, 2023
bf49c34
First version of the change storage backend story
dwhswenson Dec 14, 2023
10a40eb
docstrings; sketch out yet ANOTHER user story
dwhswenson Dec 15, 2023
4b2510c
minor docs update
dwhswenson Dec 15, 2023
3390bb0
test new codec added in staging serialization
dwhswenson Dec 21, 2023
da9955e
remove fspath from StagingRegistry
dwhswenson Dec 22, 2023
b72e6f9
Merge branch 'shared-object-v2' into staging-execute-dag
dwhswenson Dec 22, 2023
409ec06
Merge branch 'staging-execute-dag' into staging-serialization
dwhswenson Dec 22, 2023
b22445b
Attach StorageSerialization to StorageManager
dwhswenson Dec 22, 2023
8534d71
update for some review comments
dwhswenson Jan 17, 2024
01db5b1
Merge branch 'main' into staging-execute-dag
dwhswenson Jan 22, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
tests for the directories in the overlap example
  • Loading branch information
dwhswenson committed Dec 14, 2023

Unverified

The committer email address is not verified.
commit 3d7ca7b55de18e8e78ac08f92c4e40f9b827c9b8
26 changes: 22 additions & 4 deletions gufe/tests/storage/test_storage_demo.py
Original file line number Diff line number Diff line change
@@ -359,10 +359,26 @@ def get_storage_manager(self, keep, tmp_path):
)
return storage_manager

def test_overlap_directories(storage_manager):
def test_overlap_directories(self, tmp_path):
# test that the staging and shared/permanent backends overlap as
# expected
pytest.skip()
# expected; basic idea is that creating a file in staging
# automatically creates a file in the shared/permanent external
# resources
storage_manager = self.get_storage_manager(keep="nothing",
tmp_path=tmp_path)
shared = storage_manager.shared_root
perm = storage_manager.permanent_root
stage1 = storage_manager.shared_staging / "foo/file.txt"
assert len(storage_manager.shared_staging.registry) == 0
assert list(shared.iter_contents()) == []
assert list(perm.iter_contents()) == []
pathlib.Path(stage1).touch()
assert len(storage_manager.shared_staging.registry) == 1
# because iter_contents actually loops over the directory, these
# have effectively been automatically added to the external storage
# resources
assert list(shared.iter_contents()) == ["foo/file.txt"]
assert list(perm.iter_contents()) == ["foo/file.txt"]

def assert_shared_and_permanent(self, storage_manager, dag):
shared = storage_manager.shared_root
@@ -417,7 +433,9 @@ def assert_staging(self, storage_manager, dag):
def assert_empty_directories(self, storage_manager, dag):
# in this case, the staging directories should not have been cleaned
# (because they overlap with storage), so all should exist. They
# will only be empty if `keep_shared is False`
# will only be empty if `keep_shared is False`. NOTE: I don't think
# it would be an API break here if code changed to ensure that all
# empty directories were cleaned out
u1_label = self.u1_label(dag)
staging = storage_manager.scratch_root / storage_manager.staging