Skip to content

Commit 1d19423

Browse files
authored
gh-89792: Prevent test_tools from copying 1000M of "source" in freeze test (#101837)
Prevent test_tools from copying 1000M of "source" It doesn't need a git repo, just the checkout. We skip .git metadata, Doc/build, Doc/venv, and `__pycache__` subdirs, that developers often have in their clients to reduce the size of the source tree copy ten-fold. This should significantly reduce IO and presumably time on buildbots during this long test.
1 parent 3eb12df commit 1d19423

File tree

2 files changed

+16
-1
lines changed

2 files changed

+16
-1
lines changed
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
``test_tools`` now copies up to 10x less source data to a temporary
2+
directory during the ``freeze`` test by ignoring git metadata and other
3+
artifacts.

Tools/freeze/test/freeze.py

+13-1
Original file line numberDiff line numberDiff line change
@@ -80,7 +80,19 @@ def copy_source_tree(newroot, oldroot):
8080
if newroot == SRCDIR:
8181
raise Exception('this probably isn\'t what you wanted')
8282
shutil.rmtree(newroot)
83-
shutil.copytree(oldroot, newroot)
83+
84+
def ignore_non_src(src, names):
85+
"""Turns what could be a 1000M copy into a 100M copy."""
86+
# Don't copy the ~600M+ of needless git repo metadata.
87+
# source only, ignore cached .pyc files.
88+
subdirs_to_skip = {'.git', '__pycache__'}
89+
if os.path.basename(src) == 'Doc':
90+
# Another potential ~250M+ of non test related data.
91+
subdirs_to_skip.add('build')
92+
subdirs_to_skip.add('venv')
93+
return subdirs_to_skip
94+
95+
shutil.copytree(oldroot, newroot, ignore=ignore_non_src)
8496
if os.path.exists(os.path.join(newroot, 'Makefile')):
8597
_run_quiet([MAKE, 'clean'], newroot)
8698

0 commit comments

Comments
 (0)