test_persistentdataset_dist error on windows

[wyli]

Describe the bug
the error occurs randomly:

test_mp_dataset (tests.test_persistentdataset_dist.TestDistDataset) ... Process SpawnProcess-51:
persistent 1
Traceback (most recent call last):
  File "C:\hostedtoolcache\windows\Python\3.8.10\x64\lib\shutil.py", line 791, in move
    os.rename(src, real_dst)
FileExistsError: [WinError 183] Cannot create a file when that file already exists: 'C:\\Users\\RUNNER~1\\AppData\\Local\\Temp\\tmp2ndb3qxg\\54c0443faf33e022375c4a3adc6afd11.pt' -> 'C:\\Users\\RUNNER~1\\AppData\\Local\\Temp\\tmpovt2z4gf\\test\\54c0443faf33e022375c4a3adc6afd11.pt'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\hostedtoolcache\windows\Python\3.8.10\x64\lib\multiprocessing\process.py", line 315, in _bootstrap
    self.run()
  File "C:\hostedtoolcache\windows\Python\3.8.10\x64\lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "D:\a\MONAI\MONAI\tests\utils.py", line 396, in run_process
    raise e
  File "D:\a\MONAI\MONAI\tests\utils.py", line 387, in run_process
    func(*args, **kwargs)
  File "D:\a\MONAI\MONAI\tests\utils.py", line 557, in _call_original_func
    return f(*args, **kwargs)
  File "D:\a\MONAI\MONAI\tests\test_persistentdataset_dist.py", line 73, in test_mp_dataset
    self.assertEqual(list(ds1), list(ds))
  File "D:\a\MONAI\MONAI\monai\data\dataset.py", line 97, in __getitem__
    return self._transform(index)
  File "D:\a\MONAI\MONAI\monai\data\dataset.py", line 360, in _transform
    pre_random_item = self._cachecheck(self.data[index])
  File "D:\a\MONAI\MONAI\monai\data\dataset.py", line 354, in _cachecheck
    shutil.move(temp_hash_file, hashfile)
  File "C:\hostedtoolcache\windows\Python\3.8.10\x64\lib\shutil.py", line 811, in move
    copy_function(src, real_dst)
  File "C:\hostedtoolcache\windows\Python\3.8.10\x64\lib\shutil.py", line 435, in copy2
    copyfile(src, dst, follow_symlinks=follow_symlinks)
  File "C:\hostedtoolcache\windows\Python\3.8.10\x64\lib\shutil.py", line 264, in copyfile
    with open(src, 'rb') as fsrc, open(dst, 'wb') as fdst:
PermissionError: [Errno 13] Permission denied: 'C:\\Users\\RUNNER~1\\AppData\\Local\\Temp\\tmpovt2z4gf\\test\\54c0443faf33e022375c4a3adc6afd11.pt'
persistent 0
FAIL

--- update ---
A similar issue was reported at bpo-29982 where the calling app tempfile.TemporaryDirectory to shutil should handle the exception. A python PR has been merged to address that issue python/cpython#24793.
Before that PR is officially released in python 3.10, we handle the exception with another try...except in #3633.

[Nic-Ma]

Hi @wyli ,

I remember I saw this issue before, let me try to find it out.

Thanks.

[wyli]

thanks, I put a patch here: #3615

[Nic-Ma]

Hi @wyli ,

May I know where you found this error? It always happens or only rarely happens in CI env?

Thanks.

[wyli]

the most recent error is here https://github.com/Project-MONAI/MONAI/runs/4748467145?check_suite_focus=true. I think it becomes more frequent after this testing script update: 7f23f38#diff-8b8238839d9c98a26d9a1d5a504a38807afa0dbee153e4d565de8e617e98e51f

[Nic-Ma]

OK, do you have any idea what's update in this diff made it more frequent? I didn't see any obvious thing.

Thanks.

[Nic-Ma]

Hi @wyli ,

I checked some pages about this error, maybe it's related to: The test program doesn't have permission to create file on windows C: section: https://www.cnblogs.com/mstk/p/14724509.html
Do you know how to set our test program of windows CI to run as admin?

Thanks.

[Nic-Ma]

Hi @wyli ,

I just saw this issue again when running CI for PR #3625.
Do you think it's related to my above analysis about windows admin permission when the temp is randomly on C: section?

Thanks.

[wyli]

looks like a known issue python/cpython#24793 and python 3.10 has an option ignore_cleanup_errors to address it.

[Nic-Ma]

looks like a known issue python/cpython#24793 and python 3.10 has an option ignore_cleanup_errors to address it.

Hi @wyli ,

Thanks for your investigation, so if someone faces this issue, we can ask him to update python?

[wyli]

looks like a known issue python/cpython#24793 and python 3.10 has an option ignore_cleanup_errors to address it.

Hi @wyli ,

Thanks for your investigation, so if someone faces this issue, we can ask him to update python?

thanks, I updated the ticket to include more information.