branch run with "GET_REFCASE=.TRUE." failed due to failure in removing "spio_stats"

[minxu74]

The error is caused by the stage_refcase to link all files in the refcase directory to the branch run directory including "timing" and "spio_stats". Details can be seen the CIME issue at ESMCI/cime#4697.

I created the issue here just for reference due to the error from `cime_config/customize/provenance.py'.

  File "/var/spool/slurmd/job32040578/slurm_script", line 82, in <module>
    _main_func(__doc__)
  File "/var/spool/slurmd/job32040578/slurm_script", line 77, in _main_func
    success = case.case_run(skip_pnl=skip_pnl, set_continue_run=set_continue_run, submit_resubmits=resubmit)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/global/u2/m/minxu/MyGit/e3sm_v3/cime/CIME/case/case_run.py", line 498, in case_run
    lid = _run_model(self, lid, skip_pnl, da_cycle=cycle)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/global/u2/m/minxu/MyGit/e3sm_v3/cime/CIME/case/case_run.py", line 290, in _run_model
    return run_and_log_case_status(
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/global/u2/m/minxu/MyGit/e3sm_v3/cime/CIME/status.py", line 88, in run_and_log_case_status
    rv = func()
         ^^^^^^
  File "/global/u2/m/minxu/MyGit/e3sm_v3/cime/CIME/case/case_run.py", line 281, in <lambda>
    functor = lambda: _run_model_impl(case, lid, skip_pnl=skip_pnl, da_cycle=da_cycle)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/global/u2/m/minxu/MyGit/e3sm_v3/cime/CIME/case/case_run.py", line 158, in _run_model_impl
    Config.instance().save_prerun_provenance(case)
  File "/global/u2/m/minxu/MyGit/e3sm_v3/cime_config/customize/provenance.py", line 324, in save_prerun_provenance
    _cleanup_spio_stats(case)
  File "/global/u2/m/minxu/MyGit/e3sm_v3/cime_config/customize/provenance.py", line 498, in _cleanup_spio_stats
    shutil.rmtree(spio_stats_dir)
  File "/global/common/software/nersc/pe/conda-envs/24.1.0/python-3.11/nersc-python/lib/python3.11/shutil.py", line 744, in rmtree
    onerror(os.path.islink, path, sys.exc_info())
  File "/global/common/software/nersc/pe/conda-envs/24.1.0/python-3.11/nersc-python/lib/python3.11/shutil.py", line 742, in rmtree
    raise OSError("Cannot call rmtree on a symbolic link")

[jasonb5]

Will be resolved with ESMCI/cime#4698