Bug fix: Use safe rename which works across filesystems when writing checkpoints

[dantreiman]

On some systems, the temp directory and model output directory live on different filesystems. When we write checkpoint to temp directory, then os.replace or os.rename to the destination, we get:

    134 # replace is an atomic operation in python
    135 # it is POSIX compliant according to docs
    136 # https://docs.python.org/3/library/os.html#os.replace
--> 137 os.replace(tmp_path, save_path)
    138 logging.debug(f"Saved checkpoint at {save_path}.")

OSError: [Errno 18] Invalid cross-device link: '/tmp/tmphmthsa9s/temp.ckpt' -> '/tf/app/.../model/training_checkpoints/latest.ckpt'

This PR adds safe_move_file, which attempts to os.replace(), and does an atomic shutil.copyfile if we get the cross-filesystem error (errno.EXDEV == 18).

[github-actions]

Unit Test Results

       5 files   -     1         5 suites   - 1   1h 51m 14s ⏱️ - 45m 56s
2 908 tests ±    0  2 862 ✔️ ±    0    46 💤 ±  0  0 ❌ ±0 
8 556 runs   - 168  8 437 ✔️  - 145  119 💤  - 23  0 ❌ ±0 

Results for commit 696ec6d. ± Comparison against base commit 9d7da5f.