bpo-9949: Enable symlink traversal for ntpath.realpath

[zooba]

https://bugs.python.org/issue9949

[zooba]

This is ready to merge once CI passes. I'll do it and then rebase #15231 before merging that one.

[miss-islington]

Thanks @zooba for the PR 🌮🎉.. I'm working now to backport this PR to: 3.8.
🐍🍒⛏🤖

[bedevere-bot]

GH-15367 is a backport of this pull request to the 3.8 branch.

[bedevere-bot]

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 Windows7 SP1 3.x has failed when building commit 75e0649.

What do you need to do:

1. Don't panic.
2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
3. Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/40/builds/2885) and take a look at the build logs.
4. Check if the failure is related to this commit (75e0649) or if it is a false positive.
5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/40/builds/2885

Failed tests:

• test_venv
• test_ntpath

Failed subtests:

• test_realpath_curdir - test.test_ntpath.TestNtpath
• test_unicode_in_batch_file - test.test_venv.BasicTest

Summary of the results of the build (if available):

== Tests result: FAILURE then FAILURE ==

384 tests OK.

10 slowest tests:

• test_multiprocessing_spawn: 2 min 41 sec
• test_tools: 2 min 19 sec
• test_concurrent_futures: 1 min 37 sec
• test_tokenize: 1 min 33 sec
• test_lib2to3: 1 min 13 sec
• test_asyncio: 1 min 5 sec
• test_io: 1 min 3 sec
• test_mmap: 1 min 3 sec
• test_socket: 58 sec 471 ms
• test_venv: 54 sec 83 ms

2 tests failed:
test_ntpath test_venv

33 tests skipped:
test_curses test_dbm_gnu test_dbm_ndbm test_devpoll test_epoll
test_fcntl test_fork1 test_gdb test_grp test_ioctl test_kqueue
test_multiprocessing_fork test_multiprocessing_forkserver test_nis
test_openpty test_ossaudiodev test_pipes test_poll test_posix
test_pty test_pwd test_readline test_resource test_spwd
test_syslog test_threadsignals test_tix test_tk test_ttk_guionly
test_wait3 test_wait4 test_xxtestfuzz test_zipfile64

2 re-run tests:
test_ntpath test_venv

Total duration: 11 min 4 sec

Click to see traceback logs

Traceback (most recent call last):
  File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\test\test_ntpath.py", line 210, in test_realpath_curdir
    tester("ntpath.realpath('/'.join(['.'] * 100))", expected)
  File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\test\test_ntpath.py", line 30, in tester
    raise TestFailed("%s should return: %s but returned: %s" \


Traceback (most recent call last):
  File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\test\test_venv.py", line 337, in test_unicode_in_batch_file
    out, err = check_output(
  File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\test\test_venv.py", line 42, in check_output
    raise subprocess.CalledProcessError(
subprocess.CalledProcessError: Command '['\\\\?\\C:\\Users\\Buildbot\\AppData\\Local\\Temp\\tmprmrzilsg\\\u03fc\u045e\u0422\u03bb\u0424\u0419\\Scripts\\activate.bat', '&', 'python_d.exe', '-c', 'print(0)']' returned non-zero exit status 1.


Traceback (most recent call last):
  File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\test\test_venv.py", line 337, in test_unicode_in_batch_file
    out, err = check_output(
  File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\test\test_venv.py", line 42, in check_output
    raise subprocess.CalledProcessError(
subprocess.CalledProcessError: Command '['\\\\?\\C:\\Users\\Buildbot\\AppData\\Local\\Temp\\tmp9l9odtp0\\\u03fc\u045e\u0422\u03bb\u0424\u0419\\Scripts\\activate.bat', '&', 'python_d.exe', '-c', 'print(0)']' returned non-zero exit status 1.

[bedevere-bot]

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 Windows10 3.x has failed when building commit 75e0649.

What do you need to do:

1. Don't panic.
2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
3. Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/3/builds/3298) and take a look at the build logs.
4. Check if the failure is related to this commit (75e0649) or if it is a false positive.
5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/3/builds/3298

Failed tests:

• test_venv
• test_ntpath

Failed subtests:

• test_realpath_curdir - test.test_ntpath.TestNtpath
• test_unicode_in_batch_file - test.test_venv.BasicTest

Summary of the results of the build (if available):

== Tests result: FAILURE then FAILURE ==

387 tests OK.

10 slowest tests:

• test_mmap: 5 min 51 sec
• test_tools: 3 min 8 sec
• test_multiprocessing_spawn: 2 min 59 sec
• test_io: 2 min 39 sec
• test_largefile: 2 min 25 sec
• test_tokenize: 2 min 19 sec
• test_lib2to3: 1 min 55 sec
• test_concurrent_futures: 1 min 43 sec
• test_pickle: 55 sec 907 ms
• test_asyncio: 55 sec 177 ms

2 tests failed:
test_ntpath test_venv

30 tests skipped:
test_curses test_dbm_gnu test_dbm_ndbm test_devpoll test_epoll
test_fcntl test_fork1 test_gdb test_grp test_ioctl test_kqueue
test_multiprocessing_fork test_multiprocessing_forkserver test_nis
test_openpty test_ossaudiodev test_pipes test_poll test_posix
test_pty test_pwd test_readline test_resource test_spwd
test_syslog test_threadsignals test_wait3 test_wait4
test_xxtestfuzz test_zipfile64

2 re-run tests:
test_ntpath test_venv

Total duration: 13 min 37 sec

Click to see traceback logs

Traceback (most recent call last):
  File "D:\buildarea\3.x.bolen-windows10\build\lib\test\test_ntpath.py", line 210, in test_realpath_curdir
    tester("ntpath.realpath('/'.join(['.'] * 100))", expected)
  File "D:\buildarea\3.x.bolen-windows10\build\lib\test\test_ntpath.py", line 30, in tester
    raise TestFailed("%s should return: %s but returned: %s" \


Traceback (most recent call last):
  File "D:\buildarea\3.x.bolen-windows10\build\lib\test\test_venv.py", line 337, in test_unicode_in_batch_file
    out, err = check_output(
  File "D:\buildarea\3.x.bolen-windows10\build\lib\test\test_venv.py", line 42, in check_output
    raise subprocess.CalledProcessError(
subprocess.CalledProcessError: Command '['\\\\?\\D:\\temp\\tmpjvhg3gb8\\\u03fc\u045e\u0422\u03bb\u0424\u0419\\Scripts\\activate.bat', '&', 'python_d.exe', '-c', 'print(0)']' returned non-zero exit status 1.


Traceback (most recent call last):
  File "D:\buildarea\3.x.bolen-windows10\build\lib\test\test_venv.py", line 337, in test_unicode_in_batch_file
    out, err = check_output(
  File "D:\buildarea\3.x.bolen-windows10\build\lib\test\test_venv.py", line 42, in check_output
    raise subprocess.CalledProcessError(
subprocess.CalledProcessError: Command '['\\\\?\\D:\\temp\\tmp6vl8d2pf\\\u03fc\u045e\u0422\u03bb\u0424\u0419\\Scripts\\activate.bat', '&', 'python_d.exe', '-c', 'print(0)']' returned non-zero exit status 1.

[bedevere-bot]

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 Windows7 SP1 3.8 has failed when building commit c30c869.

What do you need to do:

1. Don't panic.
2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
3. Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/208/builds/279) and take a look at the build logs.
4. Check if the failure is related to this commit (c30c869) or if it is a false positive.
5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/208/builds/279

Failed tests:

• test_venv
• test_ntpath

Failed subtests:

• test_realpath_curdir - test.test_ntpath.TestNtpath
• test_unicode_in_batch_file - test.test_venv.BasicTest

Summary of the results of the build (if available):

== Tests result: FAILURE then FAILURE ==

388 tests OK.

10 slowest tests:

• test_multiprocessing_spawn: 2 min 58 sec
• test_tools: 2 min 47 sec
• test_concurrent_futures: 1 min 49 sec
• test_tokenize: 1 min 34 sec
• test_zipfile: 1 min 14 sec
• test_lib2to3: 1 min 13 sec
• test_mmap: 1 min 9 sec
• test_asyncio: 1 min 3 sec
• test_socket: 57 sec 298 ms
• test_venv: 56 sec 438 ms

2 tests failed:
test_ntpath test_venv

33 tests skipped:
test_curses test_dbm_gnu test_dbm_ndbm test_devpoll test_epoll
test_fcntl test_fork1 test_gdb test_grp test_ioctl test_kqueue
test_multiprocessing_fork test_multiprocessing_forkserver test_nis
test_openpty test_ossaudiodev test_pipes test_poll test_posix
test_pty test_pwd test_readline test_resource test_spwd
test_syslog test_threadsignals test_tix test_tk test_ttk_guionly
test_wait3 test_wait4 test_xxtestfuzz test_zipfile64

2 re-run tests:
test_ntpath test_venv

Total duration: 11 min 14 sec

Click to see traceback logs

Traceback (most recent call last):
  File "C:\buildbot.python.org\3.8.kloth-win64\build\lib\test\test_venv.py", line 315, in test_unicode_in_batch_file
    out, err = check_output(
  File "C:\buildbot.python.org\3.8.kloth-win64\build\lib\test\test_venv.py", line 42, in check_output
    raise subprocess.CalledProcessError(
subprocess.CalledProcessError: Command '['\\\\?\\C:\\Users\\Buildbot\\AppData\\Local\\Temp\\tmpyua593m9\\\u03fc\u045e\u0422\u03bb\u0424\u0419\\Scripts\\activate.bat', '&', 'python_d.exe', '-c', 'print(0)']' returned non-zero exit status 1.


Traceback (most recent call last):
  File "C:\buildbot.python.org\3.8.kloth-win64\build\lib\test\test_ntpath.py", line 210, in test_realpath_curdir
    tester("ntpath.realpath('/'.join(['.'] * 100))", expected)
  File "C:\buildbot.python.org\3.8.kloth-win64\build\lib\test\test_ntpath.py", line 30, in tester
    raise TestFailed("%s should return: %s but returned: %s" \


Traceback (most recent call last):
  File "C:\buildbot.python.org\3.8.kloth-win64\build\lib\test\test_venv.py", line 315, in test_unicode_in_batch_file
    out, err = check_output(
  File "C:\buildbot.python.org\3.8.kloth-win64\build\lib\test\test_venv.py", line 42, in check_output
    raise subprocess.CalledProcessError(
subprocess.CalledProcessError: Command '['\\\\?\\C:\\Users\\Buildbot\\AppData\\Local\\Temp\\tmpux9eusss\\\u03fc\u045e\u0422\u03bb\u0424\u0419\\Scripts\\activate.bat', '&', 'python_d.exe', '-c', 'print(0)']' returned non-zero exit status 1.

[bedevere-bot]

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 Windows10 3.8 has failed when building commit c30c869.

What do you need to do:

1. Don't panic.
2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
3. Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/217/builds/269) and take a look at the build logs.
4. Check if the failure is related to this commit (c30c869) or if it is a false positive.
5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/217/builds/269

Failed tests:

• test_venv
• test_ntpath

Failed subtests:

• test_realpath_curdir - test.test_ntpath.TestNtpath
• test_unicode_in_batch_file - test.test_venv.BasicTest

Summary of the results of the build (if available):

== Tests result: FAILURE then FAILURE ==

391 tests OK.

10 slowest tests:

• test_mmap: 5 min 49 sec
• test_io: 2 min 47 sec
• test_multiprocessing_spawn: 2 min 38 sec
• test_largefile: 2 min 26 sec
• test_tools: 2 min 10 sec
• test_tokenize: 1 min 44 sec
• test_concurrent_futures: 1 min 41 sec
• test_lib2to3: 1 min 21 sec
• test_asyncio: 57 sec 360 ms
• test_pickle: 46 sec 703 ms

2 tests failed:
test_ntpath test_venv

30 tests skipped:
test_curses test_dbm_gnu test_dbm_ndbm test_devpoll test_epoll
test_fcntl test_fork1 test_gdb test_grp test_ioctl test_kqueue
test_multiprocessing_fork test_multiprocessing_forkserver test_nis
test_openpty test_ossaudiodev test_pipes test_poll test_posix
test_pty test_pwd test_readline test_resource test_spwd
test_syslog test_threadsignals test_wait3 test_wait4
test_xxtestfuzz test_zipfile64

2 re-run tests:
test_ntpath test_venv

Total duration: 14 min 45 sec

Click to see traceback logs

Traceback (most recent call last):
  File "D:\buildarea\3.8.bolen-windows10\build\lib\test\test_venv.py", line 315, in test_unicode_in_batch_file
    out, err = check_output(
  File "D:\buildarea\3.8.bolen-windows10\build\lib\test\test_venv.py", line 42, in check_output
    raise subprocess.CalledProcessError(
subprocess.CalledProcessError: Command '['\\\\?\\D:\\temp\\tmp_ejyuk4y\\\u03fc\u045e\u0422\u03bb\u0424\u0419\\Scripts\\activate.bat', '&', 'python_d.exe', '-c', 'print(0)']' returned non-zero exit status 1.


Traceback (most recent call last):
  File "D:\buildarea\3.8.bolen-windows10\build\lib\test\test_ntpath.py", line 210, in test_realpath_curdir
    tester("ntpath.realpath('/'.join(['.'] * 100))", expected)
  File "D:\buildarea\3.8.bolen-windows10\build\lib\test\test_ntpath.py", line 30, in tester
    raise TestFailed("%s should return: %s but returned: %s" \


Traceback (most recent call last):
  File "D:\buildarea\3.8.bolen-windows10\build\lib\test\test_venv.py", line 315, in test_unicode_in_batch_file
    out, err = check_output(
  File "D:\buildarea\3.8.bolen-windows10\build\lib\test\test_venv.py", line 42, in check_output
    raise subprocess.CalledProcessError(
subprocess.CalledProcessError: Command '['\\\\?\\D:\\temp\\tmp7utx1za6\\\u03fc\u045e\u0422\u03bb\u0424\u0419\\Scripts\\activate.bat', '&', 'python_d.exe', '-c', 'print(0)']' returned non-zero exit status 1.

[bedevere-bot]

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot x86 Windows7 3.8 has failed when building commit c30c869.

What do you need to do:

1. Don't panic.
2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
3. Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/210/builds/208) and take a look at the build logs.
4. Check if the failure is related to this commit (c30c869) or if it is a false positive.
5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/210/builds/208

Failed tests:

• test_venv
• test_ntpath

Failed subtests:

• test_realpath_curdir - test.test_ntpath.TestNtpath
• test_unicode_in_batch_file - test.test_venv.BasicTest

Summary of the results of the build (if available):

== Tests result: FAILURE then FAILURE ==

391 tests OK.

10 slowest tests:

• test_multiprocessing_spawn: 15 min 18 sec
• test_zipfile: 7 min 48 sec
• test_concurrent_futures: 4 min 44 sec
• test_tarfile: 3 min 18 sec
• test_pickle: 3 min 15 sec
• test_lzma: 3 min 2 sec
• test_asyncio: 3 min 372 ms
• test_capi: 2 min 57 sec
• test_regrtest: 2 min 52 sec
• test_venv: 2 min 39 sec

2 tests failed:
test_ntpath test_venv

30 tests skipped:
test_curses test_dbm_gnu test_dbm_ndbm test_devpoll test_epoll
test_fcntl test_fork1 test_gdb test_grp test_ioctl test_kqueue
test_multiprocessing_fork test_multiprocessing_forkserver test_nis
test_openpty test_ossaudiodev test_pipes test_poll test_posix
test_pty test_pwd test_readline test_resource test_spwd
test_syslog test_threadsignals test_wait3 test_wait4
test_xxtestfuzz test_zipfile64

2 re-run tests:
test_ntpath test_venv

Total duration: 1 hour 5 min

Click to see traceback logs

Traceback (most recent call last):
  File "D:\cygwin\home\db3l\buildarea\3.8.bolen-windows7\build\lib\test\test_venv.py", line 315, in test_unicode_in_batch_file
    out, err = check_output(
  File "D:\cygwin\home\db3l\buildarea\3.8.bolen-windows7\build\lib\test\test_venv.py", line 42, in check_output
    raise subprocess.CalledProcessError(
subprocess.CalledProcessError: Command '['\\\\?\\D:\\temp\\tmpddif0hoz\\\u03fc\u045e\u0422\u03bb\u0424\u0419\\Scripts\\activate.bat', '&', 'python_d.exe', '-c', 'print(0)']' returned non-zero exit status 1.


Traceback (most recent call last):
  File "D:\cygwin\home\db3l\buildarea\3.8.bolen-windows7\build\lib\test\test_venv.py", line 315, in test_unicode_in_batch_file
    out, err = check_output(
  File "D:\cygwin\home\db3l\buildarea\3.8.bolen-windows7\build\lib\test\test_venv.py", line 42, in check_output
    raise subprocess.CalledProcessError(
subprocess.CalledProcessError: Command '['\\\\?\\D:\\temp\\tmpdfbt5vu7\\\u03fc\u045e\u0422\u03bb\u0424\u0419\\Scripts\\activate.bat', '&', 'python_d.exe', '-c', 'print(0)']' returned non-zero exit status 1.


Traceback (most recent call last):
  File "D:\cygwin\home\db3l\buildarea\3.8.bolen-windows7\build\lib\test\test_ntpath.py", line 210, in test_realpath_curdir
    tester("ntpath.realpath('/'.join(['.'] * 100))", expected)
  File "D:\cygwin\home\db3l\buildarea\3.8.bolen-windows7\build\lib\test\test_ntpath.py", line 30, in tester
    raise TestFailed("%s should return: %s but returned: %s" \

[eryksun]

If you're allowing reading junction targets, you must fail a remote path here if path is a junction, per os.lstat. A junction target in a UNC path is meaningless to us. A junction has to target local devices on a system, i.e. its local DOS drive and volume GUID names. At best these aren't defined for us, and at worst they map to an unrelated drive on our side. If for some reason junction "spam" fails to resolve on the server, we cannot go any further. We leave it as "spam" in the UNC path, and we're done.

[eryksun]

If path is a relative symlink, it must be evaluated relative to its parent directory, not out current directory.