[Install] Fix error during python/tvm installation

[vacu9708]

Expected behavior

"pip install -e /path-to-tvm/python" successfully installs the TVM package as documented on https://tvm.apache.org/docs/install/from_source.html#

Issue

If "libtvm.so" and "libtvm_runtime.so" remain in "python/tvm" for some reason,
If a previous pip installation was unsuccessful and the cleanup was not performed, running "pip install -e /path-to-tvm/python" fails because shutil.copy() and shutil.copytree() raise exceptions due to the remaining temporary files.

How I encountered the issue

I ran "pip install" without "sudo" first, and it failed without "sudo". (probably because I was not in a virtual environment)
Here the cleanup code couldn't be executed

Error log

  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [21 lines of output]
      Use git describe based version 0.20.dev125+g8ad9bffb1
      Traceback (most recent call last):
        File "/usr/lib/python3/dist-packages/pip/_vendor/pep517/in_process/_in_process.py", line 363, in <module>
          main()
        File "/usr/lib/python3/dist-packages/pip/_vendor/pep517/in_process/_in_process.py", line 345, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
        File "/usr/lib/python3/dist-packages/pip/_vendor/pep517/in_process/_in_process.py", line 130, in get_requires_for_build_wheel
          return hook(config_settings)
        File "/usr/lib/python3/dist-packages/setuptools/build_meta.py", line 162, in get_requires_for_build_wheel
          return self._get_build_requires(
        File "/usr/lib/python3/dist-packages/setuptools/build_meta.py", line 143, in _get_build_requires
          self.run_setup()
        File "/usr/lib/python3/dist-packages/setuptools/build_meta.py", line 158, in run_setup
          exec(compile(code, __file__, 'exec'), locals())
        File "setup.py", line 197, in <module>
          shutil.copy(path, os.path.join(CURRENT_DIR, "tvm"))
        File "/usr/lib/python3.10/shutil.py", line 417, in copy
          copyfile(src, dst, follow_symlinks=follow_symlinks)
        File "/usr/lib/python3.10/shutil.py", line 234, in copyfile
          raise SameFileError("{!r} and {!r} are the same file".format(src, dst))
      shutil.SameFileError: '/home/youngsik/Documents/open_source/AI/tvm/python/tvm/libtvm.so' and 'tvm/libtvm.so' are the same file
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

[Hzfengsy]

cc @tqchen

[mshr-h]

@vacu9708 Thanks.
I hit the same error before, so solving it makes sense to me. Can we just overwrite the file/directory instead of skipping it?

[vacu9708]

@mshr-h Your opinion makes sense. Overwriting instead of skipping would be the correct modification. I'll make the change shortly.

[cchung100m]

LGTM, thanks to @vacu9708 😄

[tqchen]

@mshr-h can you confirm if the patch fixes your error?

[mshr-h]

Still fails...

Here's my repro.

$ python3 -m venv .venv
$ source .venv/bin/activate
$ pip3 install cmake ninja setuptools cython
$ cmake -S . -B build -G Ninja
$ cmake --build build
$ cp build/libtvm_runtime.so python/tvm/ # copy the shared library to raise shutil.SameFileError
$ cd python
$ python3 setup.py develop
Use git describe based version 0.20.dev158+gbf6121656
Traceback (most recent call last):
  File "/home/ubuntu/data/project/tvm_work/tvm/python/setup.py", line 197, in <module>
    shutil.copy(path, os.path.join(CURRENT_DIR, "tvm"))
  File "/usr/lib/python3.12/shutil.py", line 435, in copy
    copyfile(src, dst, follow_symlinks=follow_symlinks)
  File "/usr/lib/python3.12/shutil.py", line 240, in copyfile
    raise SameFileError("{!r} and {!r} are the same file".format(src, dst))
shutil.SameFileError: '/home/ubuntu/data/project/tvm_work/tvm/python/tvm/libtvm_runtime.so' and '/home/ubuntu/data/project/tvm_work/tvm/python/tvm/libtvm_runtime.so' are the same file

[vacu9708]

@mshr-h
It seems the source and destination might point to the exact same path for some reason.
I added an exception handling for this case.

[mshr-h]

It’s probably due to the search priority of dll_path in the get_dll_directories() function in python/tvm/_ffi/libinfo.py. For some reason, when a shared object already exists under python/tvm/, that shared object is chosen as the source for copying that causes the SameFileError.
It worked by increasing the priority of the build directory.

Changin from this

    # Pip lib directory
    dll_path.append(os.path.join(ffi_dir, ".."))
    # Default cmake build directory
    dll_path.append(os.path.join(source_dir, "build"))
    dll_path.append(os.path.join(source_dir, "build", "Release"))
    # Default make build directory
    dll_path.append(os.path.join(source_dir, "lib"))

to this (prioritize build directory over pip lib directory)

    # Default cmake build directory
    dll_path.append(os.path.join(source_dir, "build"))
    dll_path.append(os.path.join(source_dir, "build", "Release"))
    # Default make build directory
    dll_path.append(os.path.join(source_dir, "lib"))
    # Pip lib directory
    dll_path.append(os.path.join(ffi_dir, ".."))

[vacu9708]

@mshr-h Thank you for your analysis.
Based on your report, I checked the following:

• The problematic python/tvm/libtvm_runtime.so and python/tvm/libtvm.so are appended at lines 146 and 147 in libinfo.py.
  • Appending these paths seems to be unintended behavior, as it is the libraries from build/ that should be used for installation—not the ones from python/tvm/.
  • As a result, lib_path at line 50 in setup.py contains:
    ['python/tvm/libtvm.so', 'build/libtvm.so', 'python/tvm/libtvm_runtime.so', 'build/libtvm_runtime.so']
  • The library selection logic in get_lib_path() favors entries that appear earlier in the list, so the problematic libraries from python/tvm/ are chosen.

In my opinion:

• It seems more appropriate to prevent these unintended libraries from being appended to the list, rather than changing the order as you proposed.
  • This will resolve the shutil.SameFileError exception raised by shutil.copy().
• Adding the cleanup logic earlier—so it runs even if a previous installation failed—also seems valid.
  • This will resolve the FileExistsError exception raised by shutil.copytree().

[mshr-h]

Overall looks good to me. Just minor changes are needed.

[mshr-h]

Thanks a lot! @vacu9708