Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BUG: failed git detection (despite it being on PATH) on windows #2185

Closed
h-vetinari opened this issue Dec 13, 2024 · 14 comments · Fixed by #2192
Closed

BUG: failed git detection (despite it being on PATH) on windows #2185

h-vetinari opened this issue Dec 13, 2024 · 14 comments · Fixed by #2192

Comments

@h-vetinari
Copy link
Member

With 3.45.0, I get

ImportError: Failed to initialize: Bad git executable.
The git executable must be specified in one of the following ways:
    - be included in your $PATH
    - be set via $GIT_PYTHON_GIT_EXECUTABLE
    - explicitly set via git.refresh(<full-path-to-git-executable>)

All git commands will error until this is rectified.

It's definitely on my path though

>where git
C:\Users\[xxx]\.pixi\envs\git\Library\bin\git.exe
C:\Users\[xxx]\.pixi\bin\git.bat
@h-vetinari
Copy link
Member Author

With set "GIT_PYTHON_GIT_EXECUTABLE=C:\Users\[xxx]\.pixi\envs\git\Library\bin\git.exe", things did progress past the initial barrier, but then fail with

Traceback (most recent call last):
  File "E:\miniforge\envs\builder\Lib\site-packages\git\cmd.py", line 1262, in execute
    proc = safer_popen(
           ^^^^^^^^^^^^
  File "E:\miniforge\envs\builder\Lib\site-packages\git\cmd.py", line 281, in _safer_popen_windows
    return Popen(
           ^^^^^^
  File "E:\miniforge\envs\builder\Lib\subprocess.py", line 1026, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "E:\miniforge\envs\builder\Lib\subprocess.py", line 1538, in _execute_child
    hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [WinError 2] The system cannot find the file specified

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "E:\miniforge\envs\builder\Scripts\conda-smithy-script.py", line 10, in <module>
    sys.exit(main())
             ^^^^^^
  File "E:\miniforge\envs\builder\Lib\site-packages\conda_smithy\cli.py", line 767, in main
    args.subcommand_func(args)
  File "E:\miniforge\envs\builder\Lib\site-packages\conda_smithy\cli.py", line 608, in __call__
    self._call(args, tmpdir)
  File "E:\miniforge\envs\builder\Lib\site-packages\conda_smithy\cli.py", line 613, in _call
    configure_feedstock.main(
  File "E:\miniforge\envs\builder\Lib\site-packages\conda_smithy\configure_feedstock.py", line 2856, in main
    set_exe_file(os.path.join(forge_dir, "build-locally.py"))
  File "E:\miniforge\envs\builder\Lib\site-packages\conda_smithy\feedstock_io.py", line 36, in set_exe_file
    repo.git.execute(["git", "update-index", f"--chmod={mode}", filename])
  File "E:\miniforge\envs\builder\Lib\site-packages\git\cmd.py", line 1275, in execute
    raise GitCommandNotFound(redacted_command, err) from err
git.exc.GitCommandNotFound: Cmd('git') not found due to: FileNotFoundError('[WinError 2] The system cannot find the file specified')
  cmdline: git update-index --chmod=+x E:\conda-forge\boost-feedstock\build-locally.py

CC @mgorny

@h-vetinari
Copy link
Member Author

h-vetinari commented Dec 13, 2024

Interestingly, I get this error now even after rolling back to conda-smithy 3.44.9 (conda hung on rollback, so I tried recreating the environment). So it might not be related to #2120 after all.

In any case, my workflow is completely broken now - cannot get thing to work despite --no-check-uptodate and/or various workarounds like setting GIT_PYTHON_GIT_EXECUTABLE.

Here's are the recent environment changes

> conda list --revision
[...]
2024-12-14 09:17:30  (rev 61)
     conda-forge-pinning  {2024.12.11.09.08.10 (https://conda.anaconda.org/conda-forge/noarch) -> 2024.12.13.14.31.10 (conda-forge)}
     conda-libmamba-solver  {24.11.1 (https://conda.anaconda.org/conda-forge/noarch) -> 24.9.0 (conda-forge)}
     conda-smithy  {3.44.9 (https://conda.anaconda.org/conda-forge/noarch) -> 3.45.0 (conda-forge)}
     libarchive  {3.7.7 (https://conda.anaconda.org/conda-forge/win-64) -> 3.7.7 (conda-forge)}
     libcurl  {8.10.1 (https://conda.anaconda.org/conda-forge/win-64) -> 8.11.1 (conda-forge)}
     libmamba  {2.0.4 (https://conda.anaconda.org/conda-forge/win-64) -> 1.5.11 (conda-forge)}
     libmambapy  {2.0.4 (https://conda.anaconda.org/conda-forge/win-64) -> 1.5.11 (conda-forge)}
     matplotlib-base  {3.9.3 (https://conda.anaconda.org/conda-forge/win-64) -> 3.9.4 (conda-forge)}
     nodeenv  {1.9.1 (https://conda.anaconda.org/conda-forge/noarch) -> 1.9.1 (conda-forge)}
     python-utils  {3.9.1 (https://conda.anaconda.org/conda-forge/noarch) -> 3.9.1 (conda-forge)}
     rattler-build  {0.32.0 (https://conda.anaconda.org/conda-forge/win-64) -> 0.32.1 (conda-forge)}
     ruff  {0.8.2 (https://conda.anaconda.org/conda-forge/win-64) -> 0.8.3 (conda-forge)}
     simdjson  {3.10.1 (https://conda.anaconda.org/conda-forge/win-64) -> 3.11.3 (conda-forge)}
     spdlog  {1.14.1 (https://conda.anaconda.org/conda-forge/win-64) -> 1.15.0 (conda-forge)}
     uv  {0.5.7 (https://conda.anaconda.org/conda-forge/win-64) -> 0.5.8 (conda-forge)}
    +libgit2-1.8.4 (conda-forge)
    +pygit2-1.16.0 (conda-forge)

2024-12-14 09:41:21  (rev 62)
     conda-smithy  {3.45.0 (https://conda.anaconda.org/conda-forge/noarch) -> 3.44.9 (conda-forge)}

2024-12-14 09:44:47  (rev 63)
    -libgit2-1.8.4 (https://conda.anaconda.org/conda-forge/win-64)
    -pygit2-1.16.0 (https://conda.anaconda.org/conda-forge/win-64)

No idea why mamba got downgraded, but that's a sideshow.

I also ran pixi global upgrade git (and to debug, downgraded it again to the previous 2.45), and interestingly, there's now one less file

>where git
C:\Users\[xxx]\.pixi\bin\git.bat

Always fun when even the roll-back attempts all fail 😑

@mgorny
Copy link
Contributor

mgorny commented Dec 14, 2024

Hmm, git package is GitPython (pygit2 is pygit2). Perhaps try uninstalling pygit2 and libgit2 and see if that changes anything.

@h-vetinari
Copy link
Member Author

That's what I did in

2024-12-14 09:44:47  (rev 63)
    -libgit2-1.8.4 (https://conda.anaconda.org/conda-forge/win-64)
    -pygit2-1.16.0 (https://conda.anaconda.org/conda-forge/win-64)

But perhaps the removal wasn't clean for some reason...

@h-vetinari
Copy link
Member Author

So conda-smithy has dependend on gitpython since forever; no idea why that would break now.

@mgorny
Copy link
Contributor

mgorny commented Dec 15, 2024

Hmm, but unless I'm mistaken, the newest version should not require GitPython at all. So perhaps try force-uninstalling it after upgrade and see if that changes anything.

@h-vetinari
Copy link
Member Author

If I do that:

  File "E:\miniforge\envs\builder2\lib\site-packages\conda_smithy\github.py", line 5, in <module>
    from git import Repo
ModuleNotFoundError: No module named 'git'

@h-vetinari
Copy link
Member Author

h-vetinari commented Dec 16, 2024

If I do that:

OK, I had done that in my debug environment which still has 3.44.9; if I do it in an up-to-date environment (with 3.45.0 but without gitpython), then I get past the initial issue, but it still fails with:

traceback
Traceback (most recent call last):
  File "E:\miniforge\envs\builder\Scripts\conda-smithy-script.py", line 10, in <module>
    sys.exit(main())
             ^^^^^^
  File "E:\miniforge\envs\builder\Lib\site-packages\conda_smithy\cli.py", line 767, in main
    args.subcommand_func(args)
  File "E:\miniforge\envs\builder\Lib\site-packages\conda_smithy\cli.py", line 608, in __call__
    self._call(args, tmpdir)
  File "E:\miniforge\envs\builder\Lib\site-packages\conda_smithy\cli.py", line 613, in _call
    configure_feedstock.main(
  File "E:\miniforge\envs\builder\Lib\site-packages\conda_smithy\configure_feedstock.py", line 2846, in main
    config = _load_forge_config(forge_dir, exclusive_config_file, forge_yml)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "E:\miniforge\envs\builder\Lib\site-packages\conda_smithy\configure_feedstock.py", line 2497, in _load_forge_config
    config = _legacy_compatibility_checks(config, forge_dir)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "E:\miniforge\envs\builder\Lib\site-packages\conda_smithy\configure_feedstock.py", line 2375, in _legacy_compatibility_checks
    remove_file_or_dir(os.path.join(forge_dir, old_file))
  File "E:\miniforge\envs\builder\Lib\site-packages\conda_smithy\feedstock_io.py", line 79, in remove_file_or_dir
    return remove_file(filename)
           ^^^^^^^^^^^^^^^^^^^^^
  File "E:\miniforge\envs\builder\Lib\site-packages\conda_smithy\feedstock_io.py", line 89, in remove_file
    touch_file(filename)
  File "E:\miniforge\envs\builder\Lib\site-packages\conda_smithy\feedstock_io.py", line 73, in touch_file
    with write_file(filename) as fh:
  File "E:\miniforge\envs\builder\Lib\contextlib.py", line 144, in __exit__
    next(self.gen)
  File "E:\miniforge\envs\builder\Lib\site-packages\conda_smithy\feedstock_io.py", line 68, in write_file
    repo.index.add(os.path.relpath(filename, repo.workdir))
  File "E:\miniforge\envs\builder\Lib\site-packages\pygit2\index.py", line 217, in add
    check_error(err, io=True)
  File "E:\miniforge\envs\builder\Lib\site-packages\pygit2\errors.py", line 66, in check_error
    raise GitError(message)
_pygit2.GitError: invalid path: 'ci_support\upload_or_check_non_existence.py'

@mgorny
Copy link
Contributor

mgorny commented Dec 16, 2024

Now, that looks like a valid bug. I'll take a look at it later today.

@mgorny
Copy link
Contributor

mgorny commented Dec 16, 2024

For a quick test, could you see if the following patch helps with this immediate error:

diff --git a/conda_smithy/feedstock_io.py b/conda_smithy/feedstock_io.py
index 900fbdc0..281d3c4f 100644
--- a/conda_smithy/feedstock_io.py
+++ b/conda_smithy/feedstock_io.py
@@ -65,7 +65,7 @@ def write_file(filename):
 
     repo = get_repo(filename)
     if repo:
-        repo.index.add(os.path.relpath(filename, repo.workdir))
+        repo.index.add(os.path.relpath(filename, repo.workdir).replace(os.path.sep, '/'))
         repo.index.write()
 

My guess is that git expects unix paths. If you confirm, I'll look into preparing a better fix. I'm very sorry about the issue.

@h-vetinari
Copy link
Member Author

I did that (plus two more lines where it was necessary; 41 & 126), and it now fails with:

Traceback (most recent call last):
  File "E:\miniforge\envs\builder\Scripts\conda-smithy-script.py", line 10, in <module>
    sys.exit(main())
             ^^^^^^
  File "E:\miniforge\envs\builder\Lib\site-packages\conda_smithy\cli.py", line 767, in main
    args.subcommand_func(args)
  File "E:\miniforge\envs\builder\Lib\site-packages\conda_smithy\cli.py", line 608, in __call__
    self._call(args, tmpdir)
  File "E:\miniforge\envs\builder\Lib\site-packages\conda_smithy\cli.py", line 613, in _call
    configure_feedstock.main(
  File "E:\miniforge\envs\builder\Lib\site-packages\conda_smithy\configure_feedstock.py", line 2914, in main
    commit_changes(
  File "E:\miniforge\envs\builder\Lib\site-packages\conda_smithy\configure_feedstock.py", line 2563, in commit_changes
    has_staged_changes = subprocess.call(
                         ^^^^^^^^^^^^^^^^
  File "E:\miniforge\envs\builder\Lib\subprocess.py", line 389, in call
    with Popen(*popenargs, **kwargs) as p:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "E:\miniforge\envs\builder\Lib\subprocess.py", line 1026, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "E:\miniforge\envs\builder\Lib\subprocess.py", line 1538, in _execute_child
    hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [WinError 2] The system cannot find the file specified

I'm very sorry about the issue.

All good, it happens. It's just very strange how things interact between the different git libraries...

@mgorny
Copy link
Contributor

mgorny commented Dec 17, 2024

Is git not in PATH for you? It looks as if subprocess.call(["git"]) didn't work for you.

@h-vetinari
Copy link
Member Author

I pointed out in the OP that it's available in the terminal where I use both git and conda smithy. It might be conceivable that some aspect of the pixi-global-provided-git-from-conda-forge doesn't work though. I checked that the folder where git.bat is present is indeed on the PATH, but perhaps what throws the logic is that it's a .bat wrapper. That wrapper does

@echo off
setlocal
@chcp 65001 > nul
@SET "Path=C:\Users\[xxx]\.pixi\envs\git;C:\Users\[xxx]\.pixi\envs\git\Library/mingw-w64/bin;C:\Users\[xxx]\.pixi\envs\git\Library/usr/bin;C:\Users\[xxx]\.pixi\envs\git\Library/bin;C:\Users\[xxx]\.pixi\envs\git\Scripts;C:\Users\[xxx]\.pixi\envs\git\bin;%Path%"
@SET "CONDA_PREFIX=C:\Users\[xxx]\.pixi\envs\git"
@"C:\Users\[xxx]\.pixi\envs\git\Library/bin/git.exe" %*

endlocal

Before I got around to checking whether the path of the actual exe was in my path, I just re-updated my pixi global git install, and now suddenly, the wrapper is gone:

C:\Users\[xxx]\.pixi\bin\git.exe

With this unexplained change, I was now also able to reinstall gitpython and things keep running (there's some spurious files created upon rerender, but that's a separate issue).

So it looks conceivable that something somewhere somehow corrupted my git install (making it invisible from subprocess, but still visible from my own terminal).

CC @wolfv for awareness

@h-vetinari
Copy link
Member Author

Ah, actually the fix above is still necessary

@h-vetinari h-vetinari reopened this Dec 17, 2024
mgorny added a commit to mgorny/conda-smithy that referenced this issue Dec 17, 2024
libgit2 expects internal git paths to be passed in git convention,
that is using forward slashes.  Use pathlib.Path.as_posix() to ensure
that the correct conversion is used when passing paths to index
operations.

Fixes conda-forge#2185
mgorny added a commit to mgorny/conda-smithy that referenced this issue Dec 17, 2024
libgit2 expects internal git paths to be passed in git convention,
that is using forward slashes.  Use pathlib.Path.as_posix() to ensure
that the correct conversion is used when passing paths to index
operations.

Fixes conda-forge#2185
mgorny added a commit to mgorny/conda-smithy that referenced this issue Dec 17, 2024
libgit2 expects internal git paths to be passed in git convention,
that is using forward slashes.  Use pathlib.Path.as_posix() to ensure
that the correct conversion is used when passing paths to index
operations.

Fixes conda-forge#2185
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
2 participants