git-maintenance doesn't work on WSL

[nedsociety]

• I was not able to find an open
  or closed issue matching
  what I'm seeing, including in the git-for-windows/git tracker.

Setup

• Which version of microsoft/git are you using? Is it 32-bit or 64-bit?

$ git --version --build-options
git version 2.39.2.vfs.0.0
cpu: x86_64
built from commit: e7fd49f4d278a3c36be6c1ef7e720e893fd5c747
sizeof-long: 8
sizeof-size_t: 8
shell-path: /bin/sh

• Are you using Scalar or VFS for Git?

I'm using scalar but I'd believe that the vanilla git-maintenance also has the problem.

• Which version of Windows are you running? Vista, 7, 8, 10? Is it 32-bit or 64-bit?

WSL, ubuntu, 22.04

• Any other interesting things about your environment that might be related
  to the issue you're seeing?

Systemd is enabled. The .gitconfig in WSL refers to the host's git credential provider.

Details

• What commands did you run to trigger this issue? If you can provide a
  Minimal, Complete, and Verifiable example
  this will help us understand the issue.

1. enable systemd in WSL
2. add this line to .gitconfig (path may differ)

[credential]
        helper = "/mnt/c/Program\\ Files/Git/mingw64/bin/git-credential-manager.exe"

scalar register <your repo>

• What did you expect to occur after running these commands?

the hourly systemd job is executed without an error.

• What actually happened instead?

it fails since WSL can't run the host credential manager.

• If the problem was occurring with a specific repository, can you specify
  the repository?

  • Other internal repo.

---

X-posting of microsoft/WSL#10459 . I'd believe that a fix from WSL should be a better solution, but git may also generate a better service unit file to workaround the issue.

git/builtin/gc.c

Line 2443 in 0e35ce9

"RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6\n"

Apparently this list is not enough for WSL to run the host-side cred provider.

[dscho]

it fails since WSL can't run the host credential manager.

Is this only the case when Git maintenance runs, or do you also experience this when running git fetch manually in WSL?

[nedsociety]

Is this only the case when Git maintenance runs, or do you also experience this when running git fetch manually in WSL?

Only when git maintenance runs through systemd. Both git fetch and git for-each-repo --config=maintenance.repo maintenance run runs fine on login shell, though it doesn't really help if it's not scheduled.

[dscho]

@mjcheetham this seems to be a Git Credential Manager problem more than a microsoft/git problem. Would you have any insights?

[nedsociety]

FYI, changing ExecStart to /mnt/c/Windows/system32/cmd.exe /c "echo test" also shows the same error. I doubt GCM is directly responsible for this issue.

[dscho]

changing ExecStart to /mnt/c/Windows/system32/cmd.exe /c "echo test" also shows the same error.

Oh, you see an error message somewhere? And where is that ExecStart coming from that you mentioned?

[dscho]

And how did you enable systemd? Could you please provide a step-by-step minimal reproducer? Like, using Git Credential Manager or CMD directly in a file in .config/ somewhere I guess?

[dscho]

git/builtin/gc.c

Line 2443 in 0e35ce9

"RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6\n"

Apparently this list is not enough for WSL to run the host-side cred provider.

Are you saying that it works if you remove that RestrictAddressFamilies line?

[nedsociety]

Reproducer

1. Install WSL from Microsoft Store. Then install Ubuntu 22.04.
2. Enable systemd as guided in here.
3. Install git on the host. I installed it with the official git-scm installer release with the following option:
   
4. From host, clone a private github repo with HTTPS. It will ask for github authentication info for the first time. Do it.
5. Install microsoft-git inside WSL.
6. In WSL create ~/.gitconfig with the following content: (the path may change if the host git installation path is different)

[credential]
        helper = "/mnt/c/Program\\ Files/Git/mingw64/bin/git-credential-manager.exe"

7. Now use scalar to clone the same repo as in the step 4. It won't ask the info twice since it now uses the credential helper from the host's git installation. This is an expected behavior.
8. During the scalar clone it would have installed a systemd timer called git-maintenance in ~/.config/systemd/user. This behavior is also expected.
9. Now trigger it with systemctl --user start git-maintenance@hourly.service. It fails:

ned@NEDHOME ~ $ systemctl --user start git-maintenance@hourly.service
Job for git-maintenance@hourly.service failed because the control process exited with error code.
See "systemctl --user status git-maintenance@hourly.service" and "journalctl --user -xeu git-maintenance@hourly.service" for details.
ned@NEDHOME ~ [1] $ systemctl --user status git-maintenance@hourly.service
× git-maintenance@hourly.service - Optimize Git repositories data
     Loaded: loaded (/home/ned/.config/systemd/user/git-maintenance@.service; static)
     Active: failed (Result: exit-code) since Fri 2023-09-08 22:01:08 KST; 12s ago
TriggeredBy: ● git-maintenance@hourly.timer
    Process: 747380 ExecStart=/usr/local/lib/git-core/git --exec-path=/usr/local/lib/git-core for-each-repo --config=ma>
   Main PID: 747380 (code=exited, status=1/FAILURE)

Sep 08 22:01:07 NEDHOME systemd[437]: Starting Optimize Git repositories data...
Sep 08 22:01:08 NEDHOME git[747387]: WSL (747387) ERROR: UtilBindVsockAnyPort:285: socket failed 97
Sep 08 22:01:08 NEDHOME git[747381]: fatal: could not read Username for 'https://github.com': No such device or address
Sep 08 22:01:08 NEDHOME git[747381]: error: failed to prefetch remotes
Sep 08 22:01:08 NEDHOME git[747381]: error: task 'prefetch' failed
Sep 08 22:01:08 NEDHOME systemd[437]: git-maintenance@hourly.service: Main process exited, code=exited, status=1/FAILURE
Sep 08 22:01:08 NEDHOME systemd[437]: git-maintenance@hourly.service: Failed with result 'exit-code'.
Sep 08 22:01:08 NEDHOME systemd[437]: Failed to start Optimize Git repositories data.

Fiddling with the configuration

The file ~/.config/systemd/user/git-maintenance@.service has a systemd timer setup for git-maintenance.

# This file was created and is maintained by Git.
# Any edits made in this file might be replaced in the future
# by a Git command.

[Unit]
Description=Optimize Git repositories data

[Service]
Type=oneshot
ExecStart="/usr/local/lib/git-core/git" --exec-path="/usr/local/lib/git-core" for-each-repo --config=maintenance.repo maintenance run --schedule=%i
LockPersonality=yes
MemoryDenyWriteExecute=yes
NoNewPrivileges=yes
RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6
RestrictNamespaces=yes
RestrictRealtime=yes
RestrictSUIDSGID=yes
SystemCallArchitectures=native
SystemCallFilter=@system-service

• Calling the ExecStart line manually (replacing %i to hourly) in the login shell does the maintenance job successfully.
• Changing ExecStart to something like /mnt/c/Windows/system32/cmd.exe /c "echo test" fails with UtilBindVsockAnyPort as noted above, which suggests that running a host binary fails when systemd runs them with these setups. That explains why git failed to run the credential helper which resides in the host side.
• As the error mentioned UtilBindVsockAnyPort, I commented out RestrictAddressFamilies to see if it's a socket problem. It made the maintenance job successful.

[dscho]

• I commented out RestrictAddressFamilies to see if it's a socket problem. It made the maintenance job successful.

Excellent finding, and valuable, important information in the context of this ticket.

Could you see whether it works if you uncomment that line and add AF_NETLINK?

[nedsociety]

AF_NETLINK

It didn't work :/ it shows the same socket error.

[dscho]

Can you see anything in dmesg or syslog that might give a hint what was blocked?

[nedsociety]

• Nothing seems relevant in dmesg.
• I don't have syslogd so no syslog available.
• journalctl shows no more clue than what's already shown in systemctl --user status.

[dscho]

@nedsociety you seem to know tons more about systemd than I do, maybe you can find a list of all available AF_* values for that line, and then see whether specifying them all "makes it work"?

[nedsociety]

okay, I've got the list from here and it seems working when I put everything:

RestrictAddressFamilies=AF_UNIX AF_LOCAL AF_INET AF_AX25 AF_IPX AF_APPLETALK AF_X25 AF_INET6 AF_DECnet AF_KEY AF_NETLINK AF_PACKET AF_RDS AF_PPPOX AF_LLC AF_IB AF_MPLS AF_CAN AF_TIPC AF_BLUETOOTH AF_ALG AF_VSOCK AF_KCM AF_XDP

well but binary searching around these list seems too much hassle 😭

[dscho]

AF_VSOCK

This sounds awfully close to UtilBindVsockAnyPort, no?

[nedsociety]

AF_VSOCK

This sounds awfully close to UtilBindVsockAnyPort, no?

Huh, good eyes. It actually worked.

[OneBlue]

AF_VSOCK

This sounds awfully close to UtilBindVsockAnyPort, no?

Huh, good eyes. It actually worked.

Indeed as a WSL maintainer I confirm that AF_VSOCK is what we use.

[nedsociety]

With the issue in WSL repo was concluded as wontfix, I guess adding AF_VSOCK to the list seems to be the only choice left?

[dscho]

Indeed as a WSL maintainer I confirm that AF_VSOCK is what we use.

@OneBlue thanks for confirming!

adding AF_VSOCK to the list seems to be the only choice left?

@nedsociety yes, and it is also the correct thing to do. See gitgitgadget#1586 and #605 for the progress of the fix.