Rebase failed with error: "unable to unlink old <filename> Invalid argument" (works Ok with older version)

[AlexeyKryshen]

• I was not able to find an open or closed issue matching what I'm seeing

Setup

• Which version of Git for Windows are you using? Is it 32-bit or 64-bit?

Portable version

git version 2.14.0.windows.2
built from commit: 0f3342804fc6fa558a7c9e220e7e9ab4c86e8226
sizeof-long: 4
machine: x86_64

• Which version of Windows are you running? Vista, 7, 8, 10? Is it 32-bit or 64-bit?

Microsoft Windows [Version 6.1.7601]

• What options did you set as part of the installation? Or did you choose the
  defaults?

Default Portable version installation

• Any other interesting things about your environment that might be related
  to the issue you're seeing?

LFS installed and used on repository caused the issue

git-lfs/2.2.1 (GitHub; windows amd64; go 1.8.3; git 621d1f82)

Details

• Which terminal/shell are you running Git from? e.g Bash/CMD/PowerShell/other

Bash

• What commands did you run to trigger this issue? If you can provide a
  Minimal, Complete, and Verifiable example
  this will help us understand the issue.

git rebase master

master is just 1 commit ahead of branching point and branch I want to rebase contains 2 commits (branch based on master). Branch commits contains only new files under LFS (no any single modified from master). Branch commits age huge (~1 Gb/ ~3000 files each). First branch commit contains only newly added files and second one delete ~50 of them and modify each remaining ( ~50 deleted & ~2950 modified). Again, all files in branch commits are under LFS.

• What did you expect to occur after running these commands?

branch successfully rebased on newest master commit

• What actually happened instead?

Rebased failed with the following message:

$ git rebase master
First, rewinding head to replay your work on top of it...
Applying: Branch commit 1
Applying: Branch commit 2
error: unable to unlink old '<Full path name>': Invalid argument
Using index info to reconstruct a base tree...
Falling back to patching base and 3-way merge...
error: Failed to merge in the changes.
Patch failed at 0002 Branch commit 2
The copy of the patch that failed is found in: .git/rebase-apply/patch

When you have resolved this problem, run "git rebase --continue".
If you prefer to skip this patch, run "git rebase --skip" instead.
To check out the original branch and stop rebasing, run "git rebase --abort".

• If the problem was occurring with a specific repository, can you provide the
  URL to that repository to help us with testing?

Unfortunately it is happened with local repository.

• Additional information

The scenario (rebasing branches with huge LFS commits) is common for the projects I am currently working with and for about an year it is the first time the issue occur (we start using LFS about an year ago). The issue occur when I was trying newest version of Portable Git for windows (2.14.0.2). Version 2.10.2 (the version we working with currently) works as expected and doesn't raise the issue with this rebase. Version 2.13.3 also cause the same rebase issue. Other versions are not tested by me.

[larsxschneider]

Thanks for the report @AlexeyKryshen !

In Git 2.11 we changed the default protocol between Git and Git LFS (see blog post, Git Merge 2017 talk, change in Git).

During the implementation I hit the same issue that you are seeing now and I fixed it for my use cases. Looks like there are other (totally valid!) use cases that my fix did not cover. That needs to be fixed!

---

Quick Solution

Edit your .gitconfig and remove process = git-lfs filter-process. Please note that this setting can be in your local, global, and system Git config. Run git config -l --show-origin | grep process to ensure it is disabled everywhere. If you remove that line then your Git LFS should use the old protocol, again.

Proper Solution

I'll try to reproduce your problem locally. If I succeed then I should be able to fix it. If not then we need to find another way to debug this. However, all this might take a while (weeks) as I have lots of other things on my todo list 🙂

/CC @ttaylorr

[AlexeyKryshen]

@larsxschneider, thank you for your reply and especially for the provided Quick Fix.

I made and store a full copy of the project in problematic state so if you will be in need, I could do some remote diagnosis/debug for you by your requests/instructions.

Many thanks to you and whole Git for Windows team for the great product! (especially for Portable version) :)

[naatje80]

We are currently facing the same issue in combination with Linux shares that are hosted on Isilon storage. However, in our case the workaround by removing the "process = git-lfs filter-process" does not temporary fix the issue. The most recent version that works is Git for Windows 2.12.2(2).

[larsxschneider]

@naatje80 Can you explain your Linux share setup further? I assume the following:

• you have a Git repo hosted on disk (file system ext3, ext4,...?) on a Linux machine
• the disk with the repo is made available on the network (via samba?)
• you are accessing this samba share with Windows version ?

Can you run GIT_TRACE=1 GIT_CURL_VERBOSE=1 git $YOUR_COMMAND_THAT_FAILS and post the output here? (this probably contains lots of personal information, make sure to redact that; especially the line Authorization: Basic ... if it exists)

In general Git repos on samba shares are not recommended and error prone AFAIK.

[naatje80]

Dear larsxschneider,

I'm very sorry for my delayed response. I got the following answers from our System Management department.

To answer your questions:

• you have a Git repo hosted on disk (file system ext3, ext4,...?) on a Linux machine
  OneFs 7.2.1.1 on EMC Isilon Cluster
• the disk with the repo is made available on the network (via samba?)
  EMC has their own protocol to serve CIFS.
• you are accessing this samba share with Windows version ?
  Windows 10 Enterprise (64-bit)

The output of the command with verbose options is:

X:\git>git clone https://github.com/PowerShell/Win32-OpenSSH.git
10:46:42.634257 git.c:328               trace: built-in: git 'clone' 'https://github.com/PowerShell/Win32-OpenSSH.git'
Cloning into 'Win32-OpenSSH'...
10:46:43.853014 run-command.c:626       trace: run_command: 'remote-https' 'origin' 'https://github.com/PowerShell/Win32-OpenSSH.git'
10:46:43.915552 git.c:560               trace: exec: 'git-remote-https' 'origin' 'https://github.com/PowerShell/Win32-OpenSSH.git'
10:46:43.915552 run-command.c:626       trace: run_command: 'git-remote-https' 'origin' 'https://github.com/PowerShell/Win32-OpenSSH.git'
* Couldn't find host github.com in the _netrc file; using defaults
* timeout on name lookup is not supported
*   Trying 192.30.253.113...
* TCP_NODELAY set
* Connected to github.com (192.30.253.113) port 443 (#0)
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: C:/Program Files/Git/mingw64/ssl/certs/ca-bundle.crt
  CApath: none
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: businessCategory=Private Organization; jurisdictionC=US; jurisdictionST=Delaware; serialNumber=5157550; street=88 Colin P Kelly, Jr Street; postalCode=94107; C=US; ST=California; L=San Francisco; O=GitHub, Inc.; CN=github.com
*  start date: Mar 10 00:00:00 2016 GMT
*  expire date: May 17 12:00:00 2018 GMT
*  subjectAltName: host "github.com" matched cert's "github.com"
*  issuer: C=US; O=DigiCert Inc; OU=www.digicert.com; CN=DigiCert SHA2 Extended Validation Server CA
*  SSL certificate verify ok.
> GET /PowerShell/Win32-OpenSSH.git/info/refs?service=git-upload-pack HTTP/1.1
Host: github.com
User-Agent: git/2.14.1.windows.1
Accept: */*
Accept-Encoding: gzip
Pragma: no-cache

< HTTP/1.1 200 OK
< Server: GitHub Babel 2.0
< Content-Type: application/x-git-upload-pack-advertisement
< Transfer-Encoding: chunked
< Expires: Fri, 01 Jan 1980 00:00:00 GMT
< Pragma: no-cache
< Cache-Control: no-cache, max-age=0, must-revalidate
< Vary: Accept-Encoding
< X-GitHub-Request-Id: BC46:5FA0:5A38BC:9C2B31:59F99804
< X-Frame-Options: DENY
<
* Connection #0 to host github.com left intact
10:46:44.759246 run-command.c:626       trace: run_command: 'fetch-pack' '--stateless-rpc' '--stdin' '--lock-pack' '--thin' '--check-self-contained-and-connected' '--cloning' 'https://github.com/PowerShell/Win32-OpenSSH.git/'
10:46:44.837370 git.c:328               trace: built-in: git 'fetch-pack' '--stateless-rpc' '--stdin' '--lock-pack' '--thin' '--check-self-contained-and-connected' '--cloning' 'https://github.com/PowerShell/Win32-OpenSSH.git/'
* Couldn't find host github.com in the _netrc file; using defaults
* Found bundle for host github.com: 0x10a76b0 [can pipeline]
* Re-using existing connection! (#0) with host github.com
* Connected to github.com (192.30.253.113) port 443 (#0)
> POST /PowerShell/Win32-OpenSSH.git/git-upload-pack HTTP/1.1
Host: github.com
User-Agent: git/2.14.1.windows.1
Accept-Encoding: gzip
Content-Type: application/x-git-upload-pack-request
Accept: application/x-git-upload-pack-result
Content-Encoding: gzip
Content-Length: 1002

* upload completely sent off: 1002 out of 1002 bytes
< HTTP/1.1 200 OK
< Server: GitHub Babel 2.0
< Content-Type: application/x-git-upload-pack-result
< Transfer-Encoding: chunked
< Expires: Fri, 01 Jan 1980 00:00:00 GMT
< Pragma: no-cache
< Cache-Control: no-cache, max-age=0, must-revalidate
< Vary: Accept-Encoding
< X-GitHub-Request-Id: BC46:5FA0:5A3934:9C2B56:59F99804
< X-Frame-Options: DENY
<
remote: Counting objects: 7922, done.
remote: Compressing objects: 100% (13/13), done.
10:46:45.306136 run-command.c:626       trace: run_command: 'index-pack' '--stdin' '-v' '--fix-thin' '--keep=fetch-pack 7572 on NLVEHVRES2NBNS1' '--check-self-contained-and-connected' '--pack_header=2,7922'
10:46:45.368619 git.c:328               trace: built-in: git 'index-pack' '--stdin' '-v' '--fix-thin' '--keep=fetch-pack 7572 on NLVEHVRES2NBNS1' '--check-self-contained-and-connected' '--pack_header=2,7922'
* Connection #0 to host github.com left intactB | 12.35 MiB/s
remote: Total 7922 (delta 3), reused 11 (delta 3), pack-reused 7906
Receiving objects: 100% (7922/7922), 134.75 MiB | 11.10 MiB/s, done.
Resolving deltas: 100% (5491/5491), done.
10:47:09.384103 run-command.c:626       trace: run_command: 'rev-list' '--objects' '--stdin' '--not' '--all' '--quiet' '--progress=Checking connectivity'
10:47:09.462276 git.c:328               trace: built-in: git 'rev-list' '--objects' '--stdin' '--not' '--all' '--quiet' '--progress=Checking connectivity'
error: unable to append to 'X:/git/Win32-OpenSSH/.git/logs/refs/remotes/origin/HEAD': Invalid argument
fatal: update_ref failed for ref 'HEAD': cannot update the ref 'HEAD': unable to append to 'X:/git/Win32-OpenSSH/.git/logs/HEAD': Invalid argument
cb0b7325a6ebe53f1ebcd19c67e30fc9a7683109.idx' failed. Should I try again? (y/n) n
Unlink of file 'Win32-OpenSSH/.git/objects/pack/pack-cb0b7325a6ebe53f1ebcd19c67e30fc9a7683109.idx' failed. Should I try again? (y/n) n

Some further information: I also setup a samba share on a Centos 7.4 workstation. The disk there is using ext4. On this machine it still seems to be working with git/2.14.1.windows.1.

[dscho]

git clone https://github.com/PowerShell/Win32-OpenSSH.git

This is very confusing, as the original report is about a rebase, not a clone. It would probably make a lot more sense to open a new ticket for that quite likely very different issue (even if the symptom is very similar).

[naatje80]

Ok, I thought the issues were related. But I will create a new ticket for this. Thank you for the reply.

[dscho]

@AlexeyKryshen @larsxschneider what is the state of this ticket?