Fixes for Git for Windows's Git Bash

[ingydotnet]

From #116 Christophe Porteneuve asks:

Apparently it doesn't work yet in Git for Windows's Git Bash. It's been updated
recently to a much more recent Bash and Git. Is that scheduled soon?

[ingydotnet]

I can look into this one today. I have Git for Windows's Git Bash set up locally. I think it should be easy.

[ingydotnet]

Just ran the tests. Good news, all code compiles and tests actually run.

prove -v test/compile.t

works. Bad news: lots of test failures.

Hopefully this is all caused by something simple. Will dig deeper later.

[ingydotnet]

Ran into this: https://gist.github.com/ingydotnet/cbcac5bb4c1886fb9606

Can anyone confirm or deny if this is a bug in Bash?

[perlpunk]

I'm not a bash expert, but if I understand this http://www.tldp.org/LDP/abs/html/comparison-ops.html correctly, it's a feature

The == comparison operator behaves differently within
  a double-brackets test than within single brackets.
[[ $a == z* ]]   # True if $a starts with an "z" (pattern matching).
[[ $a == "z*" ]] # True if $a is equal to z* (literal matching).

[ $a == z* ]     # File globbing and word splitting take place.
[ "$a" == "z*" ] # True if $a is equal to z* (literal matching).

[ingydotnet]

@perlpunk Your example has literal matching chars. I guess variable expansion happens and leads to this (buggy imho) behavior.

From the manpage:

[[ expression ]]
Return a status of 0 or 1 depending on the evaluation of the conditional expression expression. Expressions are composed of the primaries described below under CONDITIONAL EXPRESSIONS. Word splitting and pathname expansion are not performed on the words between the [[ and ]]; tilde expansion, parameter and variable expansion, arithmetic expansion, command substitution, process substitution, and quote removal are performed.

[ingydotnet]

I thought $variable expansion inside double brackets was safe (ie quotes not needed). I'll have to go back and add quotes.

[perlpunk]

@ingydotnet ok, that might explain why in zsh I get no output for all six commands.

[perlpunk]

just tested, behavior still the same in bash 4.4-beta

[ingydotnet]

See alo #38

[grimmySwe]

I am testing this (git version 2.7.0.windows.1) at home on my windows machine and this appeared:

$ bash --version
GNU bash, version 4.3.42(5)-release (x86_64-pc-msys)
...
$ git subrepo --version
D:\work\git\git-subrepo\lib/git-subrepo.d/bash+.bash: line 1: ../../ext/bashplus/lib/bash+.bash: No such file or directory

I tested and updated D:\work\git\git-subrepo\lib/git-subrepo.d/bash+.bash from ../../ext/bashplus/lib/bash+.bash to $GIT_SUBREPO_ROOT/ext/bashplus/lib/bash+.bash

And then I got:

$ git subrepo --version
D:\work\git\git-subrepo\lib\git-subrepo: line 20: bash+:import: command not found
fatal: 'subrepo' appears to be a git command, but we were not
able to execute it. Maybe git-subrepo is broken?

Will look into this further later tonight, but if anyone has any clues they are welcome :-)

[grimmySwe]

Lets have another go:
I found that there still was something missing in setup, but changing gitsubrepo with this seems to trigger something.

-source "${SOURCE_DIR}/git-subrepo.d/bash+.bash"
+source "${SOURCE_DIR}/../ext/bashplus/lib/bash+.bash"

Then the problem that you discussed above kicked in:

-[[ $BASH_SOURCE != $0 ]] || main "$@"
+[[ "$BASH_SOURCE" != "$0" ]] || main "$@"

But after that it gets somewhere:

$ git subrepo status
2 subrepos:

Git subrepo 'ext/bashplus':
  Remote URL:      git@github.com:ingydotnet/bashplus.git
  Tracking Branch: master
  Pulled Commit:   aba653b
  Pull Parent:     cde02a6

Git subrepo 'ext/test-more-bash':
  Remote URL:      git@github.com:ingydotnet/test-more-bash.git
  Tracking Branch: master
  Pulled Commit:   05a1bdd
  Pull Parent:     1a45c4c

Running the tests will still give me a lot of errors:
https://gist.github.com/grimmySwe/1e5ca8ad3cdcc4402cca

I'll start with checking the:

warning: LF will be replaced by CRLF in bar/file.
The file will have its original line endings in your working directory.

It seems that as I am running git bash and emacs so I can use:

$ git config --global core.autocrlf input

[grimmySwe]

This problem comes back as the test sources the lib/git-subrepo.d/bash+.bash

$ prove test/compile.t
test/compile.t .. 1/? lib/git-subrepo.d/bash+.bash: line 1: ../../ext/bashplus/lib/bash+.bash: No such file or directory

Don't know the meaning of the extra source step so I leave this for now.

[grimmySwe]

It seems that git converts paths as well:

$ git subrepo status  /home/user/bar/foo
'C:/Program Files/Git/home/user/bar/foo' is not a subrepo

$ ls /doesnot/exist
ls: cannot access /doesnot/exist: No such file or directory

Investigating why and if this can be handled someway

[ingydotnet]

@grimmySwe, The problems that you are having are a result of Windows symlinks.
I haven't come up with a good solution yet. One solution is to have subrepo
copy the symlinked files when it is running under windows.

We could put this logic into .rc to check if we are on windows and require
the fix.

Thanks for digging in more. I'd put this on hold...

[robe070]

I have the tests passing

{ git-subrepo } master » prove test/compile.t
test/compile.t .. ok
All tests successful.
Files=1, Tests=2, 13 wallclock secs ( 0.05 usr 0.14 sys + 3.81 cusr 8.25 csys = 12.25 CPU)
Result: PASS
{ git-subrepo } master »

I have installed babun and using zsh (There is an option to use bash but that couldn't cope with the .rc)
Then I just applied most of what @grimmySwe stated above (thanks for the help), though slightly differently.

First issue was line endings. As the repo must be used with LF endings, and Windows needs a different setting in general terms, there should be a .gitattributes file in the repo with
* text eol=lf
So I committed that change, deleted all other files dirs in working directory and reset --hard.

In .\git-subrepo\lib\git-subrepo I did
-source "${SOURCE_DIR}/git-subrepo.d/bash+.bash"
+source "${SOURCE_DIR}/../ext/bashplus/lib/bash+.bash"

I don't get the quoting issue.

And its a pretty shell :)

Now to actually use it!

[robe070]

I tried a different shell - supplied with SourceTree - mingw32 bash shell

MINGW32_NT-6.1-WOW 2.3.0(0.290/5/3) 2015-09-29 10:49 i686 Msys
git version 2.6.1.windows.1

babun is:

CYGWIN_NT-6.1-WOW 1.7.35(0.287/5/3) 2015-03-04 12:07 i686 Cygwin
git version 2.1.4

It gets no errors now, but 'does nothing'

with debug on its failing at this test just before calling main:
+ [[ C:\dev\git-subrepo\lib\git-subrepo != C:\dev\git-subrepo\lib\git-subrepo ]]

whereas in babun this is the test:

+ [[ /c/dev/git-subrepo/lib/git-subrepo != /c/dev/git-subrepo/lib/git-subrepo ]]
+ main --version

I then changed babun to use the same git as mingw32, and now it behaves the same - it does nothing. Of course all other git commands work fine.

[grimmySwe]

I tested a little more and came across a strange error:
git: 'filter-branch' is not a git command. See 'git --help'.
It occurs in the test/init.t tests (if I add some extra output, otherwise it just dies and says

Failed to get subrepo content from doc

But if I run the commands manually it works... Also tested to put in git --versionand which gitand it seems that its the same git that is running. git filter-branch works if I add it to the top level in git-subrepo but if I add it in the main function it can't be found when running the tests.

It seems that if I do this in test/setup it works:
#export GIT_EXEC_PATH="$PWD/lib:$(git --exec-path)"

[grimmySwe]

So I got almost all tests working in gitbash. Created issue/119 branch for this. There are some code commented out that I haven't been able to test it on linux/mac so please let me know if you find problems.

[quiret]

I have been looking for a submodule alternative and I admit this is very promising! Hopefully your work will find its way into the main git program.

The master branch did not work for me (with that weird bash+ issue), but the issue/119 does. Will try out your script now 👍

Thank you for caring about the Windows world, btw.

EDIT: wow... I used some "git subrepo pull --all" command and it destroyed my codebase... :/

[ingydotnet]

@robe070 @grimmySwe have you guys pushed your work somewhere so I can test it
out? I couldn't find anything, while poking around GitHub.

OK I just found https://github.com/ingydotnet/git-subrepo/commits/issue/119
with 2 commits from @grimmySwe.

I'll try installing these shells tonight and see how far I can get.

@quiret can you explain what 'destroyed my codebase' means, specifically?

[ingydotnet]

@robe070, I think babun is just simpler packaging/tooling around cygwin.

AFAIK, subrepo has always worked on cygwin. (I'm glad it still does)

re "I tried a different shell - supplied with SourceTree - mingw32 bash shell"

Can you provide a URL (or other instructions) of where you got this environment
from, so the rest of us can try it?

I'm working on testing this on all the windows permutations I can find.

[robe070]

@ingydotnet & @grimmySwe , I was just about to reply about my tests on Windows in MINGW32. It seems to be working, when it wasn't before I applied the changes from @grimmySwe. But I can't run the subrepo tests as the environment does not ship make.

SourceTree is from Atlassian. You can download it from here: https://www.sourcetreeapp.com/ Click on Terminal on the toolbar. It would be great if you could confirm it passes the tests, and inform me how you got make installed - it crashed when I installed MSYS.

I ran the tests in the babun zsh shell and some of them failed. The results are here: https://gist.github.com/robe070/7d2314e7e88d3b8d7f85

Babun uses an older Git (git version 2.1.4) so that may be the cause. Now that MINGW32 seems to be working I'll use that as its using git version 2.6.1.windows.1.

[robe070]

Update on running tests in SourceTree MINGW32
MINGW32_NT-6.1-WOW 2.3.0(0.290/5/3) 2015-09-29 10:49 i686 Msys

I bypassed make and just ran prove test/

Most tests passed. The same failures as with babun The results are here: https://gist.github.com/robe070/7d2314e7e88d3b8d7f85

[snaddenm]

I've also been looking for a solution to the problems with git submodules and subtrees for some time and git subrepo sounds like a great solution to me. But we use a Windows environment so I can't test it out yet. I don't know enough about bash to help solve this problem, but I just wanted to encourage you to keep chipping away at this issue for the benefit of the rest of us.

[snaddenm]

Actually the code in issue/119 does work for me. I removed Git for Windows and installed the new version of Git for Windows, which was 2.7.1.2 at the time. The new version uses MINGW64 rather than MINGW32. The bash version included with it is:
GNU bash, version 4.3.42(5)-release (x86_64-pc-msys)

and the git version included with it is:
git version 2.7.1.windows.2

I also use SourceTree and Git Extensions. I was able to tell both of them to use the updated bash and git that came with Git for Windows instead of using the outdated bash that is bundled with SourceTree. Then git-subrepo seems to work perfectly as long as I am using the issue/119 branch.

If I use the master branch of git-subrepo then I get this error message:
git-subrepo\lib/git-subrepo.d/bash+.bash: line 1: ../../ext/bashplus/lib/bash+.bash: No such file or directory

Is it safe for me to start using this branch of git-subrepo or should I wait till you've finished making all the tests pass?

[ingydotnet]

@grimmySwe I took your issue/119 branch, rebased with master, add a few commits of my own and pushed it back. Can you test on Git for Windows. If tests pass or mostly pass please gist this command output:

$ uname
$ git config -l
$ env

and comment the url for the gist. I can't get the tests to pass at all yet, but I think my env might be messed up, or not set up like yours.

[ingydotnet]

I am declaring that subrepo works on Windows!

I've updated the docs with some windows specific info.

All tests pass on Git for Windows with:

git config --global core.autocrlf input

Closing this. Reopen if needed.

[ingydotnet]

All tests passing on Babun too. Can someone test out Cygwin. I don't have the
disk space to install on my vm.

[grimmySwe]

I doubled checked and all tests pass for me on in my git bash.

[ingydotnet]

I'm thinking this is release worthy. Looking over other issues to close...

[robe070]

I'm loathe to re-open but I think requiring that a config setting is used which is not recommended for Windows is not optimal.

That is,
git config --global core.autocrlf input
It is standard practice to use
git config --global core.autocrlf true
especially for cross-platform repos.

I've not included the recent changes yet, but issue/119 changes seem to work for me.

I use git config --global core.autocrlf true and have a .gitattributes file with
* text eol=lf

This is not the same as autocrlf=input and there is no equivalent that I can find for .gitattributes.

A few tests fail, I believe due to CRLF being used by the test and then it being forced to LF by git

Neither does it seem optimal to require a repo config setting: git config core.autocrlf input. That could be added to the Windows install instructions, but still its possible for it to work 'out-of-the-box', so why not do that?

LF line endings are definitely required.

Why not include a .gitattributes file? I neither know git very well nor do I have a Linux environment to test on. If someone created a .gitattributes like I've suggested and then added exceptions for the readme files, and whatever else ON LINUX, and got that working, it would work on Windows too. Tests included.

Anyone put their hand up?

[ingydotnet]

@robe070 I'll try .gitattributes out. Thanks.

[robe070]

thats great! (I hope encouragement is not discouraged 👍 )

I'm giving tag github/issue/119 a test right now

[ingydotnet]

I'm getting a lot of these warnings:

The file will have its original line endings in your working directory.

Only 3 tests failed and I got a fix for one of them.

I wonder if it's possible to have the core.autocrlf input on only during
test.

[robe070]

Those messages are probably the same as the ones I'm getting.

I believe it is possible to have core.autocrlf input on only during testing. I was testing that theory out when I saw your comment above :)

I'm using prove -v test/issue29.t to reproduce the issue I had the problem with.

[robe070]

Test failed.

git config autocrlf input and not using .gitattributes before running test does not pass prove -v test/issue29.t

[ingydotnet]

All tests pass for git config --global core.autocrlf true.

People should be able to use this (and run tests) on Windows now, without any
environment changes.

Still getting warnings on issue95.t and issue96.t. Will fix that at some point.

@robe070, please test and confirm. Also still need someone to run tests on
cygwin.

[robe070]

When I had git config --global core.autocrlf true, initially, I couldn't run subrepo commands. I'll re-check that with your latest changes soon.

With old changes...
test/issue29.t passes when I do this:

git config --global core.autocrlf true
<add .gitattributes * text eol=lf, and commit>
rm .git/index
git reset --hard HEAD
git config --global core.autocrlf input
prove -v test/issue29.t

uname: MINGW32_NT-6.1-WOW VAULT2GIT 2.3.0(0.290/5/3) 2015-09-29 10:49 i686 Msys
git version 2.6.1.windows.1

[robe070]

Tested 9f2ab87 - master as at 2/3/2016 12:16:25 PM AEST

git config --global core.autocrlf true
rm .git/index
git reset --hard HEAD
<started new bash shell>
prove -v test

All tests pass on both 32 bit and 64 bit Git for Windows (MingW)

uname: MINGW64_NT-6.3 RobertPC 2.4.0(0.292/5/3) 2016-01-07 19:39 x86_64 Msys
git version 2.7.2.windows.1
bash: 4.3.42(5)-release

uname: MINGW32_NT-6.1-WOW VAULT2GIT 2.3.0(0.290/5/3) 2015-09-29 10:49 i686 Msys
git version 2.6.1.windows.1
bash: 4.3.42(2)-release

Thanks @ingydotnet !

[ingydotnet]

@robe070 you are welcome!

I'm sure there are bugs lurking, but if all the tests pass on a stock install, then what more can be done? :)

The approach I took was use .gitattributes file you suggested in the main repo. and also in one of the tests that creates a repo. Then I set git config core.autocrlf input (not --global) in the test/setup file, so the test suite is always running under that. Then I had to do the same thing for the tests that clone a fake repo.

So basically making the tests run in a posix-ish environment. I suppose that leaves a lot of room for things to be weird in Windows dev and in cross dev.

We shall see... we shall fail... we shall fix...

[robe070]

Line ending differences are a fact of life in Windows development. We have a cross platform project with some files that must be crlf in both windows and Linux, and some that must be lf on both os.

Seems like subrepo required no logic changes for windows. It was all about making it compatible with the variants of bash and allowing for line ending differences especially in tests.

I've used it extensively for the last week - at least I've used init and clone a lot! Just starting some push/pull tests.

Warm regards

Rob Goodridge

On 3 Mar 2016, at 1:47 AM, Ingy döt Net notifications@github.com wrote:

@robe070 you are welcome!

I'm sure there are bugs lurking, but if all the tests pass on a stock install, then what more can be done? :)

The approach I took was use .gitattributes file you suggested in the main repo. and also in one of the tests that creates a repo. Then I set git config core.autocrlf input (not --global) in the test/setup file, so the test suite is always running under that. Then I had to do the same thing for the tests that clone a fake repo.

So basically making the tests run in a posix-ish environment. I suppose that leaves a lot of room for things to be weird in Windows dev and in cross dev.

We shall see... we shall fail... we shall fix...

—
Reply to this email directly or view it on GitHub.

[grimmySwe]

Don't know if this is common knowledge but running git on windows might be quite slow, so there are some improvements:
https://github.com/msysgit/msysgit/wiki/Diagnosing-why-Git-is-so-slow

[ingydotnet]

@grimmySwe thanks for that. We should play around with those things and adjust the code and/or docs accordingly.

[logictom]

It might be worth adding a note under Windows installation, for those like my that don't have quite as good an understanding of bash, that .bashrc isn't read but .bash_profile instead.
I added
if [ -f "$HOME/.bashrc" ]; then
. "$HOME/.bashrc"
fi
to the top of .bash_profile as per discussion here git-for-windows/git#191
If this isn't the accepted solution perhaps the alternative could be added instead.