Accents (non ASCII 7bit chars?) are not handled correctly with vi (git commit, ...)

[Skywalker13]

Example (release 10ca1f7):

$ mkdir test

$ cd test

$ git init .
Initialized empty Git repository in S:/test/.git/

$ vi README

# Write some accents like ééééé with vi, then there is a problem
# because a space appears between each chars:
  1 é é é é é
# :wq

$ cat README
ééééé
# OK, the content is right even if on the screen (in vi) it was wrong

$ git add README
warning: LF will be replaced by CRLF in README.
The file will have its original line endings in your working directory.

$ git commit
# Same problem with `git commit`, `git commit --amend` of course...

[dscho]

I cannot reproduce this behavior here. This is what I did:

1. From my installation of Git for Windows 2.5.0 64-bit (installed via the installer, not portable Git, the only non-default option is that I checked the experimental option "enable filesystem cache"), I called the Git Bash entry in the start menu
2. Directly after the terminal opened, I called "vi"
3. Then I copied the accented characters from this ticket (because I have a US keyboard) and pasted them.

This is the result:

[Skywalker13]

With a copy/paste I've blanks added just at the end

Otherwise I use the same version and without the filesystem cache but it seems totally unrelated.

I'm on Windows 8.1 English and I use a fr_CH keyboard

[nalla]

Will $ echo $LANG outputs C.UTF-8?

[kostix]

What does vim output if you execute set enc?, set tenc? and set fenc? in it? (These commands are supposed to explain the current Vim's idea about the three encodings it's aware of.)

[Skywalker13]

@nalla nothing...

DevBox2@DevBox2 MINGW64 /s
$ echo $LANG


DevBox2@DevBox2 MINGW64 /s
$

[Skywalker13]

@kostix

:set enc
encoding=latin1
:set tenc
temrencoding=
:set fenc
fileencoding=

[kostix]

@Skywalker13, please copy-and-paste what running locale in the Git Bash window outputs to you.

[Skywalker13]

DevBox2@DevBox2 MINGW64 /s
$ locale
LANG=
LC_CTYPE="C.UTF-8"
LC_NUMERIC="C.UTF-8"
LC_TIME="C.UTF-8"
LC_COLLATE="C.UTF-8"
LC_MONETARY="C.UTF-8"
LC_MESSAGES="C.UTF-8"
LC_ALL=

[kostix]

I'd say vim picks up wrong encoding from the environment.

I have RC5 installed on Win XP SP3 32-bit with Cyrillic locale, and I observe the same behaviour as @Skywalker13 -- just I'm unable to directly input European accented characters but my Cyrillic characters appear in the same way: with a single extra space character each.

Manually setting set enc=utf-8 in vim fixes the issue right away.

[nalla]

I'd say somehow LANG becomes unset. Or is not set proper. How do you start your shell?

[kostix]

By the way, for me locale outputs

$ locale
LANG=C
LC_CTYPE="C"
LC_NUMERIC="C"
LC_TIME="C"
LC_COLLATE="C"
LC_MONETARY="C"
LC_MESSAGES="C"
LC_ALL=

but it's because I've explicitly set LANG=C in the environment to make Git speak English to me.

[kostix]

Doing export LANG=C.UTF-8 in Git Bash and re-running vim fixes its encoding setting immediately, so I'm with @nalla on it.

Though I'd honestly expect whatever layer is involved with this to make UTF-8 assumed if the user has its LANG or LC_MESSAGES set to a string which misses the encoding part.

[nalla]

C:\Program Files\Git\etc\profile.d seems to be missing lang.sh.. @dscho what do you think?

[nalla]

I think a possible permanent workaround is adding

Locale=C
Charset=UTF-8

to your ~/.minttyrc

or by setting LANG=C.UTF-8 explicitly in your ~/.bash_profile.

[kostix]

@nalla, please note that it's possible to opt for not using mintty for Git Bash during installation.

For instance, I'm not using mintty and on my system bash.exe is hosted by cmd.exe.

[nalla]

Sure! updated the comment.

[dscho]

I've explicitly set LANG=C in the environment to make Git speak English to me.

@kostix Please note that I stripped the locales out of the installer before releasing 2.5.0. The reason is a combination of installer size and your comments ;-)

[dscho]

C:\Program Files\Git\etc\profile.d seems to be missing lang.sh.. @dscho what do you think?

@nalla That makes a ton of sense, I think.