i18n: Allow custom language codes #3971

KevinGimbel · 2017-10-15T10:42:18Z

This PR uses the new RegisterPluralSpec function from go-i18n to register all languages defined. Previously only language codes available within the Unicode CLDR standard could be defined.

Use the new `RegisterPluralSpec` function to register all defined languages. This allows the usage of language identifiers which are not part of the Unicode CLDR standard. See gohugoio#3564

CLAassistant · 2017-10-15T10:42:24Z

All committers have signed the CLA.

KevinGimbel · 2017-10-15T10:54:13Z

~~Interesting. Travis fails on mage -v check but this did not fail on my machine. I'll take a look at it.~~

It now fails on my machine, too. I'll look into it.

KevinGimbel · 2017-10-27T08:39:39Z

@BoGeM Could you help me with this? I added a test for "dk.toml" to the i18n_test.go file, see https://github.com/gohugoio/hugo/pull/3971/files#diff-e0ea71d98579d0c18004dbb09fdfaff7R142

If I run mage test the test passes, which means the previously unsupported language "dk" can now be loaded.

However, with the new RegisterPluralSpec function in translationProvider.go I get a data race error. I don't know how to debug this - could you give me a hint?

n10v

LGTM, but with some correctives below.
Data race is an issue of i18n library. I will make a PR for it.

n10v · 2017-10-30T20:00:33Z

i18n/i18n_test.go

+
+	for file := range test.data {
+		id := strings.Split(file, ".toml")
+		ids = append(ids, id[0])


Use this:

id : = strings.TrimSuffix(file, ".toml") ids = append(ids, id)

n10v · 2017-10-30T20:10:16Z

i18n/translationProvider.go

@@ -37,6 +38,8 @@ func (tp *TranslationProvider) Update(d *deps.Deps) error {
 	dir := d.PathSpec.AbsPathify(d.Cfg.GetString("i18nDir"))
 	sp := source.NewSourceSpec(d.Cfg, d.Fs)
 	sources := []source.Input{sp.NewFilesystem(dir)}
+	ps := &language.PluralSpec{}


Move this just before the 60th line.

n10v · 2017-10-30T20:10:40Z

i18n/translationProvider.go

@@ -37,6 +38,8 @@ func (tp *TranslationProvider) Update(d *deps.Deps) error {
 	dir := d.PathSpec.AbsPathify(d.Cfg.GetString("i18nDir"))
 	sp := source.NewSourceSpec(d.Cfg, d.Fs)
 	sources := []source.Input{sp.NewFilesystem(dir)}
+	ps := &language.PluralSpec{}
+	langs := []string{}


Move this just before the 54th line.

n10v · 2017-10-30T20:13:55Z

i18n/translationProvider.go

@@ -50,6 +53,14 @@ func (tp *TranslationProvider) Update(d *deps.Deps) error {

 	for _, currentSource := range sources {


Add a comment here that we should plural specs of languages to prevent the error of using custom language codes.
And add a link to #3564 too.

See gohugoio/hugo#3971

n10v · 2017-10-30T20:58:14Z

See nicksnyder/go-i18n#81

KevinGimbel · 2017-10-31T15:34:52Z

@BoGeM I made the following changes:

Move variable initialization in translationProvider.go
Add comment about language specs with a reference to the original issue
Change the language ID parsing from string.Split to string.TrimSuffix

n10v · 2017-10-31T19:06:51Z

i18n/translationProvider.go

+			langs = append(langs, r.BaseFileName())
+		}
+	}
+	// we need to register all language codes as "plural spec" to prevent errors


Add dots at the end of sentences and please start sentences with capital letters.

KevinGimbel · 2017-11-01T08:39:18Z

@BoGeM I capitalized the "w" in "We" and added a dot at the end of the sentence.

n10v · 2017-11-02T08:28:28Z

To prevent data race try to do the following:

Add sync.Mutex to translationProvider:

// i18n/translationProvider.go

// TranslationProvider provides translation handling, i.e. loading
// of bundles etc.
type TranslationProvider struct {
    t Translator
    sync.Mutex // <--
}

Use mutex when registering plural spec:

// i18n/translationProvider.go

ps := &language.PluralSpec{}
tp.Lock()
language.RegisterPluralSpec(langs, ps)
tp.Unlock()

Actually I'm not sure, if it will help. But anyway let me know about the result of tests.

bep · 2017-11-02T08:35:41Z

Actually I'm not sure, if it will help. But anyway let me know about the result of tests.

It will not help. You cannot just lock the write -- this is a read/write map race. Wasn't this fixed upstream? If so, you need to update your dependencies.

KevinGimbel · 2017-11-02T08:46:13Z

@bep the fix from @BoGeM was rejected upstream nicksnyder/go-i18n#81

n10v · 2017-11-02T09:16:13Z

@bep the fix from @BoGeM was rejected upstream nicksnyder/go-i18n#81

Exactly. And I find @nicksnyder's arguments convincing.

bep · 2017-11-02T09:20:17Z

OK, but my point still holds. You cannot just lock the writer side of it.

n10v · 2017-11-02T09:25:20Z

@bep how can we add locking for read operations? Can you help please?

bep · 2017-11-02T09:27:32Z

I will look. Ideally we would do the RegisterPluralSpec before we start any processing, but this is a global used by every site, so I think concurrency is hard to avoid. But I will have a look.

bep · 2017-11-02T09:48:11Z

See nicksnyder/go-i18n#82

bep · 2017-11-02T12:13:55Z

As to "getting this one working", all the races seems to be in the Update method, so this should work:

bep@96ed4f5

KevinGimbel · 2017-11-02T12:39:14Z

@bep thanks a lot for getting into this, I was completely lost unfortunately!

Should I add the code to my PR or will you merge your branch?

bep · 2017-11-02T13:28:02Z

Just copy and paste my code.

KevinGimbel · 2017-11-02T15:17:49Z

@bep I added your changes to my PR. mage hugoRace no longer fail on my machine, but mage check still fails.

bep · 2017-11-03T16:51:03Z

@KevinGimbel I will fix/merge this tomorrow. And for the long run, I will see if I can create a PR for the go-18n lib ... some time.

bep · 2017-11-04T08:50:03Z

#4049 replaces and includes this

github-actions · 2022-02-08T01:53:03Z

This pull request has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

i18n: Allow custom language codes

8561e69

Use the new `RegisterPluralSpec` function to register all defined languages. This allows the usage of language identifiers which are not part of the Unicode CLDR standard. See gohugoio#3564

KevinGimbel mentioned this pull request Oct 15, 2017

Custom language codes not support in multilingual setup #3564

Closed

KevinGimbel added 2 commits October 18, 2017 16:01

i18n: Add test for custom language code

1742fe9

i18n: Un-comment code

74c9134

n10v reviewed Oct 30, 2017

View reviewed changes

n10v added a commit to n10v/go-i18n that referenced this pull request Oct 30, 2017

language: Prevent data race while registering plural specs

61a8ce5

See gohugoio/hugo#3971

n10v mentioned this pull request Oct 30, 2017

language: Prevent data race while registering plural specs nicksnyder/go-i18n#81

Closed

i18n: Add comments about language specs; Move variable initialization

55686ec

n10v reviewed Oct 31, 2017

View reviewed changes

i18n: Capitalize first letter in comment and add dot at the end

79343bf

i18n: Add sync to prevent data race

7803972

bep mentioned this pull request Nov 4, 2017

The custom language code saga continues #4049

Merged

bep closed this Nov 4, 2017

github-actions bot locked as resolved and limited conversation to collaborators Feb 8, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

i18n: Allow custom language codes #3971

i18n: Allow custom language codes #3971

KevinGimbel commented Oct 15, 2017

CLAassistant commented Oct 15, 2017 •

edited

Loading

KevinGimbel commented Oct 15, 2017 •

edited

Loading

KevinGimbel commented Oct 27, 2017

n10v left a comment

n10v Oct 30, 2017

n10v Oct 30, 2017

n10v Oct 30, 2017

n10v Oct 30, 2017

n10v commented Oct 30, 2017

KevinGimbel commented Oct 31, 2017 •

edited

Loading

n10v Oct 31, 2017 •

edited

Loading

KevinGimbel commented Nov 1, 2017

n10v commented Nov 2, 2017 •

edited

Loading

bep commented Nov 2, 2017

KevinGimbel commented Nov 2, 2017

n10v commented Nov 2, 2017

bep commented Nov 2, 2017

n10v commented Nov 2, 2017

bep commented Nov 2, 2017

bep commented Nov 2, 2017

bep commented Nov 2, 2017

KevinGimbel commented Nov 2, 2017

bep commented Nov 2, 2017

KevinGimbel commented Nov 2, 2017

bep commented Nov 3, 2017

bep commented Nov 4, 2017

github-actions bot commented Feb 8, 2022

		@@ -50,6 +53,14 @@ func (tp TranslationProvider) Update(d deps.Deps) error {

		for _, currentSource := range sources {

i18n: Allow custom language codes #3971

i18n: Allow custom language codes #3971

Conversation

KevinGimbel commented Oct 15, 2017

CLAassistant commented Oct 15, 2017 • edited Loading

KevinGimbel commented Oct 15, 2017 • edited Loading

KevinGimbel commented Oct 27, 2017

n10v left a comment

Choose a reason for hiding this comment

n10v Oct 30, 2017

Choose a reason for hiding this comment

n10v Oct 30, 2017

Choose a reason for hiding this comment

n10v Oct 30, 2017

Choose a reason for hiding this comment

n10v Oct 30, 2017

Choose a reason for hiding this comment

n10v commented Oct 30, 2017

KevinGimbel commented Oct 31, 2017 • edited Loading

n10v Oct 31, 2017 • edited Loading

Choose a reason for hiding this comment

KevinGimbel commented Nov 1, 2017

n10v commented Nov 2, 2017 • edited Loading

bep commented Nov 2, 2017

KevinGimbel commented Nov 2, 2017

n10v commented Nov 2, 2017

bep commented Nov 2, 2017

n10v commented Nov 2, 2017

bep commented Nov 2, 2017

bep commented Nov 2, 2017

bep commented Nov 2, 2017

KevinGimbel commented Nov 2, 2017

bep commented Nov 2, 2017

KevinGimbel commented Nov 2, 2017

bep commented Nov 3, 2017

bep commented Nov 4, 2017

github-actions bot commented Feb 8, 2022

CLAassistant commented Oct 15, 2017 •

edited

Loading

KevinGimbel commented Oct 15, 2017 •

edited

Loading

KevinGimbel commented Oct 31, 2017 •

edited

Loading

n10v Oct 31, 2017 •

edited

Loading

n10v commented Nov 2, 2017 •

edited

Loading