Use CLDR for language list source

[kmilos]

Currently iD only supports the 'name:sr' tag in the locale dropdown list, and wrongly labeled as 'Српски / Srpski' at that.

This locale is not intended to be used with both scripts, but Cyrillic only, and should be labeled as 'Српски' only, or 'Serbian (Cyrillic)'.

For Serbian written in Latin script, we have a separate locale 'name:sr-Latn' which should be added to iD dropdown as 'Srpski' only, or 'Serbian (Latin)'.

[bhousel]

I guess these come from data/wikipedia.json

Which seems to match what Wikipedia does:
http://meta.wikimedia.org/wiki/List_of_Wikipedias

[kmilos]

Thanks, I understand where this came from, but it is not a good way to make a distinction between a language (spoken word) and a locale (data written down) [1]. IMHO, Wikipedia painted itself in a corner wrt i18n and L10n with their early design misconception that language=locale.

See BCP 47 [1][2].

[1] http://icu-project.org/repos/icu/icuhtml/trunk/design/language_code_issues.html (note that RFC 3066 has been superseded by RFC 5646 since)
[2] http://www.w3.org/International/questions/qa-choosing-language-tags

[jfirebaugh]

Indeed, the list of languages is sourced from Wikipedia. @kmilos if you know of a better source, we can probably switch. We need something that provides English language name, native language name, and language code.

[kmilos]

@jfirebaugh
A good source for locale data is CLDR: http://www.unicode.org/cldr/charts/latest/supplemental/locale_coverage.html

Not the use of underscore as delimiter, whereas we've went with sr-Latn as recommended in BCP 47. They also recommend any parsers support both - and _

[1ec5]

The wikipedia tag's value specifies a Wikipedia edition, not necessarily a language. Other examples of mismatches include zh-min-nan instead of the ISO code nan and simple, which doesn't correspond to any ISO language code. So it isn't a great source for language codes.

[tmcw]

@1ec5 is CLDR the answer, then?

[1ec5]

CLDR would work, certainly, but it’s a very large package that contains much more than language names. Someone may’ve already created a subset of it specifically for this use case.

[bhousel]

Just a quick followup to this issue that it's now easy to pull in just the parts of CLDR that we want. It has been split into subpackages, automatically converted from xml to json, and published on npm:
https://github.com/unicode-cldr/cldr-json#package-organization

[1ec5]

Per #4632, we’ll need to introduce some iD-specific overrides to support name:* tags that are qualified with an ISO 15924 or ad-hoc script code, such as name:sr-Latn or name:zh-hant.