Serbian Latin should be the default script for sr

[mauritsvanrees]

See this community thread started in 2017.
This comment from @jensens is a good summary, followed four years later by a comment from my colleague @fredvd who wants me to fix it for a customer.

Problems:

• We have Plone translations in Serbian Cyrillic (folder sr) and Latin (sr_Latn), but in the language control panel there is no way to choose the latin version. Note: the Cyrillic one is sorted near the bottom of the list as српски (sorting might depend on your locale I guess).
• The Cyrillic script is the default, but it would be better if the Latin one is the default: it is the script most commonly used. At least this is how I understand it.

So what should be done:

• plone.i18n: in the language list change sr to mean Serbian Latin, and add sr-cyrl. Needs to be done separately on branch master and 4.x.
• plone.app.locales: rename the sr folder to sr_Cyrl and rename sr_Latn to sr.

I have code ready for this, and I want to push it but first describe the issue.
But now I think: what happens with existing sites that have the 'sr' language?

Say you start out a site in sr, when it still means Cyrillic. And this is the script you want, otherwise you would have done something else, maybe chosen sr-ba (Serbian for Bosnia-Herzegovina) instead.
Then you update to a Plone version where sr means Latin. Suddenly your content is in Cyrillic and your UI in Latin.
So in the language control panel you switch to the new sr-Cyrl and remove sr from the available languages.
Seems to work.
But then you edit the News folder and get a traceback: ConstraintNotSatisfied: ('sr', 'language').
You can go back and allow sr again, so you have two languages, but this is not really what you want.
You now need to write a script to update the language field of all items to sr-Cyrl.

Multilingual is probably worse.
Start with English and sr (Cyrillic) and enable multilingual. Create some content.
Update the code and you again have Cyrillic content end Latin UI.
So you enable sr-Cyrl as extra language.
Now you still have the old sr language root folder with Cyrillic content and Latin UI, and you have a new sr-cyrl folder with Cyrillic UI but no content.
You now need to move all items from sr to sr-Cyrl. And maybe edit some Apache/nginx rewrite rules.

So for the people who do use Serbian Cyrillic currently, the planned changes make things worse and lead to extra work.

What is the alternative?

• plone.i18n: in the language list keep sr to mean Serbian Cyrillic, and add sr-latn.
• plone.app.locales: actually nothing needs to change.

Now what if you wanted a Serbian Latin site and started it with the current code, so without the above changes.
Current status is that you have Cyrillic UI and Latin content. Not ideal.
New status after the alternative, and after switching to the new sr-latn language in the control panel, is that UI and content are Latin, which is good.
Well, you have the same problem that all your current content has the sr language, so you would need a script to update it. And multilingual also has the same problems.

And what if you already did a hack and created a temporary copy of plone.app.locales where you copied sr_Latn to sr? I might know a project where we did this.😉.
Then the first approach above means you can get rid of this temporary copy, which is why Fred asked me to work on this.
The alternative from above would mean extra work: choose the new sr-latn language, and create a script to update the current content to use this language. In our case it is a multilingual site, so it would suffer from the same problems as described above.
In this case we could copy the sr_Latn folder to a locales folder in the buildout of this project, which would mean we can get rid of the temporary plone.app.locales copy.

I guess there are currently sites in the wild which use the 'sr' code so they have Cyrillic UI. Some will have Cyrillic content, some will have Latin content. Whatever we change, someone will not like it, and will need to do some extra work. In both cases, a workaround is to either copy old sr_Latn folder or the new sr_Cyrl folder to a locales/sr folder in your buildout, depending on which language you want to use.

I suppose the first approach that I describe above would be good, so let 'sr' be Latin.
But since this can be considered a breaking change, it seems better to only do this in Plone 6.
It is one more reason for letting master only target Plone 6 at some point in the near future.

[jensens]

I would go for your first approach and fix it for Plone 6 to get it right in here and keep it for older versions as is.

We could add a migration step to fix content (and LanguagRootFolder's) and change sr-latn to sr and existing sr to sr-cyrl.
If a any plugin comes with translations, it needs a change, this is fine IMO.

[fredvd]

I guess there are currently sites in the wild which use the 'sr' code so they have Cyrillic UI. Some will have Cyrillic content, some will have Latin content. Whatever we change, someone will not like it, and will need to do some extra work.

The information we got from the new webmaster for this site was that nowadays most Serbian written communication is in Latin and not in Cyrillic character. but that's off course N=1.

The problem in this case is that there maybe isn't a real preference, they support both.

@mauritsvanrees What I missedin in your research as a consideration is supporting the syntax as written in the plone.i18n tests for character set variants, sr@Latn and sr@Cyrl . These should then sill be mapped to sr_Latn and sr_Cyrl folders, like the floating '-' in ab-xy is now mapped to folder ab_xy in plone.app.locales.

Without a preference we could do first come first serve alfabetically (which maps nicely sr to sr_Cyrl), or add an extra dict in plone.i18n for character set variants to language code.

Then we can patch that dict in our project where I renamed both directories, or update the content to sr@Latn

Or is supporting @ opening a few cans of worms in the current Zope i18n code base?

[mauritsvanrees]

There does not seem to be anything in the current code that does something with '@'.
If we start supporting this, how should this work?
It must work differently than with a dash, otherwise we could simply use the dash for this case as well.
What I can imagine:

• In the language control panel add an extra checkbox: 'Show character set variants', just like we currently have 'Show country-specific language variants'. When you check this and save the form, we show sr@Cyrl and sr@Latn in the gigantic list. Otherwise we only show 'sr'.
• Easier: do not add a checkbox. Simply always show both character set options.
• When you choose either of these two, it is treated as if you chose 'sr' as language. In a multilingual site with English as second language, you would get language folders 'en' and 'sr'.
• You can select both as available language. This means you can choose between the two when editing a content item. In the Negotiation Scheme you can select 'Use the language of the content item', and this should work fine, showing the UI in Latin or Cyrillic where appropriate.
• In the Negotiation Scheme you can select 'Use language codes in URL path for manual override'. What happens then when you create a folder 'sr'? Plone looks in the list of selected available languages and uses the character set from the first matching language. Note that you can change the order of the languages.
• I guess the trickiest thing is: knowing when sr@Latn should stay as it is and when to interpret it as 'sr'. Which of the two end up in the LanguageIndex in portal_catalog? Which is used as id for a Language Root Folder? How many places in Plone are there that we need to check?

We may need to limit the scope of our initial thinking:

• Only consider the use case of picking ONE character variant. Ignore the possibility of someone enabling both.
• Ignore multilingual.
• Ignore the possibility of someone switching from one character set to another.

So what could we do?

• Keep plone.app.locales the same: folders 'sr' and 'sr_Latn'. (In the future we may move sr to sr_Cyrl.)
• In plone.i18n add sr@Latn to the main _languagelist.
• We might need changes in https://github.com/zopefoundation/zope.i18n/blob/master/src/zope/i18n/negotiator.py which is called from https://github.com/zopefoundation/zope.i18n/blob/master/src/zope/i18n/translationdomain.py#L77-L81
• In general click around, see where translations that should be available are missing, and figure out where this needs changing.

A different approach may be interesting, adding the character set support at the zope level:

• In plone.app.locales, rename sr_Latn to sr@Latn. This is what Debian does, see https://www.debian.org/international/l10n/po/sr@Latn
• In plone.i18n keep only 'sr' in the _languagelist, so this bit stays as it currently is.
• Change https://github.com/zopefoundation/zope.i18n/blob/4.7.0/src/zope/i18n/translationdomain.py#L52-L55 to use 'sr' as key when registering the sr@Latn catalog, and insert it as the first catalog. Maybe have some environment variables or zope configuration to determine what to do.
• Plone then only knows of 'sr' as language. No migration needed.
• This automatically means it is not possible to choose between Latin and Cyrillic in the Plone UI, which could be good or bad. But currently you cannot choose either.

Sigh: lots of possibilities, but not yet one clear winner.

[erral]

Is there no Serbian developer in the Plone Community who may have an insight on this?

[mauritsvanrees]

Wait, how about this:

• In a buildout where you only want the Latin characters, set the zope_i18n_allowed_languages env to include sr@Latn and not sr.
• Then zope.i18n.zcml will only try to load the Latin locale.
• In zope.i18n.translationdomain in method _registerMessageCatalog use sr as key when registering the sr@Latn locale.

This requires only one simple change in zope.i18n.
In plone.app.locales we would rename sr_Latn to sr@Latn. Should be fine in 5.2 because there is no way to use the current Latin directory. Probably copy the current sr to sr@Cyrl. In Plone 6 remove sr.
I tried it and this seems to work fine.

BTW, with some print statements for debugging, it seems that first the sr@Latn directory is loaded, then sr@Cyrl, then sr. So the other way around than I would expect. This might differ per operating system or what your OS locale is.

With this approach, dynamically switching between the two character sets in the Plone UI is not possible. But it is technically the same language, so this should be fine.
Supporting both language variants would be possible if we have both sr@Latn and sr_Latn as directories in plone.app.locales and add sr-latn in the plone.i18n language list, and do the same for the Cyrillic variant.

[fredvd]

With this approach, dynamically switching between the two character sets in the Plone UI is not possible. But it is technically the same language, so this should be fine.

The main question for any Serbian users is if you have a multilingual site where you would want to have this. But as it was currently not possible with Plone, I don't think so. the only thing we want to avoid is switchting current Plone websites with Serbian Cyrillic to Serbian Latin when they install a minor/patch update.

We can add a line in the rellease notes about the zope_i18n_allowed_languages requirement, I think that suffices.

[jensens]

Where is the standard that defines the @ notation? From my findings its always dash.

[mauritsvanrees]

I don't know where the standard is defined, but see for example Serbian Latin translation files in Debian:
https://www.debian.org/international/l10n/po/sr@Latn

[mauritsvanrees]

This is done by now.