Serbian Language issues in admin

milanleon · April 28, 2017, 6:25pm

Well I need to share this issues from Plone installation in the part of drop down language options.

When admin wants to change language has option for Serbo - Croatian .
First of all this language does not exist anymore.

So this should be removed from Plone and language options.
Second is that there is Croatian language - Hrvatski (Official Croatian language) and that is fine.

But also there must be Serbian language.
Far as I see there is Serbian language on cyrilic and with small letters (српски) , which is not good for future Serbian Plone community, its seems offensive, cause native Serbian language is cyrilic but also latin.
So it should be Serbian Cyrilic (Српски Ћирилица) and Serbian Latin (Srpski Latinica)

I hope that Plone developers will take this to consider to change.
If they need help, I am here.

Thank you

gforcada · April 29, 2017, 8:25am

Hi @milanleon thanks for reaching us!

Yes, we do care a lot about languages, that's why Plone by default is multilingual aware

How are your python skills? I haven't really test it but a quick search made me think that the list of languages comes from this file:

github.com

plone/plone.i18n/blob/master/plone/i18n/locales/languages.py#120

# -*- coding: UTF-8 -*-
from plone.i18n.locales.interfaces import IContentLanguageAvailability
from plone.i18n.locales.interfaces import ILanguageAvailability
from plone.i18n.locales.interfaces import IMetadataLanguageAvailability
from zope.interface import implementer

import six


@implementer(ILanguageAvailability)
class LanguageAvailability(object):
    """A list of available languages.
    """

    def getAvailableLanguages(self, combined=False):
        """Return a sequence of language tags for available languages.
        """
        languages = list(_languagelist.keys())
        if combined:
            languages.extend(list(_combinedlanguagelist.keys()))

This file has been truncated. show original

Would you mind to provide a pull request to fix it? We can provide you guidance if you need to.

Again, thanks for taking your time to report this issue!

milanleon · April 30, 2017, 11:57am

Hi @gforcada thanks for an answer

Glad to hear that you care about languages and I am good with Python ...long term friendship

Sure, I have sent pull request from Digitalna Srbija on Github, but also if you have any guidance please send...

If I have more infos I will done better job...

I am even register new Plone domain for Serbian community... Soon will be public.
Sorry for delay and thank you for answer once again.

gforcada · April 30, 2017, 4:59pm

I'm too eager or I'm missing the pull request on plone.i18n?

milanleon · May 1, 2017, 3:07pm

I am so sorry Im having some issues here...

You can check now the file languages.py.zip on the https://github.com/plone/plone.i18n/pull/16
From some reason I couldn't upload from terminal , but I uploaded to my repo in https://github.com/digitalnaSRB/Serbian-translation

If you need anything else please let me know, cause I really want to contribute Plone...

p.s. why dont we have mobile app for Plone community ?

tkimnguyen · May 1, 2017, 3:40pm

There is work being done with Angular and React on top of Plone+RESTAPI (the Plone "headless CMS" project), which you can find described elsewhere in this forum

gforcada · May 1, 2017, 4:52pm

@milanleon I closed your pull request as it seemed you made a mistake there, no problem though, thanks to he zip I could create a proper one: https://github.com/plone/plone.i18n/pull/17 I added some questions on a comment though, would you mind double checking it?

milanleon · May 2, 2017, 6:44am

@gforcada yes I saw it and I know on what kind of mistakes you think...

intentionally I've put a different codes (meaning of "rs" and "sr") cause I knew you will check it and in lack of informations about this changes I was expected your questions...

I'ill try to answer to your questions.
If you check other CMS, such as Wordpress, Drupal, Juoomla etc... In most of them are language code is "rs-sr" not sr (cause "sr" is Suriname) if you stick to ISO code and don't ask me why and how, cause I dont know
So, if you stick to ISO code then RS (meaning of Republic of Serbia) is

So I thought maybe this will goes on same way in Plone as it is on other CMS... (ok it is not)

So, this changes was more like test file so I just need more instructions for other changes and it will be done on proper way.
Yes, my mistake was that I've put twice "sr-ba" instead of once as it was before, I want it to put "sr-rs" I mean that was sound logically for me.

Also I have to explain about this languages there is something more then this codes and I think for future community for Bosnia and Herzegovina, as well as Montenegro (don't have this in files) you should make changes in files....
This is just my suggestion, and I can PM you with screen shots if you want to know more about this.

Cause this language order is not so well for other communities.
Just one exp. If someone from Bosnia want to choose language he had only one option and this is "Bosanski" or from Macedonia has "Mакедонски" also not proper...

All this languages should be selected as example of
-Deutsch
-Deutsch (Österreich)

I am sorry for such a long message but I really believe that Plone can make real good thing in this part of world so that's why I am trying to help.

Thanks

gforcada · May 3, 2017, 1:13pm

And I'm happy that you put your effort on bringing the topic up.

I see a few things here:

The first and easier would be the last one you mention, that just like German and German (Austria) some countries do need their specific Serbian language version, is that right? A pull request with only these changes would be non-controversial to merge I guess.

Regarding the rs language that you made up, I'm rather skeptical of merging that. Is there any formal process being made to get in contact with ISO and get that code in the official list? If the idea was only to be able to have on the same listing both the name in Cyrillic as well as Latin, for other languages with digraphia (I had to look it up on wikipedia :D) like Urdu, we only use one writing system.

The whole data structure would have to be completely changed to support different locale names for a single language.

Is that something really important for Serbians, or like is stated in the Wikipedia article, all serbians are able to read equally fluently in both writing systems?

I hope I touched all the points. Let me know if not.

milanleon · May 3, 2017, 7:57pm

yep, now I will give you some links and you can compare all differences.

Now I will answer from the start and I will repeat you words

The first and easier would be the last one you mention, that just like German and German (Austria) some countries do need their specific Serbian language version, is that right? A pull request with only these changes would be non-controversial to merge I guess.

Yes, as you said , it would be best options to do with Serbian language..
So it could be made like

Serbian (Latin)
Serbian (Cyrilic) or (Српски језик) <---- translation is "Serbian language"

Regarding the rs language that you made up, I'm rather skeptical of merging that. Is there any formal process being made to get in contact with ISO and get that code in the official list? If the idea was only to be able to have on the same listing both the name in Cyrillic as well as Latin, for other languages with digraphia (I had to look it up on wikipedia :D) like Urdu, we only use one writing system.

Regarding codes take a look first this source Serbian language - Wikipedia
and you will find ISO codes
ISO_3166-2:RS

And after that this one specially second part of page, below the table Country codes of Serbia - Wikipedia

So I can suggest only that Plone could have Serbian in two options

Serbian Latin
Serbian Cyrilic (Српски језик)

About code I can't give you any suggestion cause you can see on official ISO site how is going for Internet, cause even domestic domain name for Serbia is .RS

The whole data structure would have to be completely changed to support different locale names for a single language.

I don't think so you have to change all structure of site but I am not sure of this part of Plone structure which needs to be change, but I can show you on this link how it is look like Serbian language officialy for Wordpress and I know the guys who are working on it and there is code ( sr_RS) because this makes differences from Suriname.

WP_official_translation

Also I have registered domain name for Plone at Plone.RS and it is online already. Our domain is .RS as it is for Germany .DE

Is that something really important for Serbians, or like is stated in the Wikipedia article, all serbians are able to read equally fluently in both writing systems?

For Serbian community is only important to have two options as you mentioned before in case of Austria (Germany language)
So if there is any chance to make something like Serbian Latin and Serbian Cyrilic it would be perfect.

I have in plans to make meetup and other events of Plone for Serbian community so I think the language is essential things if we want to develope something good.

So it is all up to you... ...Plone
I can only contribute and help

petri · May 4, 2017, 7:08am

Thank you @milanleon for your contribution. The suggested additions sound like a good idea.

I am not however sure at all that we should also remove Serbo-Croatian from the list of language options in Plone core:

Languages live and die by their speakers, not by political or standards committee decisions. In addition to those now speaking "Croatian" or "Serbian" or "Bosnian" etc., there are (older) generations of speakers who learned and become educated in "Serbo-Croatian". I'd imagine it reasonable to think they might still prefer to call their language that, instead.

More often than not, it is good to be inclusive (offer everyone what they need) rather than exclusive (removing options). Especially if no single truth or consensus can be found or exist.

At least both wikipedia entry on Serbo-Croatian and Encyclopedia Britannica seem to suggest the issue is not simple, clear, or black-and-white. Some seem to think "Serbo-Croatian" should be called a "macro" language, or that it could be used to refer to the Croatian, Serbian etc. languages collectively, given that the speakers of those languages can understand each other.

Perhaps that kind of "collective" or "language group" usage or semantics for Serbo-Croatian "language" option might somehow be useful in Plone, for the aggregate regional website audiences (that sort of sidesteps any linguistical distinctiveness argument). Many 'dead' languages still serve such auxiliary or secondary purposes; just think of Latin.

In any case, I think Plone should be able to serve all and everyone, as much as possible. Adding language options for people is good, whereas removing options should not be done hastily, or without consideration or a good reason.

Also, I can see no harm from keeping the language option: After all, the site admin has to explicitly enable languages in order for them to be available to users.

Just my two cents' worth, as the old saying goes.

milanleon · May 4, 2017, 9:50am

Hi @petri

thanks for this comment,
I don't mind if Plone keep this "language" "Serbo-Croatian", my point was more likely to add Serbian Latin and Serbian Cyrilic to the Plone, but not so much to remove Serbo-Croatian...

I care more for our community to have the Plone on native language, that's all.
If we have for other communities such as Wordpress, Joomla, Drupal, Mozzila etc.. on Serbian I don't see the reason why don't we have for Plone too.

Maybe I used hard word about removing this "language".
Point is that this "language" nobody used to declare anymore in this part of world.

Because if you see the list of Plone, you can find Croatian, Serbian, Bosnian, Macedonian languages and all of them there, so then Serbo-Coratian misses the point.

But if you think that should be there, that's fine with me.

I agree, that's why I want to see in Plone Serbian Latin and Serbian Cyrilic..

jensens · August 7, 2017, 11:20am

Back here, some backgrounds:

We shall use W3C Language Tags which are the same as IETF Language tags, as HTTP and HTML does.

Tags have (basically, read the above-linked texts for details) the form LANGUAGE-[SUBTAG].

An excellent reading on this topic is this W3C article https://www.w3.org/International/articles/language-tags/

Ok, @milanleon wants to support the different Serbian languages. So there are two different script types

Cyrl,
Latn

And afaik these different regions (as countries https://en.wikipedia.org/wiki/ISO_3166-2):

RS - Serbia
BA - Bosnia and Herzegovina
ME - Montenegro
(is something missing here?)

Since I dont know the serbian language at all, I am not sure if the regions need different translations?

It looks like we need at least

sr (Latin variant is default and does not need a subtag)
sr-Cyrl(Cyrillic)

If the region is important (because language is different), this may be extended in any combination with sr-RS, sr-BA, sr-ME or also as sr-RS-Cyrl and so on.

Btw., according to Wikipedia about Serbian-Croation, language code is sh and it is deprecated.

fredvd · June 30, 2021, 1:00pm

So here i find myself searching for how to activate the Latin script variant of Serbian in a Plone site and finding this thread from 2017.

After doing some research online I think @jensens has the correct summary/conclusion. Other open source like like Moments.js have had similar tickets (Wrong script used for Serbian locale, wrong date format · Issue #4740 · moment/moment · GitHub)

From the limited feedback I got, Serbian Latin(ica), Srpski is the default spoken/written language for normal day use, but Cyrillic is still used as well and sort of required for legal documents.

In the current Plone languages.py setup in plone.i18n, Cyrillic is the default. Interestingly, in plone.app.locales there is also Srpski Latinica, but it is stored in sr_Latn and sr contains the Cyrillic version.

language code wise: rs is the new TLD, but rs is not an official language code for Srpski Latinica. Also Latin and Cyrillic are not language variants but 'scripts'. One should denominate them with sr@Latn or sr@Cyrl

If you search for @Latn in plone.i18n there are even docstrings for the normalizer mentioning sr@Latn: Search · Latn · GitHub

But I have found no way to switch between the scripts in the language control panel or how I should modifiy plone.i18n to make that sr_Latn locales alo an option.

I can make my own fork of plone.app.locales and switch the two directories for the time being....

mauritsvanrees · August 31, 2021, 9:41pm

Fixing this along the lines indicated above is not so hard. I have code ready for this now.
But while working on it, I realised that this may give migration troubles for existing content.
I created an issue in plone.app.locales for it:

github.com/collective/plone.app.locales

Serbian Latin should be the default script for sr

opened 09:39PM - 31 Aug 21 UTC

mauritsvanrees

See this [community thread](https://community.plone.org/t/serbian-language-issue…s-in-admin/4115) started in 2017. This [comment from @jensens](https://community.plone.org/t/serbian-language-issues-in-admin/4115/13) is a good summary, followed four years later by a comment from my colleague @fredvd who wants me to fix it for a customer. Problems: - We have Plone translations in Serbian Cyrillic (folder `sr`) and Latin (`sr_Latn`), but in the language control panel there is no way to choose the latin version. Note: the Cyrillic one is sorted near the bottom of the list as српски (sorting might depend on your locale I guess). - The Cyrillic script is the default, but it would be better if the Latin one is the default: it is the script most commonly used. At least this is how I understand it. So what should be done: - `plone.i18n`: in the language list change `sr` to mean Serbian Latin, and add `sr-cyrl`. Needs to be done separately on branch master and 4.x. - `plone.app.locales`: rename the `sr` folder to `sr_Cyrl` and rename `sr_Latn` to `sr`. I have code ready for this, and I want to push it but first describe the issue. But now I think: what happens with existing sites that have the 'sr' language? Say you start out a site in sr, when it still means Cyrillic. And this is the script you want, otherwise you would have done something else, maybe chosen sr-ba (Serbian for Bosnia-Herzegovina) instead. Then you update to a Plone version where sr means Latin. Suddenly your content is in Cyrillic and your UI in Latin. So in the language control panel you switch to the new sr-Cyrl and remove sr from the available languages. Seems to work. But then you edit the News folder and get a traceback: ConstraintNotSatisfied: ('sr', 'language'). You can go back and allow sr again, so you have two languages, but this is not really what you want. You now need to write a script to update the language field of all items to sr-Cyrl. Multilingual is probably worse. Start with English and sr (Cyrillic) and enable multilingual. Create some content. Update the code and you again have Cyrillic content end Latin UI. So you enable sr-Cyrl as extra language. Now you still have the old sr language root folder with Cyrillic content and Latin UI, and you have a new sr-cyrl folder with Cyrillic UI but no content. You now need to move all items from sr to sr-Cyrl. And maybe edit some Apache/nginx rewrite rules. So for the people who *do* use Serbian Cyrillic currently, the planned changes make things worse and lead to extra work. What is the alternative? - `plone.i18n`: in the language list keep `sr` to mean Serbian Cyrillic, and add `sr-latn`. - `plone.app.locales`: actually nothing needs to change. Now what if you wanted a Serbian Latin site and started it with the current code, so without the above changes. Current status is that you have Cyrillic UI and Latin content. Not ideal. New status after the alternative, and after switching to the new sr-latn language in the control panel, is that UI and content are Latin, which is good. Well, you have the same problem that all your current content has the sr language, so you would need a script to update it. And multilingual also has the same problems. And what if you already did a hack and created a temporary copy of plone.app.locales where you copied sr_Latn to sr? I might know a project where we did this.😉. Then the first approach above means you can get rid of this temporary copy, which is why Fred asked me to work on this. The alternative from above would mean extra work: choose the new sr-latn language, and create a script to update the current content to use this language. In our case it is a multilingual site, so it would suffer from the same problems as described above. In this case we could copy the sr_Latn folder to a locales folder in the buildout of this project, which would mean we can get rid of the temporary plone.app.locales copy. I guess there are currently sites in the wild which use the 'sr' code so they have Cyrillic UI. Some will have Cyrillic content, some will have Latin content. Whatever we change, someone will not like it, and will need to do some extra work. In both cases, a workaround is to either copy old sr_Latn folder or the new sr_Cyrl folder to a `locales/sr` folder in your buildout, depending on which language you want to use. I suppose the first approach that I describe above would be good, so let 'sr' be Latin. But since this can be considered a breaking change, it seems better to only do this in Plone 6. It is one more reason for letting master only [target Plone 6](https://github.com/collective/plone.app.locales/issues/315) at some point in the near future.

mauritsvanrees · September 2, 2021, 7:20am

@milanleon Are you still around on the forum? Can you give guidance on this issue? See Serbian Latin should be the default script for sr · Issue #326 · collective/plone.app.locales · GitHub for some ways of how to technically tackle this.

Trickiest is how to support both Cyrillic and Latin in the same site, either with or without plone.app.multilingual installed. And question is whether this is ever needed in practice.

Also tricky is how to not break an existing site which is perfectly happy keeping everything in Cyrillic.