setlocale( LC_ALL, "") changes the locale for the entire application

[dinghram]

In tidylib.c:
TY_(tidySetLanguage)( setlocale( LC_ALL, "") );

The purpose of this line, as far as I can tell, is not to change the application locale, but only to query it. However, that is not what happens. Rather, it changes the behavior of the entire application to the point where my customers in Europe could not enter floating point values as they were parsed incorrectly after this call that I did not expect to find in a 3rd party library.

From the documentation, if you want to query the locale without changing it:

If locale is a null pointer, setlocale queries the current C locale without modifying it.

[geoffmcl]

@dinghram yes, libtidy does indeed use setlocale(LC_ALL, ""), essentially to query, and set the library language... sorry if this seems to cause you a problem...

Unfortunately, the setlocale(LC_ALL, NULL) you suggest does not yield the actual language in use, it returns C, so is not helpful in this regard...

Is there another way to get/query the system default language, compatible to WIN32 and UNIX?

But this code can be avoided by apps using libtidy, by one of 2 ways -

Either successfully using the tidySetLanguage API, called from their own app before the call to tidyCreate()... this action will set TY_(tidyGetLanguageSetByUser)() == yes...

Or compiling a libtidy without localization support, using like cmake ../.. -DSUPPORT_LOCALIZATIONS:BOOL=OFF [other options]... this will completely remove that code from libtidy... and remove all the language headers leaving only language_en.h as the single message file...

Maybe we need to improve the documentation in this regard? Suggestions appreciated... thanks...

Does this help?

[ler762]

@dinghram that setlocale call seems to be required. Take a look at
https://www.gnu.org/software/libc/manual/html_node/Setting-the-Locale.html
ISO C says that all programs start by default in the standard ‘C’ locale. To use the locales specified by the environment, you must call setlocale. Call it as follows:
setlocale (LC_ALL, "");
to select a locale based on the user choice of the appropriate environment variables.

Would it be possible to get the output from locale and examples of the input your customers said were parsed incorrectly?

[brlin-tw]

I believe the setlocale call is proper, we should fix whatever problems that the user's locale cause after adopting it instead of keep it C or C.UTF-8

[langemeijer]

It is unacceptable for any library function to have changing the locale as side-effect.
At least change the locale back to what is was!

[brlin-tw]

At least change the locale back to what is was!

Good point, this should be as easy as a setlocale(LC_ALL, "C") call.

[langemeijer]

No it isn't! I think you are missing the point on how setlocale() works in the context of an application. setlocale() keeps state for this process. In my case a php thread is setlocale()'d to the locale set for a logged-in user on a website. The user gets correctly formatted dates and such because of this.

Possibly you should not change the locale if it is set to a non-'C' value. Autodetecting the locale according to the environment variables should only be done if no locale is set.

[langemeijer]

Also, you should use LC_MESSAGES for this, not LC_ALL.

I really think you should not be doing any initialization. If library code finds an uninitialized locale situation that's just bad luck and the code should fallback to a decent default language. That is wat "C" is for.

setlocale() documentation http://man7.org/linux/man-pages/man3/setlocale.3.html says:

On startup of the main program, the portable "C" locale is selected
as default. A program may be made portable to all locales by
calling: setlocale(LC_ALL, "");

Emphasis on program is mine.

[brlin-tw]

Also, you should use LC_MESSAGES for this, not LC_ALL.

LC_MESSAGES isn't portable(it isn't in the Standard C, POSIX maybe)

[brlin-tw]

What about:

/* Backup the previous locale */
char *previous_locale = setlocale(LC_ALL, NULL);

/* Inherit the locale from the environment and use it to set Tidy's language */
TY_(tidySetLanguage)( setlocale( LC_ALL, "" ) );

/* Restore the previous locale */
setlocale(LC_ALL, previous_locale);

Does that make sense and solve the OP's problem?

RETURN VALUE
A successful call to setlocale() returns an opaque string that corresponds to the locale
set. This string may be allocated in static storage. The string returned is such that a
subsequent call with that string and its associated category will restore that part of the
process's locale. The return value is NULL if the request cannot be honored.

[geoffmcl]

I certainly agree with @langemeijer -

It is unacceptable for any library function to have changing the locale as side-effect. At least change the locale back to what is was!

@Lin-Buo-Ren maybe the above save and restore patch could be added to PR #785 ...

libTidy only wants a way to auto-detect the users preferred language for LC_MESSAGES output... not in any way effect numeric or date output of the system...

It certainly does not want the change anything... just detect...

In unix, at least, the env vars LC_MESSAGES, or LANGUAGE, may means libTidy does not need, use setlocale( LC_ALL, "" ), so seems the way to go, with or without the patch to save and restore...

Look forward to further feedback on this, and related #783 ... looking positive... thanks...

[russianfool]

@Lin-Buo-Ren: The "check, then change back" is not great in a library either. For multi-threaded users this all falls apart, since there's no guarantee that you'll cache or restore the locale appropriately (T2 caches T1's modified state, and "restores" that modified state when T2 is done; if T1 has finished first, global locale has been reset).

Although maybe Tidy isn't really meant for multi-threaded use; the only spots that I've found so far that break this are the setlocale (which is fine with SUPPORT_LOCALIZATIONS off, or by pre-initializing the language), and the tag_defs set-reset usage (although here due to concurrent table use; easy enough to thread-local the thing).

[langemeijer]

@geoffmcl wrote

Unfortunately, the setlocale(LC_ALL, NULL) you suggest does not yield the actual language in use, it returns C, so is not helpful in this regard...

If the application sets a locale, which is should if it is deemed appropriate, that locale is returned. In my opinion 'C' should fallback to 'en_EN'.
(pseudo-code)

locale = setlocale(LC_ALL, NULL);

if (locale == 'C') {
   TY_(tidySetLanguage)( 'en_EN' );
} else {
   TY_(tidySetLanguage)( locale );
}

As an alternative, (still not very elegant, because I think setlocale( LC_ALL, "" ) should not be in a library) I would suggest the following approach: (pseudo-code)

previous_locale = setlocale(LC_ALL, NULL);

if (previous_locale == 'C') {
   /* Inherit the locale from the environment and use it to set Tidy's language */
   TY_(tidySetLanguage)( setlocale( LC_ALL, "" ) );
   /* Restore the previous locale */
   setlocale(LC_ALL, previous_locale);
}

[geoffmcl]

@ler762, @Lin-Buo-Ren, @russianfool, @langemeijer, all, thank you for the helpful comments...

I too have tried lots of things... see setlocale in tidy-test and test-locale for some of these efforts...

While #783 is more about i18n console tidy support through environment variable, they certainly share the same libTidy code... and may have a common solution...

Since this is like libTidy's gettext function, I was looking at - 2.3.2 Locale Environment Variables, quoted, in part -

A locale is composed of several locale categories, see Aspects. When a program looks up locale dependent values, 
it does this according to the following environment variables, in priority order:

LANGUAGE
LC_ALL
LC_xxx, according to selected locale category: LC_CTYPE, LC_NUMERIC, LC_TIME, LC_COLLATE, LC_MONETARY, LC_MESSAGES, ...
LANG

Given that a libTidy user already has the choices of (a) excluding SUPPORT_LOCALIZATIONS in the build, or (b) by pre-setting the tidySetLanguage, before creating a document...

The following suggested pseudo-code seems to add to that, in that the user's app can further ensure LANGUAGE, ..., LANG, are set in the environment before calling tidyCreate... thus no setlocale( LC_ALL, "" ) is needed...

#if SUPPORT_LOCALIZATIONS
    if ( TY_(tidyGetLanguageSetByUser)() == no ) {
        /*\
         *  Is. #770 #783 - this should be 'last' choice, if at all?
         *  TY_(tidySetLanguage)( setlocale( LC_ALL, "") );
         *  That is take every other opportunity to set the 'language'
        \*/
        if( ! TY_(tidySetLanguage)( getenv( "LANGUAGE" ) ) ) {
            if( ! TY_(tidySetLanguage)( getenv( "LC_MESSAGES" ) ) ) {
                if( ! TY_(tidySetLanguage)( getenv( "LANG" ) ) ) {
                    char *current = setlocal( LC_ALL, NULL );
                    if( ! TY_(tidySetLanguage)( current ) ) {
#ifndef NO_LIBTIDY_SETLOCALE    /* optional, maybe overkill, ... */
                        /* no choice but to briefly disturb things */
                        TY_(tidySetLanguage)( setlocale( LC_ALL, "" ) );
                        current = setlocal( LC_ALL, current );
#endif /* NO_LIBTIDY_SETLOCALE */
                    }
                }
            }
        }
    }
#endif /* SUPPORT_LOCALIZATIONS */

This does not address any threading issue, but if neccessary the tidyCreate() could be run in a critical section... or something...

I am not sure the new suggested compile flag, like NO_LIBTIDY_SETLOCALE, is necessary, or would ever have a use case... so not keen on this...

Also this does not yet address any tidy doc changes... nor maybe any potential cross-platform issues...

To repeat, just looking for a non-invasive way to setup the libTidy gettext language to use...

Further feedback very welcome to reach the best solution... thanks...