[sldev] Issue for tonight's meeting VWR-729

[Alissa Sabre]

Robin,

> Wanted to bring up VWR-729 for tonight's meeting, or more to the point,
> what can we do about locale handing on a longer term basis that is
> possible to manage and works correctly.
> 
> Alissa, any particular comments as you are doing work that is directly
> related to locale handling?
> 
> Robin

Thank you for calling me for this issue, but I'm afraid I was too late
to know it.  I checked the Open Source Meeting page on SL wiki, but I
don't find a transcript for March 13.

The following is my general concerns on the issue.  Please forgive me
if they are all discussed on 13.

# I'm sorry, it's long.

== GTK/GNOME families of programs and locale ==

GTK assumes a good old C/Posix locale model; that is, a program
takes care of one locale at a time, and entire program, including
libraries, should concentrate on the user's locale.

In GTK 1, GTK initialization function invoked setlocale(LC_ALL, "").
GTK 2 introduced an application controllable option to skip the step,
but invoking setlocale as above is still a default.  today, 
a lot of (primarily) Linux oriented tools are build on top of GTK.
Many those tools assume setlocale(LC_ALL, "").  I believe it is a
recommendation by a GNOME team that all application program should
call setlocale(LC_ALL, "") in its initialization.

Modern mozilla for Linux and gstreamer are in a GNOME circle, so it is
natural that they assume setlocale(LC_ALL, "").  Some of you may
remember that I was reently working on VWR-2261.  GTK's IM modules
assume setlocale(LC_ALL, "").  So, it is essential to run GTK event
loop under the user's locale to support GTK IM modules, since some IM
modules add its own callback to GTK skipping application, assuming
those callbacks are executed under the user's locale.

My conclusion at this point is that, as long as we use GTK or other
GNOME friendly tools/libraries in SL viewer, we need to run the viewer
under the user's locale.

== SL viewer policy on locale handling ==

I have never seen a design document of SL viewer regarding locale
handling, but from the source codes I have a feeling that SL viewer's
policy on locale handling is something like:

- Run the viewer code under C locale.

- When necessary, switch to the user's locale, then switch back to the
  C locale as soon as it is done.

- Ignore user's locale preference at the OS level.  Instead, allow
  users to specify SL viewer's own language selection regarding UI.

In the viewer source code, there is a LLLocale class to facilitate the
"when necessary" case mentioned above, but it is broken.

Recently, SL viewer is getting more and more dependent on GTK/GNOME
derived tools.  Calls to some _critical_ functions of those library
are enclosed with locale save/restore codes for the momet.  Failure to
do so leads to viewer crashes.  That's what VWR-729 and similar
filings report.

== Why SL viewer wants C locale, by the way? ==

As far as I can see, SL viewer wants to run under the C locale so that
it can read a text representation of decimal fraction numbers written
in US convention, that is, a period as a decimal point character,
through a standard C library function sscanf.  It can be any locale as
long as its decimal point character is a period.  Some locales, such
as fr_FR, de_DE, it_IT uses a comma as a decimal point character, so
those locales triggered a crash with VWR-729.

I exemined entire SL viewer source code, but I found no other reason
SL viewer misbehaves under a locale other than C.  (Of course I may
have missed something...)

== What can we do in short term? ==

(1) Find all calls to library functions that _play_with_ the locale
    (this is a Linden's terminology that a function calls setlocale
    internally) and recover the viewer's locale after returning the
    function.  I believe this is the way following the _policy_
    discussed above and is what Lindens are doing today.

    Problem of this approach is there is no easy way to know which
    function plays with locale.  We could list all such functions up
    by examining the entire source codes of GTK, mozilla, gstreamer,
    etc. as well as other libraries used by them, but those libraries
    may be updated, and we can't guarantee developers of those
    libraries will not add calls to setlocale in a future versions.

(2) Find all parts of viewer source codes that assumes the C locale
    (that is, call to sscanf to read a decimal fraction number), and
    protect that particular portion of the code with calls to
    setlocale.  I believe this is what Robin is doing in his patch
    attached to VWR-5056.

    I think this is doable under our (i.e., SLDe community including
    Lindens) control.  Another advantage is that we can extend this
    approach to other locale dependent things if a future version of
    SL required ones.

(3) Modify the user's locale request to match SL requirements.  I took
    this approach when I submit my last patch against VWR-2261.  The
    idea is that SL viewer only require LC_NUMERIC locale category to
    be C, and IM modules don't use LC_NUMERIC.

    The primary advantage of this approach is that we can put the
    workaround in the wrapper shell script, so an (experienced) user
    can easily change it when the workaround did a bad thing.  Of
    course, we can do the same thing in an initialiation code of SL
    viewer using setenv()/getenv() if we want.

    The problem of this approach is that this is just a hack.  I don't
    think this is our long term goal.  We need more clean solution.

(4) Replace all locale sensitive functions that needs to be invoked
    under the C locale with some equivalent functions that ignore the
    active locale settings.  I saw opinions for this approach on
    SLDev, although I don't know whether someone is actually working
    on it.

    The problem is that I couldn't find any alternative sscanf that
    ignores the locale handy.  Technically it should be easy to take
    some sscanf source code and modify it to ignore the locale, but it
    will raise some licensing issues.  Writing such a function from
    the scratch is out of my numeric processing skills.  (I only know
    it is a tough job to convert decimal fraction numbers into binary
    fraction numbers with finite bits, maintaining the maximum
    possible accuracy.)

    We could write a wrapper function around the C library sscanf.
    The result can be seen as a special case of the approach (2)
    above.  The problem in the case is that setlocale() is not
    lightning fast operation.  I'm afraid of the overhead, although I
    didn't measure it.

== Long term issues ==

SL viewer provides its own language selection.  (It's on the General
tab in Preferences...)  There are some bug filings on JIRA complaining
the switching of the currently available language switching is
insufficient.  For example, SL viewer always uses a period when
presending a size of an object to the user, regardless of the language
selection in the Preferences...  Some users feel unconfortable with
that.  See VWR-1593 or VWR-721 for examples.

If we were to switch decimal point character based on the user's
language selection on the Preferences..., we need to link it with the
C library's locale, or we need to develop some equivalent functions by
ourselves.  Moreover, they say we can't assume availability of any
locale data other than C on a particular Linux installation.  If it is
true, what we should do if the underlying Linux installation has no
locale data available for any of the SL supported languages?

But, if we were to synchronize with the OS locale settings, what we
should do with locale on Windows and MacOS?
--------------------------------------
Easy + Joy + Powerful = Yahoo! Bookmarks x Toolbar
http://pr.mail.yahoo.co.jp/toolbar/