[i18n] Internationalization and localisation

It would also be nice if you could store the language with a field, so you can do content-negotiation then.

Here my proposition for gettext support:

Added new configuration vars:

INSTALLED_LOCALES - list of all installed locale files
LOCALE_PATH - path to locale/xx/LC_MESSAGES
DEFAULT_LOCALE - default locale
LOCALE_DOMAIN - locale domain name (default django)

I added logging support in modpython.py (temporary).

In template.py add support for gettext in template files (python syntax)

message example:

# from meta.py
self.help_text += ' ' + _('Hold down "Control", or "Command" on a Mac, to select more than one.')

# from admin templates index.html
<td class="x50"><a href="{{ model.admin_url }}add/" class="addlink">_("Add")</a></td>

Changes (diff against R291):

Index: modpython.py
===================================================================
RCS file: /devel/django_src/django/core/handlers/modpython.py,v
retrieving revision 1.4
diff -u -r1.4 modpython.py
--- modpython.py	20 Jul 2005 16:03:49 -0000	1.4
+++ modpython.py	22 Jul 2005 19:50:59 -0000
@@ -1,6 +1,7 @@
 from django.utils import datastructures, httpwrappers
 from pprint import pformat
 import os
+import gettext
 
 # NOTE: do *not* import settings (or any module which eventually imports
 # settings) until after ModPythonHandler has been called; otherwise os.environ
@@ -10,7 +11,83 @@
     def __init__(self, req):
         self._req = req
         self.path = req.uri
+        
+        # setup logger
+        self._setupLogger()
+        # setup translations for request
+        try:
+            from django.conf.settings import INSTALLED_LOCALES
+        except ImportError:
+            INSTALLED_LOCALES = ('en',) # default
+            logger.notice('no installed locales, set to %r', INSTALLED_LOCALES)
+        try:
+            from django.conf.settings import LOCALE_PATH
+        except ImportError:
+            LOCALE_PATH = '/usr/share/locale'# default
+            logger.notice('no locale path, set to %s', LOCALE_PATH)
+        try:
+            from django.conf.settings import DEFAULT_LOCALE
+        except ImportError:
+            DEFAULT_LOCALE = 'sr'# default
+            logger.notice('no default locale, set to %s', DEFAULT_LOCALE)
+        try:
+            from django.conf.settings import LOCALE_DOMAIN
+        except ImportError:
+            LOCALE_DOMAIN = 'django'# default
+            logger.notice('no locale domain, set to %s', LOCALE_DOMAIN)
 
+        self.locale = {}
+        for locale in INSTALLED_LOCALES:
+            try:
+                self.locale[locale] = gettext.translation(LOCALE_DOMAIN, LOCALE_PATH, languages=[locale])
+                logger.debug('loading locale %s from %s', locale, LOCALE_PATH)
+            except IOError:
+                logger.warning('translation for %s not found in %s', locale, LOCALE_PATH)
+                pass
+        #
+        # activate default locale
+        if DEFAULT_LOCALE in self.locale:
+            self.locale[DEFAULT_LOCALE].install()
+            logger.info('selected locale %s', DEFAULT_LOCALE)
+        elif len(self.locale) > 0: # fallback use first available locale
+            self.locale[INSTALLED_LOCALES[0]].install()
+            logger.warning('translation for %s not found in %s, using %r', DEFAULT_LOCALE, LOCALE_PATH, INSTALLED_LOCALES[0])
+        else:
+            from django.core.exceptions import ObjectDoesNotExist
+            raise ObjectDoesNotExist('translation for %s not found in %s' % (LOCALE_DOMAIN, LOCALE_PATH))
+        #
+
+    
+    def _setupLogger(self):
+        """ setup logger, based on ticket #55 """
+        
+        if 'log' in __builtins__:
+            return
+        #
+        
+        from django.conf.settings import SETTINGS_MODULE, DEBUG
+        a = SETTINGS_MODULE.split('.')
+    
+        if 'settings' in a:
+            idx = a.index('settings')
+            
+            if idx > 0:
+                projname = a[idx-1]
+                projmod = __import__(projname , '', '', [''])
+            else:
+                projname = 'django'
+                import django as projmod
+        #
+    
+        if DEBUG:
+            from django.utils.logger import Logger
+            __builtins__['logger'] = Logger(projname, self._req)
+        else:
+            from django.utils.logger import NoLogger
+            __builtins__['logger'] = NoLogger(projname, self._req)
+        #
+    # _setupLogger
+    
     def __repr__(self):
         return '<ModPythonRequest\npath:%s,\nGET:%s,\nPOST:%s,\nCOOKIES:%s,\nMETA:%s,\nuser:%s>' % \
             (self.path, pformat(self.GET), pformat(self.POST), pformat(self.COOKIES),
@@ -229,7 +306,7 @@
                 mail_admins(subject, message, fail_silently=True)
                 return self.get_friendly_error_response(request, conf_module)
         except exceptions.PermissionDenied:
-            return httpwrappers.HttpResponseForbidden('<h1>Permission denied</h1>')
+            return httpwrappers.HttpResponseForbidden('<h1>%s</h1>' % _('Permission denied'))
         except: # Handle everything else, including SuspiciousOperation, etc.
             if DEBUG:
                 return self.get_technical_error_response()

Index: template.py
===================================================================
RCS file: /devel/django_src/django/core/template.py,v
retrieving revision 1.2
diff -u -r1.2 template.py
--- template.py	20 Jul 2005 16:03:49 -0000	1.2
+++ template.py	22 Jul 2005 20:03:33 -0000
@@ -71,12 +71,19 @@
 BLOCK_TAG_END = '%}'
 VARIABLE_TAG_START = '{{'
 VARIABLE_TAG_END = '}}'
+GETTEXT_START = '_("'
+GETTEXT_END = '")'
 
 ALLOWED_VARIABLE_CHARS = 'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789_.'
 
 # match a variable or block tag and capture the entire tag, including start/end delimiters
-tag_re = re.compile('(%s.*?%s|%s.*?%s)' % (re.escape(BLOCK_TAG_START), re.escape(BLOCK_TAG_END),
-                                          re.escape(VARIABLE_TAG_START), re.escape(VARIABLE_TAG_END)))
+#tag_re = re.compile('(%s.*?%s|%s.*?%s)' % (re.escape(BLOCK_TAG_START), re.escape(BLOCK_TAG_END),
+#                                          re.escape(VARIABLE_TAG_START), re.escape(VARIABLE_TAG_END)))
+tag_re = re.compile(r'''(%s.*?%s|%s.*?%s|%s.*?%s)''' % (re.escape(BLOCK_TAG_START), re.escape(BLOCK_TAG_END),
+                                                re.escape(VARIABLE_TAG_START), re.escape(VARIABLE_TAG_END),
+                                                re.escape(GETTEXT_START), re.escape(GETTEXT_END),
+                                                )
+                   )
 
 # global dict used by register_tag; maps custom tags to callback functions
 registered_tags = {}
@@ -189,6 +196,8 @@
         return Token(TOKEN_VAR, token_string[len(VARIABLE_TAG_START):-len(VARIABLE_TAG_END)].strip())
     elif token_string.startswith(BLOCK_TAG_START):
         return Token(TOKEN_BLOCK, token_string[len(BLOCK_TAG_START):-len(BLOCK_TAG_END)].strip())
+    elif token_string.startswith(GETTEXT_START):
+        return Token(TOKEN_TEXT, _(token_string[len(GETTEXT_START):-len(GETTEXT_END)].strip()))
     else:
         return Token(TOKEN_TEXT, token_string)
 

I will also post this on django-devel.

p.s. sorry on my bad,bad,bad english :)

I am willing to do german translations of the gettext messagefile.

One gotcha to be aware of: some functions use "_" as a variable name for "I don't need this variable, let's ignore it". See lines 373 and 374 of django/core/defaulttags.py, for example. If we follow the standard naming and use "_" as the name of the localization function, we'll need to change the occurrences of "_" to something else, like "ignore", wherever it appears.

I'm willing to do french translation. When i18n will be implemented, don't hesitate to contact me :)

I'm willing to do czech translation. Pls, send me an email, when I could start translating.

see http://groups-beta.google.com/group/django-users/browse_thread/thread/6145cf8aaf74dedb/6ae2966e73a8c4d6?fwc=1

I'm willing to do Simple Chinese translation. Pls, send me an email, when I could start translating.

You can't use gettext like this.

The gettext translations will be done at module loading time if you use _, which means that once the module is loaded, we can't change the translations.

Many websites have more than one language active. To me, it certainly isn't worth doing this if it isn't done right, which means that I can serve both English and Spanish content, and have a spanish admin interface for my spanish users and an english one for my english users.

I'll continue this in a thread on django-devel.

This patch doesn't work at all well, please dont' use it

See wiki:InterNationalization for more discussion

i18n_add.diff is with filecache.py from Ticket #437.

Description: http://groups.google.com/group/django-developers/msg/0ddfffa4b6472d1c, http://groups.google.com/group/django-developers/msg/ffad5c326572c860 and http://groups.google.com/group/django-developers/msg/265f0acab3a8397f

Hmm. I beg to differ - due to several places in django that build on predefined strings, at least an easy way to translate the admin interface and validator messages really is needed for 1.0. Not everybody speaks english and in days like this where many contributors to the patches and ideas in this project are from non-english countries I think it's a bit wrong to drop this out of the 1.0 release ...

I for example won't be able to do any serious projects without some translation of text to german. Sure, I can fork the django source and translate those parts myself, but I think it's rather silly to do that when there are actual patches available that would allow to do translation in a useable way for the relevant parts.

Sure, 1.1 is just the next version, so it's not completely given up on - but why give up the momentum when there are _actual_ solutions to the problem available? And it would be one of those parts that Django could do right with regard to other projects: integrate i18n from the start. I know I am allways annoyed at projects that put in i18n only later as an afterthought (think Rails and it's rather non-existing standard solution to i18n) ...

Ups, didn't want to drop Jacops milestone assignement, my browser didn't refresh right, sorry ;-)

To keep the translation as a threading local, the threading.local construct could be used. For 2.3 there is a reimplementation here. It's actually just like a global, only that it is per thread - so on entry to the request, we could set the translation object from the request headers via middleware to some threading.local variable. That way a globally defined _ function (the standard name for the translation function) would be able to pull the current translator from the threading locals without the need of those being passed in as parameters or using the rather clumsy other workarounds. Just use the standard way of translation, but provide your own _ function and dynamically switch translators based on the requests and session data.

I added an alternative patch to the ticket above (it's the i18n-hugo.2.diff - the first one was taken from the wrong directory and Trac doesn't allow overwriting of older files in this installation). This is a bit different from nesh's one, but takes most of his ideas - I made it more compact, though (at least I think). It works as follows:

To translate strings in django, just put _('...') or _("...") around the strings. Those strings will be pulled out by the make-messages.py script in the bin folder. This is assumed to be run from your django/ svn directory and will place all strings it finds into django/conf/locale/ for the given locale (just pass -l de for example to the program - the -d parameter isn't needed, it should allways be django for now).

After creating or updating the message files, edit them as in django/conf/locale/de/LC_MESSAGES/django.po. After editing, just run the compile-messages.py script, this will compile all po files to mo files.

To translate strings in templates, I have created mostly the same i18n template tag as nesh did - but I don't use eval to evaluate, I actually only use the standard gettext function. And the syntax is a bit shorter by using {% i18n _('blah') %}. These files will be pulled out and written to the messages file, too.

The machinery itself works just like with nesh's patch: just add the django.middleware.locale.LocaleMiddlware? to your settings file and the middleware will discover the needed locale automatically. It will check first with the session and use any django_language session variable it finds. Then it will look at a cookie django_language to discover the needed language. If that isn't resolved, it will check the HTTP_ACCEPT_LANGUAGE http header for the language with the highest q= parameter that can be resolved against the django locales.

The current translation object is kept in a thread-global variable - any one thread can only run one request a any one time, as django fully runs from request to response without multiplexing between different requests. So it's safe to store the translation object indexed by the currentThread() object. This should work fine with multiprocess servers and multithreaded servers.

I have added translations to the isAlphaNumeric validator and to the admin index template for you to see how translations look.

One more point: the default translation object is build from the LANGUAGE_CODE setting - so if you set it to "de" for example, the german translations will be used.

Some more notes on my patch: it supports three locations for translation files. All translation files need to be of domain "django" currently. The machinery tries to find a django.mo in the following places: $PROJECT/apps/$APP/locale/$LANG/LC_MESSAGES/django.mo, $PROJECT/locale/$LANG/LC_MESSAGES/django.mo and $PYTHONLIB/django/conf/locale/$LANG/LC_MESSAGES/django.mo - with the last one being the default language files delivered with django. The project and app related language files are for user code. A string to be translated is first looked up in the application translation, then in the project translation and last in the global translation - that way translations can be overridden by user translations. And appliations can deliver their own translation files as well.

The make-messages.py script can be run in the app directory or the project directory as well as the svn tree - if run in one of the other places, it will scan the project or application to produce message files. The same holds true for the compile-messages.py script.

Of course this is rather fresh code, so there might be still bugs in it. If somebody tries it, please drop a note here if you discover some problem.

Hugo, a question about your latest comment: This doesn't assume that the app directory is within the projects directory, does it? If so, it shouldn't...

No, it discovers the app directory as follows: the middleware get's a view function to call. I take the view_func.module module name and check that against the installed app list in the settings: it discovers the one who is a prefix to the module name. Then it does import(appmodulename, {}, {}, views?) and uses the returned module (the actual app module) via it's file attribute to discover where the application itself is stored.

The same holds true for the project path: it just uses the settings module for that.

Another question: Why use {% i18n _('test') %} instead of {% i18n 'test' %}, which is simpler?

because the xgettext utility pulls out all _('...') or _("...") occurrences automatically. If I would use i18n 'xxx', I would have to tell xgettext how to find exactly those strings to pull out and store in the .po file. That's why I deliberately made the template tag work identical to the standard way how you do it in the source code. (nesh did the same in his patch)

I see. Is it possible to use the i18n template tag to look up the contents of a variable, rather than a hard-coded string?

It could be added to the template tag as functionality - yes. But the problem would be that xgettext can only pull out constant strings - it doesn't know what the variable will be filled with at runtime. So you would have to add all possible strings to the .po file yourself - but you yoursefl might not know what possible strings can come up.

Translation should allways be done to constant strings and string interpolation should be used to fill in variables. So we could have something like

{% i18n _('blah %(blubb)s blubber') %}

and then make the i18n tag resolve the %(blubb)s against the context (actually I think something like this was in nesh's patch) if you want interpolated strings that are based on variables. The translation will only translate the 'blah %(blubb)s blubber' string without resolving the interpolation, of course - so the translation needs to have the %(blubb)s part in it, too, so that the translation can be interpolated with the context.

Hi hugo-,

(I haven't read your patch yet, please be patient)

will it be possible to do something like in http://docs.python.org/lib/node337.html (deferred translations)?

Maybe we can standardize the N_() approach, that seems nicer to me?

(In [726]) i18n patch - references #65

Note we now have an i18n branch, which hugo has commit access to.

a first writeup on the i18n branch is on my blog

(In [730]) added more translations for the admin templates so that we can see how translated admin might look - references #65

I added more translations so that interested users can see what it might look like when run. One important thing to note: the locale.LocaleMiddlware? needs to be run after the SessionMiddlware? but _before_ the AuthUserRequired? middleware if you activate it for the admin, because the AuthUserRequired? redirects users out without executing any middlware that comes after it (naturally - it uses a redirect exception to do the redirection) and so the current locale isn't switched to the right locale and an anonymous user would get whatever translation object was installed by some earlier user ...

There should be some way to denote what middlware order isn't correct - something like middlware dependencies or stuff like that - so that things like this don't happen. But that's something different from i18n, more a core problem ;-)

Another current problem is with the caching middlware: since that caches based on URL, it will cache pages from one user and deliver them to another user, if the request doesn't contain either GET or POSt parameters. But the page contents with i18n will depend on cookies, session settings and http headers (the accept header for language selection). This problem isn't new - you already have it with anonymous sessions, as the contents of an anonymous session might modify your page content and so the cache will deliver wrong content to users. So for now, i18n usage (at least the i18n middleware!) will be incompatible with caching, as is already the case with some cases of session data usage ...

static translation (by using just LANGUAGE_CODE and leaving out the i18n middlware) will work fine with caching, as all users get the same translation.

Changed title to reflect that this is now being worked on in a branch; flag an other i18n related tickets with [i18n], please.

The branch now includes in it's docs/ directory a first take at documentation.

I added support for constant i18n strings to the template parser. Before the template parser would accept constant strings with "...." or '....' when the tag did a resolve_variable. I fixed the parser to do that with resolve_variable_with_filters, too. And I changed the code to accept _('...') or _("...") i18n string constants, too. They even do string interpolation to the context, just as the i18n template tag does.

So now the i18n template tag is only needed for things like ngettext or gettext_noop - the standard _() or gettext() stuff can be simpler done by just using {{ _('blah') }} or to put _() around any constant strings you might have in your template tags. This is much shorter and easier to type and looks much nicer - and fit's into the template language better, I think (the i18n tag looks a bit bloated when used for such simple things as simple constants).

Hi hugo-

I'll add a preliminary roughly italian translation of the tags off [764]. See attacched django_it.po. Maybe it can help you for testing purposes...

it's included in my branch as of [767]

(In [768]) i18n: added GET-support to language selection, added serbian translation file and added some convenience stuff to the DjangoContext? - all from a patch by nesh on the list. Thanks nesh! References #65.

Patch for filenames/paths witch may contain spaces for make-messages.py:

Index: django/bin/make-messages.py
===================================================================
--- django/bin/make-messages.py (revision 770)
+++ django/bin/make-messages.py (working copy)
@@ -60,9 +60,9 @@
                 thefile = '%s.py' % file
             if verbose: sys.stdout.write('processing file %s in %s\n' % (file, dirpath))
             if os.path.isfile(lf):
-                cmd = 'xgettext -j -d %s -L Python -p %s %s' % (domain, basedir, os.path.join(dirpath, thefile))
+                cmd = 'xgettext -j -d %s -L Python -p "%s" "%s"' % (domain, basedir, os.path.join(dirpath, thefile))
             else:
-                cmd = 'xgettext -d %s -L Python -p %s %s' % (domain, basedir, os.path.join(dirpath, thefile))
+                cmd = 'xgettext -d %s -L Python -p "%s" "%s"' % (domain, basedir, os.path.join(dirpath, thefile))
             os.system(cmd)
             if thefile != file:
                 os.unlink(os.path.join(dirpath, thefile))

your patch is in SVN as of [777]. thx.

Hi, I added a patch for serbian language. There were some errors, which are fixed now :)

it's in SVN. thx.

added the russian translation and the sr2.patch to SVN. thx.

While digging through the source, looking for possible pitfalls with translations, I encountered the first cases of strings that won't be able to be translated based on request headers. The problem is the reliance of gettext to know the full english text of strings beforehand, so that they can be pulled into the .po files. Of course those strings can be patterned strings that will be interpolated after translation, but still the pattern needs to be known beforehand.

I stumbled over two cases where this doesn't work in Django: the permission verbose names and the admin history records. Actually there is a third one, the per-user messages, but I only seen them in the model and didn't find them in the rest of the source - but they sound like the same problem area.

The problem is, to make translations the string needs to be there as a constant - so that it goes into the .po with it's translation. But if you store texts in the database, you don't want to translate them at that point - you only store them and use the marked string later in the interface to do the translation, as the language to translate to is only fully known on time of request (and storing the french translation of an english message won't help an italian user).

But when storing data in the database and translating later, the full text stored in the database needs to be known for the translation - the interpolation of the string pattern is done at storage time, not at retrieval time. Permission names are constructed dynamically - so they aren't known directly in the source, but are only known as patterns. But those patterns aren't usefull for translation later on, as the strings in the database aren't patterns but full strings.

The only non-solution I can offer up to now is to translate them on storage time to at least the default language. That way projects that don't make use of the i18n middleware (and so don't have request-time language discovery) will get messages and permission names in their native language.

The other option would be to just store them in english and send them out in english, so that users allways see them in english, even if the rest of the site is in their native language.

Some of those strings could be "fake"-translated. That is, you put those strings (knowing beforehand what they will be - like the permissions for predefined models like auth or flat pages or stuff like that) in a source file that is only scanned by the xgettext tool, but never evaluaeted (it's just a container for those strings). Then you could store translations and later on pull those translations from the file. But with every model there are automatically created permission names for those models, and those strings are not known beforehand - except if the user himself set's them up for translation. And this doesn't work for all possible history entries for the same reasons - the user would have to know beforehand what messages might show up and provide fake translation marks for those.

It's just a case where too much dynamicity kicks in and kicks our butts :-)

Hm. I think the real (long-term) solution is to just store the changelists relationally, in two tables. Ie one would be changes, one change details. Then the translation into a natural language is done on output.

The middle ground which is kind of evil, but workable, is to store the changelist as a serialised datastructure, eg pickle, xml or yaml, rather than generated text. But this gets a bit django specific and the database is not independent of the framework. This would not mean changing the database layout, but it would change semantics. In this case too, the translation is done on output.

Of course this bit of the database is not independent of the framework atm because it makes use of typeids. Not sure if this matters.

On the permissions, I've not looked at that bit very much. Is there a good reason to store the verbose names in the database rather than generating these on output?

Users can provide their own permissions for their own models - and those permissions won't necessarily follow the conventions of the django-provided permissions. So yes, there is the need for storing those permission verbose names somewhere. Of course permissions could be twofold: objects with verbose names that are used in the code and permission mappings that are stored in the database (to assign the permissions to users). That way the django permissions could be instances of a class that constructs the verbose name on runtime and the user-provided permissions will just store the verbose name in the instance variables of UserPermission? objects. User-permission verbose names are allways given as constant strings and so are easily translated, it's only the automatic permissions by django that are the problem. But that would change a bit of django code, so it's better left for some later patch.

The message stuff is ugly, because those messages can be anything the user wants. Of course we could store them in a way that only the pattern is stored and the arguments are stored independently, so that the pattern can be simply translated and the string interpolation can be done later at request time. But that would change a lot of code, too - maybe it's something you could address in your new_admin branch, as it is solely bound to the admin stuff :-)

But none of these solutions are things I would opt to implement before discussing them with adrian and jacob.

I added more translation hooks into the source. I restricted myself to the validator error messages and the help texts in the predefined models - those parts are quite stable and don't change too much, so conflicts with trunk should be no problem. I updated all messagefiles with new message ids, so if you provide a translation, you just need to pull it out of the branch and add the missing translation strings.

Another note: please look into the django/conf/global_settings.py file for the LANGUAGES setting. That should include language codes for languages and their native name. For languages that are missing there, please attach a diff with the name. Remember: it should be the native name, so it should be a string in utf-8 encoding and python repr() way of writing (with \xx instead of the direct chars)!

Just a note on translation files: please send in full message files, it's easier to integrate for me. For example the last two patches don't apply correctly any more, because of the admin-moving-merge (and because I forgot to check before doing that merge wether there are translations waiting).

To make stuff easier, I start to remove files from the attachement list to make this cleaner to see what is in and what not.

Hi Georg

I've update the .po to latest modification (actually, [998]).

A little note:

in conf/global_settings.py:42

    ('it', _('Itialian')),

should be:

    ('it', _('Italian')),

thx :)

Yep, I did integrate it and fixed the typo, too.

Sorry, a quick fix of some typos on IT l10n.

Btw, the abbreviated names of some months are missing. Is it intentional?

ok, in SVN. You should check your po file, there are duplicates in your file so that the message compilation throws warnings. The month names aren't missing - have a look at django.utils.dates, only longer month names are abbreviated, the others are full month names - that's the AP style. Of course this poses a problem if some month is short in english but long in some other language.

Little correction: conf/global_settings.py:45

('sr', _('Serbic')),

should be

('sr', _('Serbian')),

Added new Serbian translation (django-sr-2005-11-03.po). Includes Nebojša-s fix for Serbian.

(In [1068]) Merged i18n branch into the trunk! Fixes #65, and perhaps some others. NB: this means that the i18n branch is now obsolete and will be made read-only.

Milestone Version 1.1 deleted