Opened 14 years ago

Last modified 8 months ago

#14974 new New feature

Add support for translation backends other than gettext

Reported by: Marinho Brandão Owned by: nobody
Component: Internationalization Version: dev
Severity: Normal Keywords:
Cc: ethan.jucovy@…, aurelio@…, ojiidotch@…, kitsunde@…, stas@…, rtnpro, Andi Albrecht, Ülgen Sarıkavak Triage Stage: Accepted
Has patch: yes Needs documentation: yes
Needs tests: no Patch needs improvement: yes
Easy pickings: no UI/UX: no

Description

Using gettext has been a problem on many projects. It's weird to translate and is limited to (static) potfiles (nothing convince me that Pootle Server and Rosetta are great ideas, actually they are just quick fixes for a bad implemented tool).

So, I'd like to suggest you to make a contrib app to have all i18n functions.

Not only the existing functions but provide backend system for alternative ways to make translations, to replace gettext for database stored message strings, or dynamic/smart translations.

Would be helpful also if the same application has a way to translate database objects, like django-multilingual (1) and polyglot (2) do.

I can work on it if the idea get approved.

Change History (21)

comment:1 by Jannis Leidel, 14 years ago

Component: UncategorizedInternationalization
Triage Stage: UnreviewedDesign decision needed

With all due respect, calling gettext problematic for non-static translation seems a bit off since it clearly wasn't made for dynamic translations. In any case, this requires thorough discussion on the mailing list before any approval.

comment:2 by Marinho Brandão, 14 years ago

Of course... so, because it is not for dynamic translations, it's a problem to do it if the only way that the framework offers to do is using gettext.

This is why I believe a backend system would help a lot, because who want to work with non-static translations (and at the level of model objects) can avoid gettext and use database or cache to do it.

I know that gettext is mature and fast, yet a limited sollution.

in reply to:  2 ; comment:3 by Jannis Leidel, 14 years ago

Replying to marinho:

Of course... so, because it is not for dynamic translations, it's a problem to do it if the only way that the framework offers to do is using gettext.

You're basically trying to justify a new feature by saying the existing feature doesn't cover your use case, which is a moot point in my opinion.

This is why I believe a backend system would help a lot, because who want to work with non-static translations (and at the level of model objects) can avoid gettext and use database or cache to do it.

Right, I don't quite understand what you mean by backend system though. An explanation (on the mailing list) might help to clarify what you mean. The big issue I see here (at least from what I understand) is that content translation and translation of UI elements are completely different stories because the way those content types differ in structure and storage. Adding a backend system implies having a common API, but I don't see any. Can you elaborate?

I know that gettext is mature and fast, yet a limited sollution.

It's not limited but really just not supposed to be used like you use.

in reply to:  3 comment:4 by Marinho Brandão, 14 years ago

Replying to jezdez:

I didn't revise the text below, please understand I'm doing it as faster as I can (today I'm working).

Ok, I tried to be clear on the mail list, but I'm going to right a fast story with more details below:

You know, most of i18n functions are in:

  1. utils/translation
  2. core/management/commands/compilemessages.py
  3. core/management/commands/makemessages.py
  4. views/i18n.py
  5. conf/locale

What I'm saying is about to change the points 1, 2, 3 and 4.

Keep in my the following structure:

  • contrib/i18n
  • contrib/i18n/init.py
  • contrib/i18n/models.py <--- here we have the model classes for DB storage
  • contrib/i18n/views.py <--- here we have the current "views/i18n.py"
  • contrib/i18n/backends/init.py
  • contrib/i18n/backends/base.py <--- here we have the backend base class
  • contrib/i18n/backends/potfiles.py <--- here we have the backend class for gettext (with most of current "utils/translation" code)
  • contrib/i18n/backends/db.py <--- here we have the backend class for database (using the model class for DB storage, etc)
  • contrib/i18n/management/init.py
  • contrib/i18n/management/commands/init.py
  • contrib/i18n/management/commands/compilemessages.py <--- we move the current to here
  • contrib/i18n/management/commands/makemessages.py <--- we move the current to here

Now a prototype of "contrib/i18n/models.py":

class MessageString(models.Model):
    class Meta:
        unique_together = (
            ('hash','language'),
        )
    hash = models.CharField()
    language = models.CharField()
    message_id = models.TextField() # if necessary
    message_str = models.TextField()

Now a prototype of "contrib/i18n/backends/base.py":

class BaseBackend(object):
    def gettext(self, msg):
        raise NotImplemented

    def ugettext(self, msg):
        raise NotImplemented

    def gettext_noop(self, msg):
        raise NotImplemented

    def get_language(self):
        return self._language # Of course, different

    def makemessages(self):
        raise NotImplemented

    def compilemessages(self):
        raise NotImplemented

    # etc...

Now a prototype of "contrib/i18n/backends/potfiles.py":

class GettextBackend(BaseBackend):
    potfile = 'django.po' # etc...

    def gettext(self, msg):
        return do_translate(message, 'gettext') # Like it does nowadays

    def ugettext(self, msg):
        return do_translate(message, 'ugettext') # Like it does nowadays

    def gettext_noop(self, msg):
        return message

    def makemessages(self):
        # do the same that makemessages.py actually does

    def compilemessages(self):
        # do the same that compilemessages.py actually does

    # etc...

Now a prototype of "contrib/i18n/backends/db.py":

class DatabaseBackend(BaseBackend):
    def get_hash(self, msg):
        return hashlib.sha1(msg).hexdigest()

    def gettext(self, msg):
        try:
            obj = MessageString.objects.get(
                    language = self.get_language(),
                    hash = self.get_hash(msg),
                    )
            return obj.message_str
        except MessageString.DoesNotExist:
            return msg

    def ugettext(self, msg):
        return do_translate(message, 'ugettext') # Like it does nowadays

    def gettext_noop(self, msg):
        return message

    def makemessages(self):
        # do the same that makemessages.py actually does, but stores using
        # MessageString instead of using the gettext tools

    def compilemessages(self):
        # do nothing, probably

    # etc...

Now the setting to set the current backend (settings.py):

I18N_BACKEND = 'django.contrib.i18n.backends.DatabaseBackend'

So, the template tags and ugettext functions must be changed to use the current
backend instead of just use gettext.

comment:5 by Marinho Brandão, 14 years ago

Just fixing: ignore the method "ugettext" from "DatabaseBackend". It should be like the method "gettext" from the same class.

comment:6 by anonymous, 14 years ago

Severity: Normal
Type: New feature

comment:7 by Aymeric Augustin, 13 years ago

Easy pickings: unset
Triage Stage: Design decision neededAccepted
UI/UX: unset

The original report proposes three main changes:

  • 1) a django.contrib.i18n application — which is basically a large refactoring
  • 2) support for translation backends (other file formats / string databases)
  • 3) support for dynamic translations (translation of database content)

Refactorings are complicated, because they introduce backwards incompatibilities that only disappear after three releases — that's the length of our deprecation path. They're also very hard to review. We tend to accept them only when they're a pre-requisite to add interesting features. Anyway, the amount of effort needed for the refactoring is dwarfed by the the amount needed for the features, so it doesn't really matter at this step.

Features (2) and (3) are rather large — as a wild guess, defining, developing, testing and documenting them for Django would take a few hundred hours.

Reading the thread you started on this topic, I understand that your proposal is mostly about (2). As you say in your last message, what we need now is a more detailed proposal, explaining the consequences of the change:

  • which backends would you include initially?
  • how much would this affect performance?
  • how would this affect distribution of translations for Django itself, and for pluggable third-party applications?
  • etc.

A proof of concept in a fork on GitHub would be interesting too.

To sum up:

  • I agree that pluggable translation backends and dynamic translations are interesting ideas. They're separate ideas, so let's limit this ticket to translation backends. I'm accepting the ticket with this scope.
  • Please don't focus too much on refactoring — it adds more noise than value. Deal with the hard part (APIs, proof of concept) first. Try to be as much backwards compatible as possible. Then, if necessary, we can decide to refactor things.

comment:8 by Aymeric Augustin, 13 years ago

#16960 is a duplicate that suggests introducing an XMB/XTB backend.

comment:9 by Aymeric Augustin, 13 years ago

Summary: Contrib application for i18n functionsAdd support for translation backends other than gettext

comment:10 by Ethan Jucovy, 13 years ago

Cc: ethan.jucovy@… added

comment:11 by Aurelio Tinio, 12 years ago

Cc: aurelio@… added

comment:12 by Jonas Obrist, 12 years ago

Cc: ojiidotch@… added

I'm not sure point 3 in comment:7 should really be part of this ticket. Dynamic translations are a completely different thing than static translations. Wouldn't it make more sense to move the discussion etc about dynamic translations in Django to another ticket and keep this one purely for supporting additional backends for static translations?

comment:13 by Jonas Obrist, 12 years ago

Ticket for dynamic translations (aka: model translations #6460)

comment:14 by Ramiro Morales, 12 years ago

#19480 is a proposal related to this ticket and has a pull request at https://github.com/django/django/pull/590

comment:15 by Kit Sunde, 12 years ago

Cc: kitsunde@… added

comment:16 by Staś Małolepszy, 11 years ago

Cc: stas@… added

comment:17 by rtnpro, 11 years ago

Is any one working on this feature request?

comment:18 by rtnpro, 11 years ago

Cc: rtnpro added

comment:19 by Tim Graham, 10 years ago

Has patch: set
Needs documentation: set
Patch needs improvement: set

The PR above doesn't merge cleanly and lacks documentation.

comment:20 by Andi Albrecht, 7 years ago

Cc: Andi Albrecht added

comment:21 by Ülgen Sarıkavak, 8 months ago

Cc: Ülgen Sarıkavak added
Note: See TracTickets for help on using tickets.
Back to Top