Opened 5 years ago

Last modified 15 months ago

#14974 new New feature

Add support for translation backends other than gettext

Reported by: marinho Owned by: nobody
Component: Internationalization Version: master
Severity: Normal Keywords:
Cc: ethan.jucovy@…, aurelio@…, ojiidotch@…, kitsunde@…, stas@…, rtnpro Triage Stage: Accepted
Has patch: yes Needs documentation: yes
Needs tests: no Patch needs improvement: yes
Easy pickings: no UI/UX: no

Description

Using gettext has been a problem on many projects. It's weird to translate and is limited to (static) potfiles (nothing convince me that Pootle Server and Rosetta are great ideas, actually they are just quick fixes for a bad implemented tool).

So, I'd like to suggest you to make a contrib app to have all i18n functions.

Not only the existing functions but provide backend system for alternative ways to make translations, to replace gettext for database stored message strings, or dynamic/smart translations.

Would be helpful also if the same application has a way to translate database objects, like django-multilingual (1) and polyglot (2) do.

I can work on it if the idea get approved.

Change History (19)

comment:1 Changed 5 years ago by jezdez

  • Component changed from Uncategorized to Internationalization
  • Needs documentation unset
  • Needs tests unset
  • Patch needs improvement unset
  • Triage Stage changed from Unreviewed to Design decision needed

With all due respect, calling gettext problematic for non-static translation seems a bit off since it clearly wasn't made for dynamic translations. In any case, this requires thorough discussion on the mailing list before any approval.

comment:2 follow-up: Changed 5 years ago by marinho

Of course... so, because it is not for dynamic translations, it's a problem to do it if the only way that the framework offers to do is using gettext.

This is why I believe a backend system would help a lot, because who want to work with non-static translations (and at the level of model objects) can avoid gettext and use database or cache to do it.

I know that gettext is mature and fast, yet a limited sollution.

comment:3 in reply to: ↑ 2 ; follow-up: Changed 5 years ago by jezdez

Replying to marinho:

Of course... so, because it is not for dynamic translations, it's a problem to do it if the only way that the framework offers to do is using gettext.

You're basically trying to justify a new feature by saying the existing feature doesn't cover your use case, which is a moot point in my opinion.

This is why I believe a backend system would help a lot, because who want to work with non-static translations (and at the level of model objects) can avoid gettext and use database or cache to do it.

Right, I don't quite understand what you mean by backend system though. An explanation (on the mailing list) might help to clarify what you mean. The big issue I see here (at least from what I understand) is that content translation and translation of UI elements are completely different stories because the way those content types differ in structure and storage. Adding a backend system implies having a common API, but I don't see any. Can you elaborate?

I know that gettext is mature and fast, yet a limited sollution.

It's not limited but really just not supposed to be used like you use.

comment:4 in reply to: ↑ 3 Changed 5 years ago by marinho

Replying to jezdez:

I didn't revise the text below, please understand I'm doing it as faster as I can (today I'm working).

Ok, I tried to be clear on the mail list, but I'm going to right a fast story with more details below:

You know, most of i18n functions are in:

  1. utils/translation
  2. core/management/commands/compilemessages.py
  3. core/management/commands/makemessages.py
  4. views/i18n.py
  5. conf/locale

What I'm saying is about to change the points 1, 2, 3 and 4.

Keep in my the following structure:

  • contrib/i18n
  • contrib/i18n/init.py
  • contrib/i18n/models.py <--- here we have the model classes for DB storage
  • contrib/i18n/views.py <--- here we have the current "views/i18n.py"
  • contrib/i18n/backends/init.py
  • contrib/i18n/backends/base.py <--- here we have the backend base class
  • contrib/i18n/backends/potfiles.py <--- here we have the backend class for gettext (with most of current "utils/translation" code)
  • contrib/i18n/backends/db.py <--- here we have the backend class for database (using the model class for DB storage, etc)
  • contrib/i18n/management/init.py
  • contrib/i18n/management/commands/init.py
  • contrib/i18n/management/commands/compilemessages.py <--- we move the current to here
  • contrib/i18n/management/commands/makemessages.py <--- we move the current to here

Now a prototype of "contrib/i18n/models.py":

class MessageString(models.Model):
    class Meta:
        unique_together = (
            ('hash','language'),
        )
    hash = models.CharField()
    language = models.CharField()
    message_id = models.TextField() # if necessary
    message_str = models.TextField()

Now a prototype of "contrib/i18n/backends/base.py":

class BaseBackend(object):
    def gettext(self, msg):
        raise NotImplemented

    def ugettext(self, msg):
        raise NotImplemented

    def gettext_noop(self, msg):
        raise NotImplemented

    def get_language(self):
        return self._language # Of course, different

    def makemessages(self):
        raise NotImplemented

    def compilemessages(self):
        raise NotImplemented

    # etc...

Now a prototype of "contrib/i18n/backends/potfiles.py":

class GettextBackend(BaseBackend):
    potfile = 'django.po' # etc...

    def gettext(self, msg):
        return do_translate(message, 'gettext') # Like it does nowadays

    def ugettext(self, msg):
        return do_translate(message, 'ugettext') # Like it does nowadays

    def gettext_noop(self, msg):
        return message

    def makemessages(self):
        # do the same that makemessages.py actually does

    def compilemessages(self):
        # do the same that compilemessages.py actually does

    # etc...

Now a prototype of "contrib/i18n/backends/db.py":

class DatabaseBackend(BaseBackend):
    def get_hash(self, msg):
        return hashlib.sha1(msg).hexdigest()

    def gettext(self, msg):
        try:
            obj = MessageString.objects.get(
                    language = self.get_language(),
                    hash = self.get_hash(msg),
                    )
            return obj.message_str
        except MessageString.DoesNotExist:
            return msg

    def ugettext(self, msg):
        return do_translate(message, 'ugettext') # Like it does nowadays

    def gettext_noop(self, msg):
        return message

    def makemessages(self):
        # do the same that makemessages.py actually does, but stores using
        # MessageString instead of using the gettext tools

    def compilemessages(self):
        # do nothing, probably

    # etc...

Now the setting to set the current backend (settings.py):

I18N_BACKEND = 'django.contrib.i18n.backends.DatabaseBackend'

So, the template tags and ugettext functions must be changed to use the current
backend instead of just use gettext.

comment:5 Changed 5 years ago by marinho

Just fixing: ignore the method "ugettext" from "DatabaseBackend". It should be like the method "gettext" from the same class.

comment:6 Changed 4 years ago by anonymous

  • Severity set to Normal
  • Type set to New feature

comment:7 Changed 4 years ago by aaugustin

  • Easy pickings unset
  • Triage Stage changed from Design decision needed to Accepted
  • UI/UX unset

The original report proposes three main changes:

  • 1) a django.contrib.i18n application — which is basically a large refactoring
  • 2) support for translation backends (other file formats / string databases)
  • 3) support for dynamic translations (translation of database content)

Refactorings are complicated, because they introduce backwards incompatibilities that only disappear after three releases — that's the length of our deprecation path. They're also very hard to review. We tend to accept them only when they're a pre-requisite to add interesting features. Anyway, the amount of effort needed for the refactoring is dwarfed by the the amount needed for the features, so it doesn't really matter at this step.

Features (2) and (3) are rather large — as a wild guess, defining, developing, testing and documenting them for Django would take a few hundred hours.

Reading the thread you started on this topic, I understand that your proposal is mostly about (2). As you say in your last message, what we need now is a more detailed proposal, explaining the consequences of the change:

  • which backends would you include initially?
  • how much would this affect performance?
  • how would this affect distribution of translations for Django itself, and for pluggable third-party applications?
  • etc.

A proof of concept in a fork on GitHub would be interesting too.

To sum up:

  • I agree that pluggable translation backends and dynamic translations are interesting ideas. They're separate ideas, so let's limit this ticket to translation backends. I'm accepting the ticket with this scope.
  • Please don't focus too much on refactoring — it adds more noise than value. Deal with the hard part (APIs, proof of concept) first. Try to be as much backwards compatible as possible. Then, if necessary, we can decide to refactor things.

comment:8 Changed 4 years ago by aaugustin

#16960 is a duplicate that suggests introducing an XMB/XTB backend.

comment:9 Changed 4 years ago by aaugustin

  • Summary changed from Contrib application for i18n functions to Add support for translation backends other than gettext

comment:10 Changed 4 years ago by ejucovy

  • Cc ethan.jucovy@… added

comment:11 Changed 3 years ago by aurelio

  • Cc aurelio@… added

comment:12 Changed 3 years ago by ojii

  • Cc ojiidotch@… added

I'm not sure point 3 in comment:7 should really be part of this ticket. Dynamic translations are a completely different thing than static translations. Wouldn't it make more sense to move the discussion etc about dynamic translations in Django to another ticket and keep this one purely for supporting additional backends for static translations?

comment:13 Changed 3 years ago by ojii

Ticket for dynamic translations (aka: model translations #6460)

comment:14 Changed 3 years ago by ramiro

#19480 is a proposal related to this ticket and has a pull request at https://github.com/django/django/pull/590

comment:15 Changed 3 years ago by kitsunde

  • Cc kitsunde@… added

comment:16 Changed 2 years ago by stas

  • Cc stas@… added

comment:17 Changed 23 months ago by rtnpro

Is any one working on this feature request?

comment:18 Changed 23 months ago by rtnpro

  • Cc rtnpro added

comment:19 Changed 15 months ago by timo

  • Has patch set
  • Needs documentation set
  • Patch needs improvement set

The PR above doesn't merge cleanly and lacks documentation.

Note: See TracTickets for help on using tickets.
Back to Top