psycopg2 throws an "can't adapt" error on ugettext_lazy translated strings

[Mitar]

I have a custom model field (based on a CharField) for which I am setting a default value to a string I want for it to be translated for each user based on her language settings.

So in definition of my custom field I do something like:

class CustomField(fields.CharField):
  def __init__(self, *args, **kwargs):
    ...
    kwargs.setdefault('default', ugettext_lazy('Default value'))
    ...

    super(fields.CustomField, self).__init__(*args, **kwargs)

  def get_internal_type(self):
    return "CharField"

Sadly this fails when creating a default (empty) object with get_or_create. psycopg2 raises "can't adapt" error where it is visible that proxy object was passed as an argument.

[Mitar]

The workaround is to use small lambda wrapper (default can be also a callable) with non-lazy ugettext. Or to use a lambda with force_unicode if you are getting lazily translated string from some other code somewhere else (this above is just a simple example).

[Horst Gutmann]

Could you please provide some more details regarding version numbers and such? I tried to reproduce it using psycopg2 '2.2.2 (dt dec ext pq3)' on PostgreSQL 8.4.4 with following field::

   header = CustomField(max_length=255,
                verbose_name=_('Header'),
                help_text=_("An optional header for this content"))

and it worked for me.

[Horst Gutmann]

... using Django 1.2.1 (sorry, hit the wrong button).

[Mitar]

I use Django 1.2.1, psycopg2 2.2.1, Debian 5.0.5. PostgreSQL 8.3.11.

A example session in Python shell:

>>> p = models.UserProfile.objects.get(pk=7)
>>> p.city = 'foobar'
>>> p.save()
>>> from django.utils.translation import ugettext_lazy 
>>> p.city = ugettext_lazy('foobar')
>>> p.save()
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/usr/lib/pymodules/python2.5/django/db/models/base.py", line 435, in save
    self.save_base(using=using, force_insert=force_insert, force_update=force_update)
  File "/usr/lib/pymodules/python2.5/django/db/models/base.py", line 501, in save_base
    rows = manager.using(using).filter(pk=pk_val)._update(values)
  File "/usr/lib/pymodules/python2.5/django/db/models/query.py", line 491, in _update
    return query.get_compiler(self.db).execute_sql(None)
  File "/usr/lib/pymodules/python2.5/django/db/models/sql/compiler.py", line 861, in execute_sql
    cursor = super(SQLUpdateCompiler, self).execute_sql(result_type)
  File "/usr/lib/pymodules/python2.5/django/db/models/sql/compiler.py", line 727, in execute_sql
    cursor.execute(sql, params)
  File "/usr/lib/pymodules/python2.5/django/db/backends/util.py", line 15, in execute
    return self.cursor.execute(sql, params)
  File "/usr/lib/pymodules/python2.5/django/db/backends/postgresql_psycopg2/base.py", line 44, in execute
    return self.cursor.execute(query, args)
DatabaseError: can't adapt type '__proxy__'

The same happens when using a translated string for default value for a field.

[Mitar]

I attached fields​ I am using to #5446. city field above is CityField.

[ch0wn]

Same for lxml.etree._ElementStringResult even though it's coercable to bytestrings or unicode. Wrapping the element in unicode() fixes it.
Important to note is, that this works without manual casting with the mysql driver.

[Matthias Kestenholz]

I'm not sure whether lazy strings should be automatically converted to unicode on save(). I'd say this constitutes silent loss of information.

Btw, this does not work on sqlite either: It fails with an InterfaceError: InterfaceError: Error binding parameter 5 - probably unsupported type.

I'd close this ticket as "wontfix".

[Mitar]

Replying to mk:

I'm not sure whether lazy strings should be automatically converted to unicode on save(). I'd say this constitutes silent loss of information.

Ha? Lazy strings are just that - lazy. They should behave exactly the same as strict strings, only their evaluation is postponed. This behavior should be transparent, semantics should be the same. This should only have influence on performance. So you can change strict into lazy and vice versa without worrying that something will explode.

[Matthias Kestenholz]

No, semantics aren't the same. A lazy i18n string depends on the context and this is lost when the string is stored in a DB. I'll rather be warned (with an explosion, if it has to be) when I try doing something stupid. "Silent data loss" is a strong description for what's happening if a lazy string is translated upon entering storage, but I'd still argue that's what is happening here. And silent data loss is bad(tm).

I'll mark this design decision needed, hopefully someone from core will weigh in on this issue.

[Mitar]

Hm, but in this case this is not a silent data loss because it is stored for concrete user in his context. It is only fixation of his settings at the time of storage. Please read initial description. I have a custom model field with default value which should be populated based on context and that value should be fixed and stored in the database. It is the same thing as I would display to the user values in his language context and then store that value. Would we say that there was a loss of data?

[Paul Oswald]

I expect lazy evaluation to mean: in the context of the user doing the saving. I would expect the postgres backend to evaluate and translate at the time of saving rather than throw an error in an attempt to save me from myself. I believe this is the way the other database back-ends do it, no? It's also not like this is an intentional exception being raised... I think it is because it is trying to write the function name into the SQL which is definitely not right.

The docs say this:

Always use lazy translations in Django models. Field names and table names should be marked for translation (otherwise, they won't be translated in the admin interface).

While they are talking about things like help text and labels here (not values that get stored in the db) it's the reason that people hit this issue: they define _() in their models.py to be lazy evaluation.

The point of lazy evaluation is to delay the string translation until the correct locale is in effect rather then when the model is first loaded up. In the case of saving data like field defaults, I think "lazy" means when the data is saved into the database. You're calling it silent data loss but I think the common expectation would be for lazy to mean "when being serialized" wether that be to html, json, or a database table. For example, I have a multi-lingual user signup where I create defaults when the user signs up. I want those defaults to be saved in the user's language when I save the model. The only time I would want translation to happen to a string being serialized to the db would be on a choices field type and in that case it wouldn't be marked as a translation string.

In short, I don't think anybody expects a lazy string to be translated *after* the data is retrieved from the database again (there's a template tag for that if you wanted it), and I don't think anyone expects it to blow up which is what it is doing. I think this issue is valid. It would be good to write a test case for it. I wonder if it worked in the past versions and is a regression.

[Aymeric Augustin]

Change UI/UX from NULL to False.

[Aymeric Augustin]

Change Easy pickings from NULL to False.

[Eric Holscher]

Just ran into this on updating permissions with:

class Meta:
    permissions = (
        ('view_version', _('View Version')),
    )

[Eric Holscher]

Workaround was to remove the translatable string for View Version, and then run syncdb, and then add it back :/

[Tim Graham]

Couldn't reproduce these issues as far back as Django 1.4.