Opened 2 years ago

Last modified 10 months ago

#20572 new Bug

Localized PostgreSQL error messages break loaddata

Reported by: gerard.henry@… Owned by: nobody
Component: Core (Management commands) Version: 1.5
Severity: Normal Keywords:
Cc: Triage Stage: Accepted
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

Doing loaddata:

python manage.py loaddata -v 3 --traceback /tmp/mc1.json 
...
Traceback (most recent call last):
  File "/Users/me/python/virtualenv/django15x/lib/python2.7/site-packages/django/core/management/base.py", line 222, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "/Users/me/python/virtualenv/django15x/lib/python2.7/site-packages/django/core/management/base.py", line 255, in execute
    output = self.handle(*args, **options)
  File "/Users/me/python/virtualenv/django15x/lib/python2.7/site-packages/django/core/management/commands/loaddata.py", line 226, in handle
    e.args = ("Problem installing fixtures: %s" % e,)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 126: ordinal not in range(128)

changing line 226 in loaddata.py (replace %s by %r), i see the error:

IntegrityError: Problem installing fixtures: IntegrityError('insert or update on table "eventsmgmt_mycalendar" violates foreign key constraint "eventsmgmt_mycalendar_created_by_id_fkey"\nD\xc3\x89TAIL : Key (created_by_id)=(46) is not present in table "auth_user".\n',)

in the terminal, the locale is:

$localeLANG="fr_FR.UTF-8"
LC_COLLATE="fr_FR.UTF-8"
LC_CTYPE="fr_FR.UTF-8"
LC_MESSAGES="fr_FR.UTF-8"
LC_MONETARY="fr_FR.UTF-8"
LC_NUMERIC="fr_FR.UTF-8"
LC_TIME="fr_FR.UTF-8"
LC_ALL=

the workaround is to change the locale to en_US:

$ export LANG=en_US.UTF-8

and it works

Change History (7)

comment:1 Changed 2 years ago by claudep

  • Component changed from Uncategorized to Core (Management commands)
  • Needs documentation unset
  • Needs tests unset
  • Patch needs improvement unset
  • Triage Stage changed from Unreviewed to Accepted

comment:2 Changed 2 years ago by claudep

Note that I can currently reproduce the error with fixtures_regress.tests.TestFixtures.test_loaddata_raises_error_when_fixture_has_invalid_foreign_key

comment:3 Changed 2 years ago by claudep

Also note that this is a Python2-only issue.

Some possible solutions:

  • using %r instead %s. Drawback: the resulting message is not quite the same:
    e.args = ("Problem installing fixtures: %s" % e,)
    
    (u"Problem installing fixtures: The row in table 'fixtures_regress_book' with primary key '1' has an invalid
    foreign key: fixtures_regress_book.author_id contains a value '3' that does not have a corresponding value in
    fixtures_regress_person.id.",)
    
    e.args = ("Problem installing fixtures: %r" % e,)
    
    (u'Problem installing fixtures: IntegrityError(u"The row in table \'fixtures_regress_book\' with primary key
    \'1\' has an invalid foreign key: fixtures_regress_book.author_id contains a value \'3\' that does not have a
    corresponding value in fixtures_regress_person.id.",)',)
    
    And we should still track this pattern all over Django code.
  • using force_text in DatabaseErrorWrapper to convert message to unicode.
    Drawback: non-ascii unicode args for exceptions are not well supported, notably in unittest2 assertRaisesRegexp where str(exc_value) will raise a UnicodeEncodeError.
  • using force_text(s, 'ascii', errors='ignore') in DatabaseErrorWrapper to strip out any non-ascii char from the exception message.
    Drawback: the error message will be altered.

Opinions or other solutions welcome.

comment:4 Changed 2 years ago by claudep

  • Summary changed from locale fr_FR.UTF-8 breaks loaddata to Localized PostgreSQL error messages break loaddata

comment:5 Changed 13 months ago by EmilStenstrom

There are lots of other cases where printing error messages in ascii could lead to problems. I'm not sure if there are more bugs covering this, so I'll add a small description here:

I have a model who's unicode() method returns:

<MailComment: \n\nK\xc3\xb6r jobbet nu och det verkar bli fel...>

When loading my fixture I get: ValueError('Cannot assign [unicode representation of MailComment here]') which get propagated to fixture loading and makes that crash with:

UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 32: ordinal not in range(128)

Using %r instead of %s seems reasonable, as it doesn't remove any of the offending characters (they are often the reason for the exception in the first place), and it will likely work well with all sorts of environments.

comment:6 Changed 12 months ago by claudep

#22882 reports a similar issue with the flush command.

comment:7 Changed 10 months ago by timgraham

Can we add from __future__ import unicode_literals to dumpdata.py? This strategy seems to fixed the issue for flush.

Note: See TracTickets for help on using tickets.
Back to Top