Opened 11 years ago

Closed 7 years ago

#20572 closed Bug (wontfix)

Localized PostgreSQL error messages break loaddata

Reported by: gerard.henry@… Owned by: nobody
Component: Core (Management commands) Version: 1.5
Severity: Normal Keywords:
Cc: Triage Stage: Accepted
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

Doing loaddata:

python manage.py loaddata -v 3 --traceback /tmp/mc1.json 
...
Traceback (most recent call last):
  File "/Users/me/python/virtualenv/django15x/lib/python2.7/site-packages/django/core/management/base.py", line 222, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "/Users/me/python/virtualenv/django15x/lib/python2.7/site-packages/django/core/management/base.py", line 255, in execute
    output = self.handle(*args, **options)
  File "/Users/me/python/virtualenv/django15x/lib/python2.7/site-packages/django/core/management/commands/loaddata.py", line 226, in handle
    e.args = ("Problem installing fixtures: %s" % e,)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 126: ordinal not in range(128)

changing line 226 in loaddata.py (replace %s by %r), i see the error:

IntegrityError: Problem installing fixtures: IntegrityError('insert or update on table "eventsmgmt_mycalendar" violates foreign key constraint "eventsmgmt_mycalendar_created_by_id_fkey"\nD\xc3\x89TAIL : Key (created_by_id)=(46) is not present in table "auth_user".\n',)

in the terminal, the locale is:

$localeLANG="fr_FR.UTF-8"
LC_COLLATE="fr_FR.UTF-8"
LC_CTYPE="fr_FR.UTF-8"
LC_MESSAGES="fr_FR.UTF-8"
LC_MONETARY="fr_FR.UTF-8"
LC_NUMERIC="fr_FR.UTF-8"
LC_TIME="fr_FR.UTF-8"
LC_ALL=

the workaround is to change the locale to en_US:

$ export LANG=en_US.UTF-8

and it works

Change History (9)

comment:1 by Claude Paroz, 11 years ago

Component: UncategorizedCore (Management commands)
Triage Stage: UnreviewedAccepted

comment:2 by Claude Paroz, 11 years ago

Note that I can currently reproduce the error with fixtures_regress.tests.TestFixtures.test_loaddata_raises_error_when_fixture_has_invalid_foreign_key

comment:3 by Claude Paroz, 11 years ago

Also note that this is a Python2-only issue.

Some possible solutions:

  • using %r instead %s. Drawback: the resulting message is not quite the same:
    e.args = ("Problem installing fixtures: %s" % e,)
    
    (u"Problem installing fixtures: The row in table 'fixtures_regress_book' with primary key '1' has an invalid
    foreign key: fixtures_regress_book.author_id contains a value '3' that does not have a corresponding value in
    fixtures_regress_person.id.",)
    
    e.args = ("Problem installing fixtures: %r" % e,)
    
    (u'Problem installing fixtures: IntegrityError(u"The row in table \'fixtures_regress_book\' with primary key
    \'1\' has an invalid foreign key: fixtures_regress_book.author_id contains a value \'3\' that does not have a
    corresponding value in fixtures_regress_person.id.",)',)
    
    And we should still track this pattern all over Django code.
  • using force_text in DatabaseErrorWrapper to convert message to unicode.
    Drawback: non-ascii unicode args for exceptions are not well supported, notably in unittest2 assertRaisesRegexp where str(exc_value) will raise a UnicodeEncodeError.
  • using force_text(s, 'ascii', errors='ignore') in DatabaseErrorWrapper to strip out any non-ascii char from the exception message.
    Drawback: the error message will be altered.

Opinions or other solutions welcome.

comment:4 by Claude Paroz, 11 years ago

Summary: locale fr_FR.UTF-8 breaks loaddataLocalized PostgreSQL error messages break loaddata

comment:5 by EmilStenstrom, 10 years ago

There are lots of other cases where printing error messages in ascii could lead to problems. I'm not sure if there are more bugs covering this, so I'll add a small description here:

I have a model who's unicode() method returns:

<MailComment: \n\nK\xc3\xb6r jobbet nu och det verkar bli fel...>

When loading my fixture I get: ValueError('Cannot assign [unicode representation of MailComment here]') which get propagated to fixture loading and makes that crash with:

UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 32: ordinal not in range(128)

Using %r instead of %s seems reasonable, as it doesn't remove any of the offending characters (they are often the reason for the exception in the first place), and it will likely work well with all sorts of environments.

comment:6 by Claude Paroz, 10 years ago

#22882 reports a similar issue with the flush command.

comment:7 by Tim Graham, 10 years ago

Can we add from __future__ import unicode_literals to dumpdata.py? This strategy seems to fixed the issue for flush.

comment:8 by Claude Paroz, 8 years ago

#25719 was a duplicate.

comment:9 by Claude Paroz, 7 years ago

Resolution: wontfix
Status: newclosed

Django master is now Python 3 only, therefore I'll close this ticket.

Note: See TracTickets for help on using tickets.
Back to Top