Opened 5 years ago

Closed 4 years ago

#28117 closed Cleanup/optimization (fixed)

Add a helpful message to loaddata when psycopg2 raises ValueError due to NUL characters in data

Reported by: Nicolas Kuttler Owned by: Srinivas Reddy Thatiparthy
Component: Core (Management commands) Version: 1.11
Severity: Normal Keywords:
Cc: Triage Stage: Ready for checkin
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no


I have a site that currently uses mysql. The admin.logentry table contains object_repr values from spam submissions like

\u589f0\u0005\u0001\u0000\u0000: Nice article

While migrating the site to postgres importing fixtures created with dumpdata raises a ValueError originating in (I think) and not handled in

One possible fix, to give users a more informative error message, would be to catch ValueErrors as well on that line.

Traceback (most recent call last):
  File "./src/", line 8, in <module>
  File "/srv/www/project/staging/20170422-113847/virtualenv/local/lib/python2.7/site-packages/django/core/management/", line 367, in execute_from_command_line
  File "/srv/www/project/staging/20170422-113847/virtualenv/local/lib/python2.7/site-packages/django/core/management/", line 359, in execute
  File "/srv/www/project/staging/20170422-113847/virtualenv/local/lib/python2.7/site-packages/django/core/management/", line 294, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/srv/www/project/staging/20170422-113847/virtualenv/local/lib/python2.7/site-packages/django/core/management/", line 345, in execute
    output = self.handle(*args, **options)
  File "/srv/www/project/staging/20170422-113847/virtualenv/local/lib/python2.7/site-packages/django/core/management/commands/", line 64, in handle
  File "/srv/www/project/staging/20170422-113847/virtualenv/local/lib/python2.7/site-packages/django/core/management/commands/", line 104, in loaddata
  File "/srv/www/project/staging/20170422-113847/virtualenv/local/lib/python2.7/site-packages/django/core/management/commands/", line 167, in load_label
  File "/srv/www/project/staging/20170422-113847/virtualenv/local/lib/python2.7/site-packages/django/core/serializers/", line 201, in save
    models.Model.save_base(self.object, using=using, raw=True, **kwargs)
  File "/srv/www/project/staging/20170422-113847/virtualenv/local/lib/python2.7/site-packages/django/db/models/", line 824, in save_base
    updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)
  File "/srv/www/project/staging/20170422-113847/virtualenv/local/lib/python2.7/site-packages/django/db/models/", line 889, in _save_table
  File "/srv/www/project/staging/20170422-113847/virtualenv/local/lib/python2.7/site-packages/django/db/models/", line 939, in _do_update
    return filtered._update(values) > 0
  File "/srv/www/project/staging/20170422-113847/virtualenv/local/lib/python2.7/site-packages/django/db/models/", line 654, in _update
    return query.get_compiler(self.db).execute_sql(CURSOR)
  File "/srv/www/project/staging/20170422-113847/virtualenv/local/lib/python2.7/site-packages/django/db/models/sql/", line 1148, in execute_sql
    cursor = super(SQLUpdateCompiler, self).execute_sql(result_type)
  File "/srv/www/project/staging/20170422-113847/virtualenv/local/lib/python2.7/site-packages/django/db/models/sql/", line 835, in execute_sql
    cursor.execute(sql, params)
  File "/srv/www/project/staging/20170422-113847/virtualenv/local/lib/python2.7/site-packages/django/db/backends/", line 64, in execute
    return self.cursor.execute(sql, params)
ValueError: Problem installing fixture '/srv/www/project/data.json': A string literal cannot contain NUL (0x00) characters.

Change History (5)

comment:1 Changed 5 years ago by Tim Graham

Resolution: duplicate
Status: newclosed

As discussed in #28201, the solution might be to have CharField and TextField remove null bytes from their values. I'm tentatively closed this as a duplicate but will reopen if the solution to that ticket doesn't address this.

comment:2 Changed 5 years ago by Tim Graham

Resolution: duplicate
Status: closednew
Triage Stage: UnreviewedAccepted

The django-developers discussion yielded a consensus to go with a fix in the direction of what's suggested in the ticket description.

comment:3 Changed 4 years ago by Srinivas Reddy Thatiparthy

Owner: changed from nobody to Srinivas Reddy Thatiparthy
Status: newassigned

comment:4 Changed 4 years ago by Tim Graham

Has patch: set
Summary: loaddata raises ValueError with psycopg2 backendAdd a helpful message to loaddata when psycopg2 raises ValueError due to NUL characters in data
Triage Stage: AcceptedReady for checkin

comment:5 Changed 4 years ago by Tim Graham <timograham@…>

Resolution: fixed
Status: assignedclosed

In 58ec55b1:

Fixed #28117 -- Added a helpful message in loaddata when psycopg2 can't load a fixture due to NUL characters.

Note: See TracTickets for help on using tickets.
Back to Top