Opened 7 years ago

Closed 6 years ago

#28117 closed Cleanup/optimization (fixed)

Add a helpful message to loaddata when psycopg2 raises ValueError due to NUL characters in data

Reported by: Nicolas Kuttler Owned by: Srinivas Reddy Thatiparthy
Component: Core (Management commands) Version: 1.11
Severity: Normal Keywords:
Cc: Triage Stage: Ready for checkin
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

I have a site that currently uses mysql. The admin.logentry table contains object_repr values from spam submissions like

\u589f0\u0005\u0001\u0000\u0000: Nice article

While migrating the site to postgres importing fixtures created with dumpdata raises a ValueError originating in (I think) https://github.com/psycopg/psycopg2/blob/2_7_1/psycopg/utils.c#L58 and not handled in https://github.com/django/django/blob/stable/1.11.x/django/core/management/commands/loaddata.py#L181

One possible fix, to give users a more informative error message, would be to catch ValueErrors as well on that line.

Traceback (most recent call last):
  File "./src/manage.py", line 8, in <module>
    execute_from_command_line(sys.argv)
  File "/srv/www/project/staging/20170422-113847/virtualenv/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 367, in execute_from_command_line
    utility.execute()
  File "/srv/www/project/staging/20170422-113847/virtualenv/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 359, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/srv/www/project/staging/20170422-113847/virtualenv/local/lib/python2.7/site-packages/django/core/management/base.py", line 294, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/srv/www/project/staging/20170422-113847/virtualenv/local/lib/python2.7/site-packages/django/core/management/base.py", line 345, in execute
    output = self.handle(*args, **options)
  File "/srv/www/project/staging/20170422-113847/virtualenv/local/lib/python2.7/site-packages/django/core/management/commands/loaddata.py", line 64, in handle
    self.loaddata(fixture_labels)
  File "/srv/www/project/staging/20170422-113847/virtualenv/local/lib/python2.7/site-packages/django/core/management/commands/loaddata.py", line 104, in loaddata
    self.load_label(fixture_label)
  File "/srv/www/project/staging/20170422-113847/virtualenv/local/lib/python2.7/site-packages/django/core/management/commands/loaddata.py", line 167, in load_label
    obj.save(using=self.using)
  File "/srv/www/project/staging/20170422-113847/virtualenv/local/lib/python2.7/site-packages/django/core/serializers/base.py", line 201, in save
    models.Model.save_base(self.object, using=using, raw=True, **kwargs)
  File "/srv/www/project/staging/20170422-113847/virtualenv/local/lib/python2.7/site-packages/django/db/models/base.py", line 824, in save_base
    updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)
  File "/srv/www/project/staging/20170422-113847/virtualenv/local/lib/python2.7/site-packages/django/db/models/base.py", line 889, in _save_table
    forced_update)
  File "/srv/www/project/staging/20170422-113847/virtualenv/local/lib/python2.7/site-packages/django/db/models/base.py", line 939, in _do_update
    return filtered._update(values) > 0
  File "/srv/www/project/staging/20170422-113847/virtualenv/local/lib/python2.7/site-packages/django/db/models/query.py", line 654, in _update
    return query.get_compiler(self.db).execute_sql(CURSOR)
  File "/srv/www/project/staging/20170422-113847/virtualenv/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 1148, in execute_sql
    cursor = super(SQLUpdateCompiler, self).execute_sql(result_type)
  File "/srv/www/project/staging/20170422-113847/virtualenv/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 835, in execute_sql
    cursor.execute(sql, params)
  File "/srv/www/project/staging/20170422-113847/virtualenv/local/lib/python2.7/site-packages/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
ValueError: Problem installing fixture '/srv/www/project/data.json': A string literal cannot contain NUL (0x00) characters.

Change History (5)

comment:1 by Tim Graham, 7 years ago

Resolution: duplicate
Status: newclosed

As discussed in #28201, the solution might be to have CharField and TextField remove null bytes from their values. I'm tentatively closed this as a duplicate but will reopen if the solution to that ticket doesn't address this.

comment:2 by Tim Graham, 7 years ago

Resolution: duplicate
Status: closednew
Triage Stage: UnreviewedAccepted

The django-developers discussion yielded a consensus to go with a fix in the direction of what's suggested in the ticket description.

comment:3 by Srinivas Reddy Thatiparthy, 6 years ago

Owner: changed from nobody to Srinivas Reddy Thatiparthy
Status: newassigned

comment:4 by Tim Graham, 6 years ago

Has patch: set
Summary: loaddata raises ValueError with psycopg2 backendAdd a helpful message to loaddata when psycopg2 raises ValueError due to NUL characters in data
Triage Stage: AcceptedReady for checkin

comment:5 by Tim Graham <timograham@…>, 6 years ago

Resolution: fixed
Status: assignedclosed

In 58ec55b1:

Fixed #28117 -- Added a helpful message in loaddata when psycopg2 can't load a fixture due to NUL characters.

Note: See TracTickets for help on using tickets.
Back to Top