Add a helpful message to loaddata when psycopg2 raises ValueError due to NUL characters in data

[Nicolas Kuttler]

I have a site that currently uses mysql. The admin.logentry table contains object_repr values from spam submissions like

\u589f0\u0005\u0001\u0000\u0000: Nice article

While migrating the site to postgres importing fixtures created with dumpdata raises a ValueError originating in (I think) ​https://github.com/psycopg/psycopg2/blob/2_7_1/psycopg/utils.c#L58 and not handled in ​https://github.com/django/django/blob/stable/1.11.x/django/core/management/commands/loaddata.py#L181

One possible fix, to give users a more informative error message, would be to catch ValueErrors as well on that line.

Traceback (most recent call last):
  File "./src/manage.py", line 8, in <module>
    execute_from_command_line(sys.argv)
  File "/srv/www/project/staging/20170422-113847/virtualenv/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 367, in execute_from_command_line
    utility.execute()
  File "/srv/www/project/staging/20170422-113847/virtualenv/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 359, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/srv/www/project/staging/20170422-113847/virtualenv/local/lib/python2.7/site-packages/django/core/management/base.py", line 294, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/srv/www/project/staging/20170422-113847/virtualenv/local/lib/python2.7/site-packages/django/core/management/base.py", line 345, in execute
    output = self.handle(*args, **options)
  File "/srv/www/project/staging/20170422-113847/virtualenv/local/lib/python2.7/site-packages/django/core/management/commands/loaddata.py", line 64, in handle
    self.loaddata(fixture_labels)
  File "/srv/www/project/staging/20170422-113847/virtualenv/local/lib/python2.7/site-packages/django/core/management/commands/loaddata.py", line 104, in loaddata
    self.load_label(fixture_label)
  File "/srv/www/project/staging/20170422-113847/virtualenv/local/lib/python2.7/site-packages/django/core/management/commands/loaddata.py", line 167, in load_label
    obj.save(using=self.using)
  File "/srv/www/project/staging/20170422-113847/virtualenv/local/lib/python2.7/site-packages/django/core/serializers/base.py", line 201, in save
    models.Model.save_base(self.object, using=using, raw=True, **kwargs)
  File "/srv/www/project/staging/20170422-113847/virtualenv/local/lib/python2.7/site-packages/django/db/models/base.py", line 824, in save_base
    updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)
  File "/srv/www/project/staging/20170422-113847/virtualenv/local/lib/python2.7/site-packages/django/db/models/base.py", line 889, in _save_table
    forced_update)
  File "/srv/www/project/staging/20170422-113847/virtualenv/local/lib/python2.7/site-packages/django/db/models/base.py", line 939, in _do_update
    return filtered._update(values) > 0
  File "/srv/www/project/staging/20170422-113847/virtualenv/local/lib/python2.7/site-packages/django/db/models/query.py", line 654, in _update
    return query.get_compiler(self.db).execute_sql(CURSOR)
  File "/srv/www/project/staging/20170422-113847/virtualenv/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 1148, in execute_sql
    cursor = super(SQLUpdateCompiler, self).execute_sql(result_type)
  File "/srv/www/project/staging/20170422-113847/virtualenv/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 835, in execute_sql
    cursor.execute(sql, params)
  File "/srv/www/project/staging/20170422-113847/virtualenv/local/lib/python2.7/site-packages/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
ValueError: Problem installing fixture '/srv/www/project/data.json': A string literal cannot contain NUL (0x00) characters.

[Tim Graham]

As discussed in #28201, the solution might be to have CharField and TextField remove null bytes from their values. I'm tentatively closed this as a duplicate but will reopen if the solution to that ticket doesn't address this.

[Tim Graham]

The ​django-developers discussion yielded a consensus to go with a fix in the direction of what's suggested in the ticket description.

[Tim Graham]

​PR

[Tim Graham <timograham@…>]

In 58ec55b1:

Fixed #28117 -- Added a helpful message in loaddata when psycopg2 can't load a fixture due to NUL characters.