Make CharField form field prohibit null characters

[CM Lubinski]

Input with null characters cause Django to crash when saving data.

Reproduction recipe:

0. Setup Django 1.11.1, Postgres (I don't think the version matters) using Python 3.5 (though this may apply to any 3.x).
1. Create an admin account & login.
2. Navigate to /admin/auth/group/add/
3. Using the JavaScript console, execute

   $('#id_name').val("\x00something")
   

4. Submit the form. See the error

Result will be similar to:

dev-api_1           | Internal Server Error: /admin/auth/group/add/
dev-api_1           | Traceback (most recent call last):
dev-api_1           |   File "/usr/src/app/.venv-dev/lib/python3.5/site-packages/django/core/handlers/exception.py", line 41, in inner
dev-api_1           |     response = get_response(request)
dev-api_1           |   File "/usr/src/app/.venv-dev/lib/python3.5/site-packages/django/core/handlers/base.py", line 187, in _get_response
dev-api_1           |     response = self.process_exception_by_middleware(e, request)
dev-api_1           |   File "/usr/src/app/.venv-dev/lib/python3.5/site-packages/django/core/handlers/base.py", line 185, in _get_response
dev-api_1           |     response = wrapped_callback(request, *callback_args, **callback_kwargs)
dev-api_1           |   File "/usr/src/app/.venv-dev/lib/python3.5/site-packages/django/contrib/admin/options.py", line 551, in wrapper
dev-api_1           |     return self.admin_site.admin_view(view)(*args, **kwargs)
dev-api_1           |   File "/usr/src/app/.venv-dev/lib/python3.5/site-packages/django/utils/decorators.py", line 149, in _wrapped_view
dev-api_1           |     response = view_func(request, *args, **kwargs)
dev-api_1           |   File "/usr/src/app/.venv-dev/lib/python3.5/site-packages/django/views/decorators/cache.py", line 57, in _wrapped_view_func
dev-api_1           |     response = view_func(request, *args, **kwargs)
dev-api_1           |   File "/usr/src/app/.venv-dev/lib/python3.5/site-packages/django/contrib/admin/sites.py", line 224, in inner
dev-api_1           |     return view(request, *args, **kwargs)
dev-api_1           |   File "/usr/src/app/.venv-dev/lib/python3.5/site-packages/django/contrib/admin/options.py", line 1508, in add_view
dev-api_1           |     return self.changeform_view(request, None, form_url, extra_context)
dev-api_1           |   File "/usr/src/app/.venv-dev/lib/python3.5/site-packages/django/utils/decorators.py", line 67, in _wrapper
dev-api_1           |     return bound_func(*args, **kwargs)
dev-api_1           |   File "/usr/src/app/.venv-dev/lib/python3.5/site-packages/django/utils/decorators.py", line 149, in _wrapped_view
dev-api_1           |     response = view_func(request, *args, **kwargs)
dev-api_1           |   File "/usr/src/app/.venv-dev/lib/python3.5/site-packages/django/utils/decorators.py", line 63, in bound_func
dev-api_1           |     return func.__get__(self, type(self))(*args2, **kwargs2)
dev-api_1           |   File "/usr/src/app/.venv-dev/lib/python3.5/site-packages/django/contrib/admin/options.py", line 1408, in changeform_view
dev-api_1           |     return self._changeform_view(request, object_id, form_url, extra_context)
dev-api_1           |   File "/usr/src/app/.venv-dev/lib/python3.5/site-packages/django/contrib/admin/options.py", line 1440, in _changeform_view
dev-api_1           |     if form.is_valid():
dev-api_1           |   File "/usr/src/app/.venv-dev/lib/python3.5/site-packages/django/forms/forms.py", line 183, in is_valid
dev-api_1           |     return self.is_bound and not self.errors
dev-api_1           |   File "/usr/src/app/.venv-dev/lib/python3.5/site-packages/django/forms/forms.py", line 175, in errors
dev-api_1           |     self.full_clean()
dev-api_1           |   File "/usr/src/app/.venv-dev/lib/python3.5/site-packages/django/forms/forms.py", line 386, in full_clean
dev-api_1           |     self._post_clean()
dev-api_1           |   File "/usr/src/app/.venv-dev/lib/python3.5/site-packages/django/forms/models.py", line 402, in _post_clean
dev-api_1           |     self.validate_unique()
dev-api_1           |   File "/usr/src/app/.venv-dev/lib/python3.5/site-packages/django/forms/models.py", line 411, in validate_unique
dev-api_1           |     self.instance.validate_unique(exclude=exclude)
dev-api_1           |   File "/usr/src/app/.venv-dev/lib/python3.5/site-packages/django/db/models/base.py", line 1032, in validate_unique
dev-api_1           |     errors = self._perform_unique_checks(unique_checks)
dev-api_1           |   File "/usr/src/app/.venv-dev/lib/python3.5/site-packages/django/db/models/base.py", line 1129, in _perform_unique_checks
dev-api_1           |     if qs.exists():
dev-api_1           |   File "/usr/src/app/.venv-dev/lib/python3.5/site-packages/django/db/models/query.py", line 668, in exists
dev-api_1           |     return self.query.has_results(using=self.db)
dev-api_1           |   File "/usr/src/app/.venv-dev/lib/python3.5/site-packages/django/db/models/sql/query.py", line 517, in has_results
dev-api_1           |     return compiler.has_results()
dev-api_1           |   File "/usr/src/app/.venv-dev/lib/python3.5/site-packages/django/db/models/sql/compiler.py", line 845, in has_results
dev-api_1           |     return bool(self.execute_sql(SINGLE))
dev-api_1           |   File "/usr/src/app/.venv-dev/lib/python3.5/site-packages/django/db/models/sql/compiler.py", line 886, in execute_sql
dev-api_1           |     raise original_exception
dev-api_1           |   File "/usr/src/app/.venv-dev/lib/python3.5/site-packages/django/db/models/sql/compiler.py", line 876, in execute_sql
dev-api_1           |     cursor.execute(sql, params)
dev-api_1           |   File "/usr/src/app/.venv-dev/lib/python3.5/site-packages/debug_toolbar/panels/sql/tracking.py", line 165, in execute
dev-api_1           |     return self._record(self.cursor.execute, sql, params)
dev-api_1           |   File "/usr/src/app/.venv-dev/lib/python3.5/site-packages/debug_toolbar/panels/sql/tracking.py", line 107, in _record
dev-api_1           |     return method(sql, params)
dev-api_1           |   File "/usr/src/app/.venv-dev/lib/python3.5/site-packages/django/db/backends/utils.py", line 80, in execute
dev-api_1           |     return super(CursorDebugWrapper, self).execute(sql, params)
dev-api_1           |   File "/usr/src/app/.venv-dev/lib/python3.5/site-packages/django/db/backends/utils.py", line 65, in execute
dev-api_1           |     return self.cursor.execute(sql, params)
dev-api_1           | ValueError: A string literal cannot contain NUL (0x00) characters.

[Tim Graham]

This exception was ​introduced in psycopg2 2.7+. With psycopg2 < 2.7, there is no exception and null bytes are silently truncated by PostgreSQL. Other databases that I tested (SQLite, MySQL, Oracle) allow saving null bytes. This creates possible cross-database compatibility problems when moving data from those databases to PostgreSQL, e.g.#28117. I've tentatively closed that ticket as a duplicate but will reopen it if the solution for this ticket doesn't address it.

I propose having CharField and TextField strip null bytes from the value either a) only on PostgreSQL or b) on all databases. I raised ​a django-developers thread to ask for feedback.

[Tim Graham]

Please don't work on this issue until there's a consensus about the solution.

[Tim Graham]

The consensus on the mailing list is to make CharField and TextField prohibit null characters.

[Rajesh Veeranki]

So the way to achieve this is by adding a default validator to CharField and TextField and raise a validation error. Am I right?

[Tim Graham]

That seems reasonable.

[Alejandro Zamora Fonseca]

Added default null characters validation to CharField and TextField in
order to avoid raising errors on database layer. Added test for both
fields. Tested on PostgreSQL, MySQL, and SQLite backends.

[Alejandro Zamora Fonseca]

Linked PR against master branch ​PR

[Alejandro Zamora Fonseca]

Maybe only adding the null char validator to CharField would be enough?
Because TextField is converted to CharField when Model to Field translation occurs.

[Tim Graham]

Correct, there's isn't a TextField form field.

[Alejandro Zamora Fonseca]

I added documentation and unit tests for the validator.

[Tim Graham <timograham@…>]

In 90d7b91:

Fixed #28201 -- Added ProhibitNullCharactersValidator and used it on CharField form field.

[Vlada Macek]

We just ran into this on 1.11 LTS just as the original author. May I ask whether it is usual to backport fixes like this?

Thank you.

[Tim Graham]

Per our ​supported versions policy, 1.11 is only receiving data loss and security fixes.