Storing big strings with special chars fails when using PostgreSQL with SQL_ASCII encoding

[JoseTomasTocino]

If a project uses a PostgreSQL database with SQL_ASCII encoding, a model with a CharField may raise a DataError: value too long error when storing strings with special characters.

Supposing a simple model like this:

class TestModel(models.Model):
    title = models.CharField(max_length=5)

And a PostgreSQL database called testing with SQL_ASCII encoding, that can be created using:

CREATE DATABASE testing WITH TEMPLATE = template0 ENCODING = 'SQL_ASCII';

If you try to add an instance of the model with a title with special characters it fails, even being under the length limit:

TestModel.objects.create(title="ñññ")

Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/home/vagrant/ENV/local/lib/python2.7/site-packages/django/db/models/manager.py", line 127, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/home/vagrant/ENV/local/lib/python2.7/site-packages/django/db/models/query.py", line 348, in create
    obj.save(force_insert=True, using=self.db)
  File "/home/vagrant/ENV/local/lib/python2.7/site-packages/django/db/models/base.py", line 734, in save
    force_update=force_update, update_fields=update_fields)
  File "/home/vagrant/ENV/local/lib/python2.7/site-packages/django/db/models/base.py", line 762, in save_base
    updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)
  File "/home/vagrant/ENV/local/lib/python2.7/site-packages/django/db/models/base.py", line 846, in _save_table
    result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)
  File "/home/vagrant/ENV/local/lib/python2.7/site-packages/django/db/models/base.py", line 885, in _do_insert
    using=using, raw=raw)
  File "/home/vagrant/ENV/local/lib/python2.7/site-packages/django/db/models/manager.py", line 127, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/home/vagrant/ENV/local/lib/python2.7/site-packages/django/db/models/query.py", line 920, in _insert
    return query.get_compiler(using=using).execute_sql(return_id)
  File "/home/vagrant/ENV/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 974, in execute_sql
    cursor.execute(sql, params)
  File "/home/vagrant/ENV/local/lib/python2.7/site-packages/django/db/backends/utils.py", line 79, in execute
    return super(CursorDebugWrapper, self).execute(sql, params)
  File "/home/vagrant/ENV/local/lib/python2.7/site-packages/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
  File "/home/vagrant/ENV/local/lib/python2.7/site-packages/django/db/utils.py", line 97, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/home/vagrant/ENV/local/lib/python2.7/site-packages/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
DataError: value too long for type character varying(5)

This is particularly relevant in the admin app, where the LogEntry model has a object_repr field with max_length=200 that's prone to fail whenever a user does something on a model with a potentially big repr string (which is exactly how I came across this bug).

IMHO this has to do with how non-ascii characters are represented in a ASCII database.

[Tim Graham]

​As documented, "Django assumes that all databases use UTF-8 encoding. Using other encodings may result in unexpected behavior such as “value too long” errors from your database for data that is valid in Django."