Django

Code

Ticket #2765 (closed: fixed)

Opened 2 years ago

Last modified 2 years ago

MySQL with utf8_bin collation for case-sensitive comparison goodness requires MySQLdb 1.2.1

Reported by: dave AT avaragado.org Assigned to: adrian
Milestone: Component: Core framework
Version: SVN Keywords: mysql mysqldb utf8 case
Cc: Triage Stage: Design decision needed
Has patch: 0 Needs documentation: 0
Needs tests: 0 Patch needs improvement: 0

Description

By default, MySQL 4.1 (at least) uses a case-insensitive collation, utf8_general_ci. This means that checks for equality are, annoyingly, case-insensitive. You can set the collation for a database in the CREATE DATABASE command, or you can set it per-server with an option in the MySQL my.cnf file. You can also set it per-column if you want.

Sadly, using a collation like utf8_bin, which gives case-sensitive comparisons, causes unexpected results through the ORM.

To reproduce:

  • Edit your my.cnf file to include the line
    collation-server = utf8_bin
    
  • Restart mysqld

Run "python manage.py test" such that it tests this models.py:

from django.db import models

class Dummy(models.Model):
    """
    >>> d = Dummy(ok='ok', notok='not ok')
    >>> d.save()

    >>> d = Dummy.objects.all()[0]
    >>> d.ok, d.notok
    ('ok', 'not ok')
    """
    ok = models.CharField(maxlength=100)
    notok = models.TextField()

You'll get something like this (irrelevant info removed):

Failed example:
    d.ok, d.notok
Expected:
    ('ok', 'not ok')
Got:
    ('ok', array('c', 'not ok'))

Without the collation-server line in my.cnf, the tests pass. Even without that line the tests fail if the CREATE DATABASE line specifies utf8_bin as the collation (and I imagine other binary collations), but the test framework uses the server defaults for those.

My own scrabbling around discovered that MySQLdb (version 1.2.0, at least) includes a suspicious class in init.py:

def Binary(x):
    from array import array
    return array('c', x)

... but that's as far as my limited knowledge goes - I have no idea how it gets used.

Attachments

Change History

09/19/06 23:35:35 changed by django@poelzi.org

  • status changed from new to closed.
  • resolution set to invalid.

I had this problem, too. Upgrade to the last version of mysqldb and the issue should be gone. Its not a django bug.

09/20/06 05:01:12 changed by dave AT avaragado.org

  • status changed from closed to reopened.
  • resolution deleted.
  • component changed from Core framework to Documentation.
  • summary changed from MySQL with utf8_bin collation for case-sensitive comparison goodness causes TextFields to be returned as arrays to MySQL with utf8_bin collation for case-sensitive comparison goodness requires MySQLdb 1.2.1.

Sigh. Is there any documentation that says the minimum supported version of MySQLdb is 1.2.1? (Or at least, it is if you want to use binary collations.) I can't find any.

If not, then there should be. It might stop others spending hours trying to debug this same problem. Reopening this as a documentation ticket, changing summary.

01/25/07 04:20:08 changed by Simon G. <dev@simon.net.nz>

  • keywords set to mysql mysqldb utf8 case.
  • component changed from Documentation to Core framework.
  • stage changed from Unreviewed to Design decision needed.

Marked as requiring a design decision - do we need to work around this or can we just document that users should MySQLdb > 1.2.1 ?

01/25/07 08:26:14 changed by Michael Radziej <mir@noris.de>

There are other issues with python-mysqldb-1.2.0: threading does not work (#3279). I propose to discourage use of this version.

01/25/07 17:04:35 changed by Marc Fargas <telenieko@telenieko.com>

+1 to require at least 1.2.1 (1.2.1p2 as per #3279)

04/06/07 03:54:05 changed by Michael Radziej <mir@noris.de>

  • status changed from reopened to closed.
  • resolution set to fixed.

This ticket has become obsolete since we now use 1.2.1 (unless you use mysql_old, but it's more or less deprecated).


Add/Change #2765 (MySQL with utf8_bin collation for case-sensitive comparison goodness requires MySQLdb 1.2.1)




Change Properties
Action