SafeUnicode is not type.UnicodeType in _mysql.connection.escape

[Alexander]

Hello.

cleaned_data['content'] is type django.utils.safestring.SafeUnicode

This get error when recording in a database.

Because, the function can only identify unicode or any other as string.

In _mysql.c:

static PyObject * _escape_item( PyObject *item, PyObject *d){
    PyObject *quoted=NULL, *itemtype, *itemconv;
    if (!(itemtype = PyObject_Type(item))) goto error;
    itemconv = PyObject_GetItem(d, itemtype);

Temporary solution to the problem: unicode(cleaned_data['content'])

[Malcolm Tredinnick]

Fixed description formatting.

[Malcolm Tredinnick]

Please don't mark tickets as invalid as an anonymous user (and specify a reason whenever you do mark a ticket as invalid).

[Remco Wendt]

Can you please post what you are trying to do here? Are you inserting data into mysql by hand?

We tested with a model using a mysql database and giving the CharField of SafeUnicode, this works.

Marking as invalid for now until we can properly reproduce this error.

[djbenji@…]

I think this is a MySQLdb problem. If you set a Django model variable to a SafeUnicode object (for example, when storing the result of a template render) and save it, MySQLdb treats it as a string and tries to encode it to ascii before executing the query. You have to cast to unicode to get round it.

[olau@…]

I'm going to reopen this as I just ran into it too and wasted an hour or so debugging it. I concur with Ben, to reproduce it create a model like this:

from django.db import models

class Invoice(models.Model):
    text = models.TextField()

Then instantiate an object, save it, set the text field like this

    from django.utils.safestring import SafeUnicode
    o = Invoice()
    o.save()
    o.text = SafeUnicode(u'\xc6')
    o.save() # dies here

Then on MySQL (not on Sqlite) I get UnicodeEncodeError, 'ascii' codec can't encode character u'\xc6'.

The traceback says that crash happens here in "[...]MySQLdb/cursors.py in execute":

 147. charset = db.character_set_name()
 148. if isinstance(query, unicode):
 149.    query = query.encode(charset)
 150. if args is not None:
 151.    query = query % db.literal(args)

If you remove the call to SafeUnicode, everything is fine. I'm on Debian testing (Python 2.5, MySQL-python-1.2.2).

As Ben says, this happens when you store the result of a template render and that result has non-ASCII characters in it, that was what I was doing too. I put unicode(...) around it and everything is fine.

[Jacob]

Looks like this is a bug in MySQLdb: SafeUnicode is just a subclass of basestring so it ought to work anywhere a "real" unicode object does.

[Thomas Steinacher <tom@…>]

I created a MySQLdb bug report with a testcase that illustrates the problem.

​http://sourceforge.net/tracker/index.php?func=detail&aid=2001850&group_id=22307&atid=374932

[Jarek Zgoda]

Psycopg2 raises ProgrammingError ("cann't adapt") with the same testcase. If there will be any possibility to report issues at initd.org, i'll file a ticket. Now my C skills are too basic to hack psycopg2 code.

[Remco Wendt]

Hey zgoda,

Which version of Django are you using? (trunk? which revision?)

I remember fixing this issue a while ago by registering an adapter in the psycopg2 backend (in base.py)

psycopg2.extensions.register_adapter(SafeUnicode, psycopg2.extensions.QuotedString)

[Jarek Zgoda]

Revision: 7732

[Remco Wendt]

Zgoda,

I've not been able to reproduce the issue with the example posted by olau@…. Using django-trunk rev 7537 and psycopg2 2.0.6 (dec dt ext pq3) and python 2.5.1. Are other people able to reproduce it?

[Simon Law]

Patch to let MySQLdb know about SafeString and SafeUnicode

[Simon Law]

Yes, we can reproduce.

Here is one way to work around MySQLdb's brokenness. Patch attached.

[Malcolm Tredinnick]

(In [9467]) Fixed #6052 -- Worked around a bug in MySQLdb with regards to handling
SafeUnicode (handle SafeString similarly, just to be safe). Based on a patch
from sfllaw.

[Malcolm Tredinnick]

(In [9469]) [1.0.X] Fixed #6052 -- Worked around a bug in MySQLdb with regards to handling
SafeUnicode (handle SafeString similarly, just to be safe). Based on a patch
from sfllaw.

Backport of r9467 from trunk.

[Tim Graham <timograham@…>]

In 5b95d42:

Refs #23919 -- Removed a MySQLdb workaround (refs #6052) for Python 2.