Opened 16 years ago

Closed 15 years ago

Last modified 7 years ago

#6052 closed (fixed)

SafeUnicode is not type.UnicodeType in _mysql.connection.escape

Reported by: Alexander Owned by: nobody
Component: Database layer (models, ORM) Version: dev
Severity: Keywords: UnicodeBranch UnicodeEncodeError mysqldb SafeUnicode
Cc: djbenji@…, olau@…, tom@…, jarek.zgoda@…, eric@…, martin@…, simon@… Triage Stage: Unreviewed
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description (last modified by Malcolm Tredinnick)

Hello.

cleaned_data['content'] is type django.utils.safestring.SafeUnicode

This get error when recording in a database.

Because, the function can only identify unicode or any other as string.

In _mysql.c:

static PyObject * _escape_item( PyObject *item, PyObject *d){
    PyObject *quoted=NULL, *itemtype, *itemconv;
    if (!(itemtype = PyObject_Type(item))) goto error;
    itemconv = PyObject_GetItem(d, itemtype);

Temporary solution to the problem: unicode(cleaned_data['content'])

Attachments (1)

safe-string-mysql (1.1 KB ) - added by Simon Law 15 years ago.
Patch to let MySQLdb know about SafeString and SafeUnicode

Download all attachments as: .zip

Change History (19)

comment:1 by Malcolm Tredinnick, 16 years ago

Description: modified (diff)

Fixed description formatting.

comment:2 by anonymous, 16 years ago

Resolution: invalid
Status: newclosed

comment:3 by Malcolm Tredinnick, 16 years ago

Resolution: invalid
Status: closedreopened

Please don't mark tickets as invalid as an anonymous user (and specify a reason whenever you do mark a ticket as invalid).

comment:4 by Remco Wendt, 16 years ago

Component: Django Web siteDatabase wrapper
Resolution: invalid
Status: reopenedclosed

Can you please post what you are trying to do here? Are you inserting data into mysql by hand?

We tested with a model using a mysql database and giving the CharField of SafeUnicode, this works.

Marking as invalid for now until we can properly reproduce this error.

comment:5 by djbenji@…, 16 years ago

I think this is a MySQLdb problem. If you set a Django model variable to a SafeUnicode object (for example, when storing the result of a template render) and save it, MySQLdb treats it as a string and tries to encode it to ascii before executing the query. You have to cast to unicode to get round it.

comment:6 by Ben <djbenji@…>, 16 years ago

Cc: djbenji@… added

comment:7 by olau@…, 16 years ago

Cc: olau@… added
Resolution: invalid
Status: closedreopened

I'm going to reopen this as I just ran into it too and wasted an hour or so debugging it. I concur with Ben, to reproduce it create a model like this:

from django.db import models

class Invoice(models.Model):
    text = models.TextField()

Then instantiate an object, save it, set the text field like this

    from django.utils.safestring import SafeUnicode
    o = Invoice()
    o.save()
    o.text = SafeUnicode(u'\xc6')
    o.save() # dies here

Then on MySQL (not on Sqlite) I get UnicodeEncodeError, 'ascii' codec can't encode character u'\xc6'.

The traceback says that crash happens here in "[...]MySQLdb/cursors.py in execute":

 147. charset = db.character_set_name()
 148. if isinstance(query, unicode):
 149.    query = query.encode(charset)
 150. if args is not None:
 151.    query = query % db.literal(args)

If you remove the call to SafeUnicode, everything is fine. I'm on Debian testing (Python 2.5, MySQL-python-1.2.2).

As Ben says, this happens when you store the result of a template render and that result has non-ASCII characters in it, that was what I was doing too. I put unicode(...) around it and everything is fine.

comment:8 by Jacob, 16 years ago

Resolution: invalid
Status: reopenedclosed

Looks like this is a bug in MySQLdb: SafeUnicode is just a subclass of basestring so it ought to work anywhere a "real" unicode object does.

comment:9 by Thomas Steinacher <tom@…>, 16 years ago

Cc: tom@… added

I created a MySQLdb bug report with a testcase that illustrates the problem.

http://sourceforge.net/tracker/index.php?func=detail&aid=2001850&group_id=22307&atid=374932

comment:10 by Jarek Zgoda, 16 years ago

Cc: jarek.zgoda@… added

Psycopg2 raises ProgrammingError ("cann't adapt") with the same testcase. If there will be any possibility to report issues at initd.org, i'll file a ticket. Now my C skills are too basic to hack psycopg2 code.

comment:11 by Remco Wendt, 16 years ago

Hey zgoda,

Which version of Django are you using? (trunk? which revision?)

I remember fixing this issue a while ago by registering an adapter in the psycopg2 backend (in base.py)

psycopg2.extensions.register_adapter(SafeUnicode, psycopg2.extensions.QuotedString)

comment:12 by Jarek Zgoda, 16 years ago

Revision: 7732

comment:13 by Remco Wendt, 16 years ago

Zgoda,

I've not been able to reproduce the issue with the example posted by olau@…. Using django-trunk rev 7537 and psycopg2 2.0.6 (dec dt ext pq3) and python 2.5.1. Are other people able to reproduce it?

by Simon Law, 15 years ago

Attachment: safe-string-mysql added

Patch to let MySQLdb know about SafeString and SafeUnicode

comment:14 by Simon Law, 15 years ago

Has patch: set
Resolution: invalid
Status: closedreopened

Yes, we can reproduce.

Here is one way to work around MySQLdb's brokenness. Patch attached.

comment:15 by Simon Law, 15 years ago

Cc: eric@… martin@… simon@… added

comment:16 by Malcolm Tredinnick, 15 years ago

Resolution: fixed
Status: reopenedclosed

(In [9467]) Fixed #6052 -- Worked around a bug in MySQLdb with regards to handling
SafeUnicode (handle SafeString similarly, just to be safe). Based on a patch
from sfllaw.

comment:17 by Malcolm Tredinnick, 15 years ago

(In [9469]) [1.0.X] Fixed #6052 -- Worked around a bug in MySQLdb with regards to handling
SafeUnicode (handle SafeString similarly, just to be safe). Based on a patch
from sfllaw.

Backport of r9467 from trunk.

comment:18 by Tim Graham <timograham@…>, 7 years ago

In 5b95d42:

Refs #23919 -- Removed a MySQLdb workaround (refs #6052) for Python 2.

Note: See TracTickets for help on using tickets.
Back to Top