Opened 7 years ago

Closed 7 years ago

Last modified 7 years ago

#6052 closed (fixed)

SafeUnicode is not type.UnicodeType in _mysql.connection.escape

Reported by: Alexander Owned by: nobody
Component: Database layer (models, ORM) Version: master
Severity: Keywords: UnicodeBranch UnicodeEncodeError mysqldb SafeUnicode
Cc: djbenji@…, olau@…, tom@…, jarek.zgoda@…, eric@…, martin@…, simon@… Triage Stage: Unreviewed
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: UI/UX:

Description (last modified by mtredinnick)

Hello.

cleaned_data['content'] is type django.utils.safestring.SafeUnicode

This get error when recording in a database.

Because, the function can only identify unicode or any other as string.

In _mysql.c:

static PyObject * _escape_item( PyObject *item, PyObject *d){
    PyObject *quoted=NULL, *itemtype, *itemconv;
    if (!(itemtype = PyObject_Type(item))) goto error;
    itemconv = PyObject_GetItem(d, itemtype);

Temporary solution to the problem: unicode(cleaned_data['content'])

Attachments (1)

safe-string-mysql (1.1 KB) - added by sfllaw 7 years ago.
Patch to let MySQLdb know about SafeString and SafeUnicode

Download all attachments as: .zip

Change History (18)

comment:1 Changed 7 years ago by mtredinnick

  • Description modified (diff)
  • Needs documentation unset
  • Needs tests unset
  • Patch needs improvement unset

Fixed description formatting.

comment:2 Changed 7 years ago by anonymous

  • Resolution set to invalid
  • Status changed from new to closed

comment:3 Changed 7 years ago by mtredinnick

  • Resolution invalid deleted
  • Status changed from closed to reopened

Please don't mark tickets as invalid as an anonymous user (and specify a reason whenever you do mark a ticket as invalid).

comment:4 Changed 7 years ago by shanx

  • Component changed from Django Web site to Database wrapper
  • Resolution set to invalid
  • Status changed from reopened to closed

Can you please post what you are trying to do here? Are you inserting data into mysql by hand?

We tested with a model using a mysql database and giving the CharField of SafeUnicode, this works.

Marking as invalid for now until we can properly reproduce this error.

comment:5 Changed 7 years ago by djbenji@…

I think this is a MySQLdb problem. If you set a Django model variable to a SafeUnicode object (for example, when storing the result of a template render) and save it, MySQLdb treats it as a string and tries to encode it to ascii before executing the query. You have to cast to unicode to get round it.

comment:6 Changed 7 years ago by Ben <djbenji@…>

  • Cc djbenji@… added

comment:7 Changed 7 years ago by olau@…

  • Cc olau@… added
  • Resolution invalid deleted
  • Status changed from closed to reopened

I'm going to reopen this as I just ran into it too and wasted an hour or so debugging it. I concur with Ben, to reproduce it create a model like this:

from django.db import models

class Invoice(models.Model):
    text = models.TextField()

Then instantiate an object, save it, set the text field like this

    from django.utils.safestring import SafeUnicode
    o = Invoice()
    o.save()
    o.text = SafeUnicode(u'\xc6')
    o.save() # dies here

Then on MySQL (not on Sqlite) I get UnicodeEncodeError, 'ascii' codec can't encode character u'\xc6'.

The traceback says that crash happens here in "[...]MySQLdb/cursors.py in execute":

 147. charset = db.character_set_name()
 148. if isinstance(query, unicode):
 149.    query = query.encode(charset)
 150. if args is not None:
 151.    query = query % db.literal(args)

If you remove the call to SafeUnicode, everything is fine. I'm on Debian testing (Python 2.5, MySQL-python-1.2.2).

As Ben says, this happens when you store the result of a template render and that result has non-ASCII characters in it, that was what I was doing too. I put unicode(...) around it and everything is fine.

comment:8 Changed 7 years ago by jacob

  • Resolution set to invalid
  • Status changed from reopened to closed

Looks like this is a bug in MySQLdb: SafeUnicode is just a subclass of basestring so it ought to work anywhere a "real" unicode object does.

comment:9 Changed 7 years ago by Thomas Steinacher <tom@…>

  • Cc tom@… added

I created a MySQLdb bug report with a testcase that illustrates the problem.

http://sourceforge.net/tracker/index.php?func=detail&aid=2001850&group_id=22307&atid=374932

comment:10 Changed 7 years ago by zgoda

  • Cc jarek.zgoda@… added

Psycopg2 raises ProgrammingError ("cann't adapt") with the same testcase. If there will be any possibility to report issues at initd.org, i'll file a ticket. Now my C skills are too basic to hack psycopg2 code.

comment:11 Changed 7 years ago by shanx

Hey zgoda,

Which version of Django are you using? (trunk? which revision?)

I remember fixing this issue a while ago by registering an adapter in the psycopg2 backend (in base.py)

psycopg2.extensions.register_adapter(SafeUnicode, psycopg2.extensions.QuotedString)

comment:12 Changed 7 years ago by zgoda

Revision: 7732

comment:13 Changed 7 years ago by shanx

Zgoda,

I've not been able to reproduce the issue with the example posted by olau@…. Using django-trunk rev 7537 and psycopg2 2.0.6 (dec dt ext pq3) and python 2.5.1. Are other people able to reproduce it?

Changed 7 years ago by sfllaw

Patch to let MySQLdb know about SafeString and SafeUnicode

comment:14 Changed 7 years ago by sfllaw

  • Has patch set
  • Resolution invalid deleted
  • Status changed from closed to reopened

Yes, we can reproduce.

Here is one way to work around MySQLdb's brokenness. Patch attached.

comment:15 Changed 7 years ago by sfllaw

  • Cc eric@… martin@… simon@… added

comment:16 Changed 7 years ago by mtredinnick

  • Resolution set to fixed
  • Status changed from reopened to closed

(In [9467]) Fixed #6052 -- Worked around a bug in MySQLdb with regards to handling
SafeUnicode (handle SafeString similarly, just to be safe). Based on a patch
from sfllaw.

comment:17 Changed 7 years ago by mtredinnick

(In [9469]) [1.0.X] Fixed #6052 -- Worked around a bug in MySQLdb with regards to handling
SafeUnicode (handle SafeString similarly, just to be safe). Based on a patch
from sfllaw.

Backport of r9467 from trunk.

Note: See TracTickets for help on using tickets.
Back to Top