id,summary,reporter,owner,description,type,status,component,version,severity,resolution,keywords,cc,stage,has_patch,needs_docs,needs_tests,needs_better_patch,easy,ui_ux 13758,MySQLdb utf8_bin and django causes UnicodeDecodeError,sam.vevang@…,nobody,"Issue: I have a Model with a FileField. When I delete that instances of that model that have unicode characters in their filenames, I get a {{{ 'ascii' codec can't decode byte 0xc3 in position 18: ordinal not in range(128) }}} I finally traced the problem back to my database collation: utf8_bin. I chose utf8_bin so I could order the strings in a case-sensitive manner. FYI, MySQLdb does not return python unicode strings with a utf8_bin collation, it returns utf8 bytestrings. for a brief description of that issue see: http://code.djangoproject.com/ticket/8340#comment:4 The traceback from my exception reveals the exception being thrown in ""django/db/models/fields/files.py"" in get_prep_value (line 248). FileField is a subclass of Field, but implements the same backend MySQL type (varchar) as a CharField. However it seems that FileField and CharField have completely different implementations of get_prep_db. Here is CharField's implementation: {{{ def to_python(self, value): if isinstance(value, basestring) or value is None: return value return smart_unicode(value) def get_prep_value(self, value): return self.to_python(value) }}} Here is Filefield's: {{{ def get_prep_value(self, value): ""Returns field's value prepared for saving into a database."" # Need to convert File objects provided via a form to unicode for database insertion if value is None: return None return unicode(value) }}} My experimentations revealed that if I replace the FileField implementation of get_prep_value with CharField's implementation, the exception goes away. The issue is that the default encoding is ascii and so unicode() called on a utf8 byte str blows up. The CharField implementation simply checks if the value is an instance of basestring and quietly passes it through. ",Bug,closed,"Database layer (models, ORM)",dev,Normal,fixed,utf8_binMySQLdb collation unicode bytestring,,Accepted,1,0,1,0,0,0