It may cause error in django.db.backends.last_executed_query.

[bear330]

In CursorDebugWrapper.execute:

        start = time()
        try:
            return self.cursor.execute(sql, params)
        finally:
            stop = time()
            sql = self.db.ops.last_executed_query(self.cursor, sql, params)
            self.db.queries.append({
                'sql': sql,
                'time': "%.3f" % (stop - start),
            })

Line: sql = self.db.ops.last_executed_query(self.cursor, sql, params)
Might cause error if params can't be convert to unicode.
In last_executed_query:

        to_unicode = lambda s: force_unicode(s, strings_only=True)
        if isinstance(params, (list, tuple)):
            u_params = tuple([to_unicode(val) for val in params])
        else:
            u_params = dict([(to_unicode(k), to_unicode(v)) for k, v in params.items()])

This will fail if params contains pickled object in some situation.

It should be wrapped by try ... except block.
=>

    def last_executed_query(self, cursor, sql, params):
        from django.utils.encoding import smart_unicode, force_unicode
        
        # Convert params to contain Unicode values.
        to_unicode = lambda s: force_unicode(s, strings_only=True)
        try:
            if isinstance(params, (list, tuple)):
                u_params = tuple([to_unicode(val) for val in params])
            else:
                u_params = dict([(to_unicode(k), to_unicode(v)) for k, v in params.items()])

            return smart_unicode(sql) % u_params
        except:
            return smart_unicode(sql)

Thanks.

[Malcolm Tredinnick]

Wrapping things in a try...except block isn't a particularly nice solution. It hides the problem by throwing away all the data. Better to use "errors='replace'" in the unicode conversion.

At the same time, we might move the parameters away from being substituted into the query, since that always leads to confusion. Then we can document that parameters are a "best attempt" at being represented.

[bear330]

Sorry, I found this only causes error when self.cursor.execute(sql, params) fails.

If I save model with a pickled object (it contains '»' character in its property), line:
self.cursor.execute(sql, params)
will fails. (#7921. ??)

It will raise
ProgrammingError: You must not use 8-bit bytestrings unless you use a text_factory that can interpret 8-bit bytestrings (like text_factory = str). It is highly recommended that you instead just switch your application to Unicode strings.

I am using sqlite3.

[bear330]

I got this when I dig into sqlite3 backend:

'utf8' codec can't decode byte 0xbb in position 654: unexpected code byte

caused by django.db.backends.sqlite3.base, line 51:

Database.register_adapter(str, lambda s:s.decode('utf-8'))

Parameter s is attached in attachment file.

Thanks.

[bear330]

This is my Trace object's pickled data, that will cause error if try to save it.

[Malcolm Tredinnick]

Hold on, you seem to be changing what the bug is about. Are you saying that it's the database interaction that is failing, or is only the recording of the query failing?

If it's the database interaction, then that's because Django doesn't support raw binary types for data at the moment. It's up to you to encode the data appropriately before passing it through in a text field (in which case, this ticket is an "invalid" for now). Passing arbitrary binary data simply isn't supported and there's nothing to suggest that it does, so it's unsurprising you get errors.

If the database saving is working and only the storing of information in the debug log is causing a problem, then that's something we should fix.

So please clarify what the real problem is here.