Code

Opened 5 years ago

Closed 3 years ago

#10398 closed Bug (needsinfo)

It may cause error in django.db.backends.last_executed_query.

Reported by: bear330 Owned by: nobody
Component: Database layer (models, ORM) Version: 1.0
Severity: Normal Keywords: unicode pickle db backends
Cc: Triage Stage: Accepted
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: UI/UX:

Description

In CursorDebugWrapper.execute:

        start = time()
        try:
            return self.cursor.execute(sql, params)
        finally:
            stop = time()
            sql = self.db.ops.last_executed_query(self.cursor, sql, params)
            self.db.queries.append({
                'sql': sql,
                'time': "%.3f" % (stop - start),
            })

Line: sql = self.db.ops.last_executed_query(self.cursor, sql, params)
Might cause error if params can't be convert to unicode.
In last_executed_query:

        to_unicode = lambda s: force_unicode(s, strings_only=True)
        if isinstance(params, (list, tuple)):
            u_params = tuple([to_unicode(val) for val in params])
        else:
            u_params = dict([(to_unicode(k), to_unicode(v)) for k, v in params.items()])

This will fail if params contains pickled object in some situation.

It should be wrapped by try ... except block.
=>

    def last_executed_query(self, cursor, sql, params):
        from django.utils.encoding import smart_unicode, force_unicode
        
        # Convert params to contain Unicode values.
        to_unicode = lambda s: force_unicode(s, strings_only=True)
        try:
            if isinstance(params, (list, tuple)):
                u_params = tuple([to_unicode(val) for val in params])
            else:
                u_params = dict([(to_unicode(k), to_unicode(v)) for k, v in params.items()])

            return smart_unicode(sql) % u_params
        except:
            return smart_unicode(sql)

Thanks.

Attachments (1)

2.txt (833 bytes) - added by bear330 5 years ago.
This is my Trace object's pickled data, that will cause error if try to save it.

Download all attachments as: .zip

Change History (6)

comment:1 Changed 5 years ago by mtredinnick

  • Component changed from Uncategorized to Database layer (models, ORM)
  • Needs documentation unset
  • Needs tests unset
  • Patch needs improvement unset
  • Triage Stage changed from Unreviewed to Accepted

Wrapping things in a try...except block isn't a particularly nice solution. It hides the problem by throwing away all the data. Better to use "errors='replace'" in the unicode conversion.

At the same time, we might move the parameters away from being substituted into the query, since that always leads to confusion. Then we can document that parameters are a "best attempt" at being represented.

comment:2 Changed 5 years ago by bear330

Sorry, I found this only causes error when self.cursor.execute(sql, params) fails.

If I save model with a pickled object (it contains '»' character in its property), line:
self.cursor.execute(sql, params)
will fails. (#7921. ??)

It will raise
ProgrammingError: You must not use 8-bit bytestrings unless you use a text_factory that can interpret 8-bit bytestrings (like text_factory = str). It is highly recommended that you instead just switch your application to Unicode strings.

I am using sqlite3.

comment:3 Changed 5 years ago by bear330

I got this when I dig into sqlite3 backend:

'utf8' codec can't decode byte 0xbb in position 654: unexpected code byte

caused by django.db.backends.sqlite3.base, line 51:

Database.register_adapter(str, lambda s:s.decode('utf-8'))

Parameter s is attached in attachment file.

Thanks.

Changed 5 years ago by bear330

This is my Trace object's pickled data, that will cause error if try to save it.

comment:4 Changed 5 years ago by mtredinnick

Hold on, you seem to be changing what the bug is about. Are you saying that it's the database interaction that is failing, or is only the recording of the query failing?

If it's the database interaction, then that's because Django doesn't support raw binary types for data at the moment. It's up to you to encode the data appropriately before passing it through in a text field (in which case, this ticket is an "invalid" for now). Passing arbitrary binary data simply isn't supported and there's nothing to suggest that it does, so it's unsurprising you get errors.

If the database saving is working and only the storing of information in the debug log is causing a problem, then that's something we should fix.

So please clarify what the real problem is here.

comment:5 Changed 3 years ago by SmileyChris

  • Resolution set to needsinfo
  • Severity set to Normal
  • Status changed from new to closed
  • Type set to Bug

Add Comment

Modify Ticket

Change Properties
<Author field>
Action
as closed
as The resolution will be set. Next status will be 'closed'
The resolution will be deleted. Next status will be 'new'
Author


E-mail address and user name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.