Opened 11 years ago

Closed 10 years ago

Last modified 2 years ago

#14091 closed Bug (fixed)

Fix incorrect quoting in connection.queries

Reported by: Daniel Roseman Owned by: Jacob
Component: Database layer (models, ORM) Version: 1.2
Severity: Normal Keywords:
Cc: Triage Stage: Ready for checkin
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no


The default implementation of DatabaseOperations.last_executed_query returns incorrectly quoted values for many common queries in MySQL. This means that connection.queries does not properly reflect the query that was actually executed by the database, leading to hard-to-debug errors.

For example, given this model:

    class Foo(models.Model):
        name = models.CharField(max_length=10)
        added = models.DateTimeField()

doing this query:


gives this result from connection.queries:

[{'sql': "SELECT `foo_foo`.`id`, `foo_foo`.`name`, `foo_foo`.`added` FROM `foo_foo` WHERE `foo_foo`.`added` <= 2010-08-11 02:58:21  LIMIT 21",
  'time': '0.052'}]

which is invalid SQL because the timestamp is not quoted at all - although the query actually sent to the db is correctly quoted. More seriously, doing something like this:

values = ((u'1', '2010-08-09T15:55:28'),
          (u'2', '2010-08-10T15:55:28'),
          (u'3', '2010-08-11T15:55:28'))
Foo.objects.extra(where=['(name, added) in %s'], params=[values])

wrongly reports that the values tuple has been correctly quoted:

[{'sql': "SELECT `foo_foo`.`id`, `foo_foo`.`name`, `foo_foo`.`added` FROM `foo_foo` WHERE (name, added) in ((u'0', '2010-08-08T15:55:28'), 
(u'1', '2010-08-09T15:55:28'), (u'2', '2010-08-10T15:55:28'), (u'3', '2010-08-11T15:55:28')) LIMIT 21",  'time': '0.052'}]

when in fact the query sent to the database is

SELECT `foo_foo`.`id`, `foo_foo`.`name`, `foo_foo`.`added` FROM `foo_foo` WHERE (name, added) in (("\'0\'", "\'2010-08-09T15:55:28\'"), 
("\'0\'", "\'2010-08-09T15:55:28\'"), ("\'0\'", "\'2010-08-09T15:55:28\'"), ("\'0\'", "\'2010-08-09T15:55:28\'"), ("\'0\'", "\'2010-08-09T15:55:28\'"), 
("\'0\'", "\'2010-08-09T15:55:28\'"), ("\'0\'", "\'2010-08-09T15:55:28\'"), ("\'0\'", "\'2010-08-09T15:55:28\'"), ("\'0\'", "\'2010-08-09T15:55:28\'")) 

which is obviously not what was intended.

The Postgres backend overrides the last_executed_query method to use the cursor.query atribute, ensuring that the query is correctly represented. MySQLdb actually provides a similar attribute, cursor._executed, and this should be used by the MySQL backend. Attached patch uses this, but falls back to the default implementation if the attribute is not populated (the current version of MySQLdb occasionally does not populate _executed, although this appears to be fixed in the forthcoming version 2.)

No tests attached, because I couldn't work out where to put them - happy to do so with some advice.

Attachments (4)

mysql_lastexecuted.diff (685 bytes) - added by Daniel Roseman 11 years ago.
14091.patch (6.2 KB) - added by Aymeric Augustin 10 years ago.
14091.2.patch (5.8 KB) - added by Aymeric Augustin 10 years ago.
14091.3.patch (6.6 KB) - added by Aymeric Augustin 10 years ago.

Download all attachments as: .zip

Change History (23)

Changed 11 years ago by Daniel Roseman

Attachment: mysql_lastexecuted.diff added

comment:1 in reply to:  description Changed 10 years ago by Ramiro Morales

Triage Stage: UnreviewedAccepted

Replying to danielr:

No tests attached, because I couldn't work out where to put them - happy to do so with some advice.

tests/regressiontests/backends would be an appropriate location.

comment:2 Changed 10 years ago by Luke Plant

Needs tests: set

comment:3 Changed 10 years ago by Aymeric Augustin

Owner: changed from nobody to Aymeric Augustin

#11209 reports the same problem; it was "fixed" in r10847 by a note in the docs.

#8877 reports the same problem plus another case sensitivity problem; it was closed after the latter was considered invalid.

comment:4 Changed 10 years ago by Aymeric Augustin

Needs tests: unset

Attached patch implements query retrieval for MySQL and Oracle, and proper query rebuilding for SQLite.

With this patch, the postgresql (psycopg v1) backend still does not have proper escaping, but it is deprecated. A discussion on IRC determined that implementing a generic (SQL-92 style) parameter escaping function would be a bad idea, because it's extremely difficult to get right. For starters, mapping the Python types to database parameters is database dependent... So we will stick to database-specific implementations.

I tested the patch under SQLite and MySQL.

I do not have access to an Oracle server. My proposal is purely based on the doc. However, the change is trivial and could easily be reverted if the tests do not pass under Oracle.

Changed 10 years ago by Aymeric Augustin

Attachment: 14091.patch added

comment:5 Changed 10 years ago by Aymeric Augustin

Severity: Normal
Summary: PATCH: fix incorrect quoting in connection.queries for MySQLFix incorrect quoting in connection.queries
Type: Uncategorized

I updated the patch after the postgresql backend was removed and the postgresql_psycopg2 backend was refactored.

Changed 10 years ago by Aymeric Augustin

Attachment: 14091.2.patch added

comment:6 Changed 10 years ago by Aymeric Augustin

Type: UncategorizedBug

comment:7 Changed 10 years ago by Ian Kelly

I've run the test under Oracle, and it passes.

comment:8 Changed 10 years ago by Aymeric Augustin

Owner: Aymeric Augustin deleted

I de-assign myself since I've proposed a patch and I have nothing more to do until someone reviews it.

comment:9 Changed 10 years ago by Alex Gaynor

Easy pickings: unset

connections.queries is explicitly documented as not quoting things correctly, here, I'm not sure why we'd bother to fix this, the patch is non-trivial (it involves running queries on SQLite) and adds little value IMO.

comment:10 Changed 10 years ago by Aymeric Augustin

Alex, the reasons for fixing this are:

  • make it possible to test queries by copy-pasting from the debug bar or a debug log. That is useful to troubleshoot performance issues: ./manage dbshell then EXPLAIN <<Ctrl-V>>. Re-quoting manually complex queries is just a pain. The framework should do that for us.
  • remove a common source of questions and frustration for users, as seen on #django and in duplicate tickets.

Also, Django is supposed be a framework for perfectionnists; IMHO the status quo yeah, it's buggy, but we say it in the small print... doesn't live up to this goal. Should we ask django-developers for other opinions?

comment:11 Changed 10 years ago by Jacob

I disagree with Alex about the lack of value, but like Alex, I'm not sold on the approach under SQLite - it seems rather hackish and prone to failure. I aprove (highly) of the fixes for MySQL and Oracle, and I'd like to see those go in.

I say we fix this where we can (MySQL and Oracle) and leave it as-is for SQLite. If there's a non-gross way of retrieving the last query from the sqlite3 library then let's by all means use that, but if not we'll need to leave SQLite alone. We already have a good document explaining the differences between databases, so this can be another caveat there (and another reason to use a real database).

aaugustin, if you make those changes feel free to mark this ticket RFC. Everything else looks good to me.

comment:12 Changed 10 years ago by Jacob

aaugustin: also, add your name to AUTHORS if you'd like credit when we check this in!

comment:13 Changed 10 years ago by Alex Gaynor

FWIW jacob, I spoke with aaugustin on IRC and I agree with your position, if we can do it in a non-hacky way (as psycopg2 is now), +1.

Changed 10 years ago by Aymeric Augustin

Attachment: 14091.3.patch added

comment:14 Changed 10 years ago by Aymeric Augustin

Triage Stage: AcceptedReady for checkin

This new patch implements Alex and Jacob's recommendation.

Regarding SQLite, parameters are substituted at line 642 in _sqlite/cursor.c:

pysqlite_statement_bind_parameters(self->statement, parameters, allow_8bit_chars);

Unfortunately there is no way to reach self->statement from Python code. That's why I ended up quoting parameters with SELECT QUOTE(?).

comment:15 Changed 10 years ago by Jacob

Owner: set to Jacob
Resolution: fixed
Status: newclosed

In [16081]:

Fixed #14091 - be more correct about logging queries in connection.queries.

Thanks to Aymeric Augustin for figuring out how to make this work across
multiple databases.

comment:16 Changed 6 years ago by Aymeric Augustin

UI/UX: unset

Four and a half years later, I still believe this should be fixed on SQLite as well.

I submitted a pull request:

comment:17 Changed 6 years ago by Aymeric Augustin <aymeric.augustin@…>

In 4f6a766:

Refs #14091 -- Fixed connection.queries on SQLite.

comment:18 Changed 2 years ago by John Cant


I am having this problem on PostgreSQL. I think my use case is sane:

Use the Django ORM to generate some SQL -> Use Pandas to evaluate the SQL and return a data frame.

comment:19 Changed 2 years ago by Tim Graham

Please open a new issue with steps to reproduce.

Note: See TracTickets for help on using tickets.
Back to Top