Opened 18 years ago

Closed 16 years ago

#2306 closed defect (fixed)

Many-to-many queries very slow on MySQL 3.23.58

Reported by: anonymous Owned by: nobody
Component: Database layer (models, ORM) Version: dev
Severity: normal Keywords: qs-rf-fixed
Cc: mir@…, freakboy3742@… Triage Stage: Accepted
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description (last modified by Adrian Holovaty)

We have a MySQL 3.23.58 database with about 1000 Room and Photo objects, connected by a many-to-many relationship.

Given a room r, asking Django to evaluate r.photos.all() runs an explicit join:

SELECT `kords3_photo`.`id`,`kords3_photo`.`image`
FROM `kords3_photo`
LEFT OUTER JOIN `kords3_room_photos` AS `m2m_kords3_photo__room`
ON `kords3_photo`.`id` = `m2m_kords3_photo__room`.`photo_id`
WHERE (`m2m_kords3_photo__room`.`room_id` = 1)
ORDER BY `kords3_photo`.`image` ASC

which takes around 1 second, when the same query rewritten as an implicit join:

SELECT `kords3_photo`.`id`,`kords3_photo`.`image`
FROM `kords3_photo` , `kords3_room_photos`
WHERE `kords3_photo`.`id` = `kords3_room_photos`.`photo_id`
AND `kords3_room_photos`.`room_id` = 1
ORDER BY `kords3_photo`.`image` ASC

would take under a millisecond.

Attachments (1)

2306.diff (444 bytes ) - added by Marc Fargas <telenieko@…> 17 years ago.
Just add a note on database backend notes.

Download all attachments as: .zip

Change History (12)

comment:1 by Adrian Holovaty, 18 years ago

OK, why the heck is Django doing an outer join? This is upsetting. Thanks for pointing this out.

comment:2 by Luke Plant, 18 years ago

I changed it in r2613 due to #1535

I was unaware of serious performance implications - sorry!

comment:3 by mir@…, 18 years ago

Cc: mir@… added

The ORM does have severe problems handling 'or' and 'and' for queries with joins. #1535 only solved parts of them. Look at #1801 and #2080.

comment:4 by Adrian Holovaty, 18 years ago

Description: modified (diff)
Status: newassigned

comment:5 by Russell Keith-Magee, 18 years ago

Cc: freakboy3742@… added

comment:6 by Simon G. <dev@…>, 17 years ago

Triage Stage: UnreviewedAccepted

comment:7 by Malcolm Tredinnick, 17 years ago

Owner: changed from Adrian Holovaty to Malcolm Tredinnick
Status: assignednew

by Marc Fargas <telenieko@…>, 17 years ago

Attachment: 2306.diff added

Just add a note on database backend notes.

comment:8 by Marc Fargas <telenieko@…>, 17 years ago

Has patch: set

Does this only affec 3.23.x ? If so, as the documentation recommends 4.1 or 5.0 we can place a warning about that (2306.diff does so) and either leave the ticket open in case someone wants to deal with it anyway or close it as "you have been warned" ;)

comment:9 by Russell Keith-Magee, 17 years ago

Has patch: unset

The problem isn't with the MySQL version - it's with the SQL that is generated. We shouldn't be doing a LEFT INNER JOIN in this situation. Every database will have the problem - it's just particularly obvious on old MySQL's due to the speed at which joins are performed. The upcoming query.py refactor should address this issue.

comment:10 by Malcolm Tredinnick, 17 years ago

Keywords: qs-rf-fixed added

comment:12 by Malcolm Tredinnick, 16 years ago

Resolution: fixed
Status: newclosed

(In [7477]) Merged the queryset-refactor branch into trunk.

This is a big internal change, but mostly backwards compatible with existing
code. Also adds a couple of new features.

Fixed #245, #1050, #1656, #1801, #2076, #2091, #2150, #2253, #2306, #2400, #2430, #2482, #2496, #2676, #2737, #2874, #2902, #2939, #3037, #3141, #3288, #3440, #3592, #3739, #4088, #4260, #4289, #4306, #4358, #4464, #4510, #4858, #5012, #5020, #5261, #5295, #5321, #5324, #5325, #5555, #5707, #5796, #5817, #5987, #6018, #6074, #6088, #6154, #6177, #6180, #6203, #6658

Note: See TracTickets for help on using tickets.
Back to Top