Many-to-many queries very slow on MySQL 3.23.58

[anonymous]

We have a MySQL 3.23.58 database with about 1000 Room and Photo objects, connected by a many-to-many relationship.

Given a room r, asking Django to evaluate r.photos.all() runs an explicit join:

SELECT `kords3_photo`.`id`,`kords3_photo`.`image`
FROM `kords3_photo`
LEFT OUTER JOIN `kords3_room_photos` AS `m2m_kords3_photo__room`
ON `kords3_photo`.`id` = `m2m_kords3_photo__room`.`photo_id`
WHERE (`m2m_kords3_photo__room`.`room_id` = 1)
ORDER BY `kords3_photo`.`image` ASC

which takes around 1 second, when the same query rewritten as an implicit join:

SELECT `kords3_photo`.`id`,`kords3_photo`.`image`
FROM `kords3_photo` , `kords3_room_photos`
WHERE `kords3_photo`.`id` = `kords3_room_photos`.`photo_id`
AND `kords3_room_photos`.`room_id` = 1
ORDER BY `kords3_photo`.`image` ASC

would take under a millisecond.

[Adrian Holovaty]

OK, why the heck is Django doing an outer join? This is upsetting. Thanks for pointing this out.

[Luke Plant]

I changed it in r2613 due to #1535

I was unaware of serious performance implications - sorry!

[mir@…]

The ORM does have severe problems handling 'or' and 'and' for queries with joins. #1535 only solved parts of them. Look at #1801 and #2080.

[Marc Fargas <telenieko@…>]

Just add a note on database backend notes.

[Marc Fargas <telenieko@…>]

Does this only affec 3.23.x ? If so, as the documentation recommends 4.1 or 5.0 we can place a warning about that (2306.diff does so) and either leave the ticket open in case someone wants to deal with it anyway or close it as "you have been warned" ;)

[Russell Keith-Magee]

The problem isn't with the MySQL version - it's with the SQL that is generated. We shouldn't be doing a LEFT INNER JOIN in this situation. Every database will have the problem - it's just particularly obvious on old MySQL's due to the speed at which joins are performed. The upcoming query.py refactor should address this issue.

[Malcolm Tredinnick]

(In [7477]) Merged the queryset-refactor branch into trunk.

This is a big internal change, but mostly backwards compatible with existing
code. Also adds a couple of new features.

Fixed #245, #1050, #1656, #1801, #2076, #2091, #2150, #2253, #2306, #2400, #2430, #2482, #2496, #2676, #2737, #2874, #2902, #2939, #3037, #3141, #3288, #3440, #3592, #3739, #4088, #4260, #4289, #4306, #4358, #4464, #4510, #4858, #5012, #5020, #5261, #5295, #5321, #5324, #5325, #5555, #5707, #5796, #5817, #5987, #6018, #6074, #6088, #6154, #6177, #6180, #6203, #6658