Code

Opened 8 years ago

Closed 6 years ago

#2306 closed defect (fixed)

Many-to-many queries very slow on MySQL 3.23.58

Reported by: anonymous Owned by: nobody
Component: Database layer (models, ORM) Version: master
Severity: normal Keywords: qs-rf-fixed
Cc: mir@…, freakboy3742@… Triage Stage: Accepted
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: UI/UX:

Description (last modified by adrian)

We have a MySQL 3.23.58 database with about 1000 Room and Photo objects, connected by a many-to-many relationship.

Given a room r, asking Django to evaluate r.photos.all() runs an explicit join:

SELECT `kords3_photo`.`id`,`kords3_photo`.`image`
FROM `kords3_photo`
LEFT OUTER JOIN `kords3_room_photos` AS `m2m_kords3_photo__room`
ON `kords3_photo`.`id` = `m2m_kords3_photo__room`.`photo_id`
WHERE (`m2m_kords3_photo__room`.`room_id` = 1)
ORDER BY `kords3_photo`.`image` ASC

which takes around 1 second, when the same query rewritten as an implicit join:

SELECT `kords3_photo`.`id`,`kords3_photo`.`image`
FROM `kords3_photo` , `kords3_room_photos`
WHERE `kords3_photo`.`id` = `kords3_room_photos`.`photo_id`
AND `kords3_room_photos`.`room_id` = 1
ORDER BY `kords3_photo`.`image` ASC

would take under a millisecond.

Attachments (1)

2306.diff (444 bytes) - added by Marc Fargas <telenieko@…> 7 years ago.
Just add a note on database backend notes.

Download all attachments as: .zip

Change History (12)

comment:1 Changed 8 years ago by adrian

OK, why the heck is Django doing an outer join? This is upsetting. Thanks for pointing this out.

comment:2 Changed 8 years ago by lukeplant

I changed it in r2613 due to #1535

I was unaware of serious performance implications - sorry!

comment:3 Changed 8 years ago by mir@…

  • Cc mir@… added

The ORM does have severe problems handling 'or' and 'and' for queries with joins. #1535 only solved parts of them. Look at #1801 and #2080.

comment:4 Changed 8 years ago by adrian

  • Description modified (diff)
  • Status changed from new to assigned

comment:5 Changed 8 years ago by russellm

  • Cc freakboy3742@… added

comment:6 Changed 7 years ago by Simon G. <dev@…>

  • Triage Stage changed from Unreviewed to Accepted

comment:7 Changed 7 years ago by mtredinnick

  • Owner changed from adrian to mtredinnick
  • Status changed from assigned to new

Changed 7 years ago by Marc Fargas <telenieko@…>

Just add a note on database backend notes.

comment:8 Changed 7 years ago by Marc Fargas <telenieko@…>

  • Has patch set

Does this only affec 3.23.x ? If so, as the documentation recommends 4.1 or 5.0 we can place a warning about that (2306.diff does so) and either leave the ticket open in case someone wants to deal with it anyway or close it as "you have been warned" ;)

comment:9 Changed 7 years ago by russellm

  • Has patch unset

The problem isn't with the MySQL version - it's with the SQL that is generated. We shouldn't be doing a LEFT INNER JOIN in this situation. Every database will have the problem - it's just particularly obvious on old MySQL's due to the speed at which joins are performed. The upcoming query.py refactor should address this issue.

comment:10 Changed 7 years ago by mtredinnick

  • Keywords qs-rf-fixed added

comment:12 Changed 6 years ago by mtredinnick

  • Resolution set to fixed
  • Status changed from new to closed

(In [7477]) Merged the queryset-refactor branch into trunk.

This is a big internal change, but mostly backwards compatible with existing
code. Also adds a couple of new features.

Fixed #245, #1050, #1656, #1801, #2076, #2091, #2150, #2253, #2306, #2400, #2430, #2482, #2496, #2676, #2737, #2874, #2902, #2939, #3037, #3141, #3288, #3440, #3592, #3739, #4088, #4260, #4289, #4306, #4358, #4464, #4510, #4858, #5012, #5020, #5261, #5295, #5321, #5324, #5325, #5555, #5707, #5796, #5817, #5987, #6018, #6074, #6088, #6154, #6177, #6180, #6203, #6658

Add Comment

Modify Ticket

Change Properties
<Author field>
Action
as closed
as The resolution will be set. Next status will be 'closed'
The resolution will be deleted. Next status will be 'new'
Author


E-mail address and user name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.