Opened 3 years ago

Last modified 7 months ago

#18729 assigned Bug

Admin changelist view defaults to `DISTINCT`, unusable on reasonably sized databases

Reported by: hcarvalhoalves Owned by: AlexMalek
Component: contrib.admin Version: master
Severity: Normal Keywords: admin sql distinct slow performance
Cc: Triage Stage: Accepted
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: yes
Easy pickings: no UI/UX: no

Description

The default behavior on the admin is not usable for reasonably sized tables (a couple thousand tuples is enough for >1s queries).

Right now, it always includes a DISTINCT clause if you add a M2M field on list_filter, even if you're not applying any filters. Use of DISTINCT is known to be painfully slow, and even more so without a WHERE clause, because it forces SQL databases into a full table scan. The behavior is not easily overridden because the logic is scattered on a couple ChangeList methods (https://github.com/django/django/blob/master/django/contrib/admin/views/main.py#L299).

The only solution right now is not including a M2M field on list_filter and limiting functionality. It would be better if that behavior could be overridden (a use_distinct parameter on get_query_set?) or, optimally, moving this logic somewhere else. Ticket #15819 already includes a patch for that.

Change History (7)

comment:1 Changed 2 years ago by tswicegood

  • Needs documentation unset
  • Needs tests unset
  • Patch needs improvement unset
  • Resolution set to duplicate
  • Status changed from new to closed
  • Type changed from Uncategorized to Bug

Resolving this as a dupe #15819. Looks like that issue has all of the same points here and fixes this issue.

Feel free to re-open if I missed something in this ticket that's different than #15819.

comment:2 Changed 10 months ago by AlexMalek

  • Resolution duplicate deleted
  • Status changed from closed to new

I think it is a different issue.
It is possible to alter the check that examines list_filter to determine if distinct() is needed based on if the field appears in the query/request but this does not address the deeper issues in #15819
For example in Django 1.5.2 I think the following changes solves the issue:

django/contrib/admin/util.py:
20c20
< def lookup_needs_distinct(opts, lookup_path, query_dict=None):
---

def lookup_needs_distinct(opts, lookup_path):

29,31c29
< not field.field.unique) and
< (query_dict is None or (field_name in ",".join(query_dict.keys())))
< ):
---

not field.field.unique)):

django/contrib/admin/views/main.py:
128c128
< field_path, lookup_params))
---

field_path))

comment:3 Changed 9 months ago by AlexMalek

  • Owner changed from nobody to AlexMalek
  • Status changed from new to assigned

comment:4 Changed 9 months ago by AlexMalek

  • Has patch set

comment:5 Changed 9 months ago by timo

  • Triage Stage changed from Unreviewed to Accepted

comment:6 Changed 7 months ago by slurms

  • Triage Stage changed from Accepted to Ready for checkin

comment:7 Changed 7 months ago by timo

  • Patch needs improvement set
  • Triage Stage changed from Ready for checkin to Accepted

I left comments for improvement on the PR. Please uncheck "Patch needs improvement" when you update it, thanks.

Note: See TracTickets for help on using tickets.
Back to Top