Opened 12 months ago

Closed 6 weeks ago

Last modified 5 weeks ago

#28462 closed Bug (fixed)

ModelAdmin.list_editable unusably slow and memory intensive with large datasets

Reported by: Ben Cole Owned by: nobody
Component: contrib.admin Version: 1.10
Severity: Normal Keywords:
Cc: josh.smeaton@… Triage Stage: Ready for checkin
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

Since 1.10 list_editable on ModelAdmin is unusable for Models with a large-ish dataset.

The problem is caused by a recent change to how the FormSet is generated for the admin. Previously it was generated from the ChangeList result list, but it has been changed to use the admin's get_queryset which will return more than the current "pageful" of results (potentially the entire dataset) causing Django to generate a form for each instance. This results in Django consuming all available RAM and in some cases the python instance crashing. My personal laptop became unresponsive and I had to force power off.

See https://github.com/django/django/commit/917cc288a38f3c114a5440f0749b7e5e1086eb36#commitcomment-23412084

Attachments (1)

memory-problem.png (6.7 KB) - added by Pablo Martín 4 months ago.

Download all attachments as: .zip

Change History (19)

comment:1 Changed 12 months ago by Tim Graham

Summary: list_editable unusable with large datasetsModelAdmin.list_editable unusably slow and memory intensive with large datasets
Triage Stage: UnreviewedAccepted

comment:2 Changed 11 months ago by Tim Graham

Has patch: set
Needs tests: set

PR (without a test)

comment:3 Changed 4 months ago by Pablo Martín

+1 Same error. We have overwritten the ModelAdmin for this problem. I think to solve a strange problem and getting a memory problem is not a good solution. I prefer a 500 error one time every three month that a serious memory problem every day.

This commit https://github.com/django/django/commit/917cc288a38f3c114a5440f0749b7e5e1086eb36#commitcomment-23412084 does not make any sense.

Four examples of this big problem:


Please consider revert this change, it is very dangerous for other people if they don't identify this bug/trap.

Last edited 4 months ago by Pablo Martín (previous) (diff)

Changed 4 months ago by Pablo Martín

Attachment: memory-problem.png added

comment:4 Changed 4 months ago by Pablo Martín

Easy pickings: set

comment:5 Changed 4 months ago by Tim Graham

Easy pickings: unset

I don't think reverting is a good idea as that reintroduces possible data loss.

comment:6 Changed 4 months ago by Rubén Gómez

I just made a PR to solve this problem: https://github.com/django/django/pull/9820

comment:7 Changed 2 months ago by Josh Smeaton

Cc: josh.smeaton@… added

comment:8 Changed 2 months ago by AdamDonna

Needs tests: unset
Triage Stage: AcceptedReady for checkin

Has tests

https://github.com/django/django/pull/9920

I'm proposing a backport to 1.11

comment:9 Changed 2 months ago by Tim Graham

Triage Stage: Ready for checkinAccepted

"Ready for checkin" is set by the patch reviewer, not the author.

comment:10 Changed 2 months ago by Carlton Gibson

Patch needs improvement: set

Hi AdamDonna. I'm going to mark this "Patch needs improvement" for now, pending the extra testing outlined on the ticket.

  • Please uncheck when that's ready
  • @carltongibson me on GitHub if you want/need me to input/help.

Thanks for your effort!

comment:11 Changed 2 months ago by Carlton Gibson

Patch needs improvement: unset
Triage Stage: AcceptedReady for checkin

comment:12 Changed 6 weeks ago by Tim Graham <timograham@…>

Resolution: fixed
Status: newclosed

In b18650a2:

Fixed #28462 -- Decreased memory usage with ModelAdmin.list_editable.

Regression in 917cc288a38f3c114a5440f0749b7e5e1086eb36.

comment:13 Changed 6 weeks ago by Tim Graham <timograham@…>

In 10a6d444:

[2.1.x] Fixed #28462 -- Decreased memory usage with ModelAdmin.list_editable.

Regression in 917cc288a38f3c114a5440f0749b7e5e1086eb36.

Backport of b18650a2634890aa758abae2f33875daa13a9ba3 from master

comment:14 Changed 6 weeks ago by Tim Graham <timograham@…>

In 22c7c2d:

[2.0.x] Fixed #28462 -- Decreased memory usage with ModelAdmin.list_editable.

Regression in 917cc288a38f3c114a5440f0749b7e5e1086eb36.

Backport of b18650a2634890aa758abae2f33875daa13a9ba3 from master

comment:15 Changed 6 weeks ago by Tim Graham <timograham@…>

In e9bd1a3:

Refs #28462 -- Fixed 'invalid escape sequence' warning on Python 3.6+.

comment:16 Changed 6 weeks ago by Tim Graham <timograham@…>

In 38bdac47:

[2.1.x] Refs #28462 -- Fixed 'invalid escape sequence' warning on Python 3.6+.

Backport of e9bd1a3e12df132527a8d8bea95858e856ac7be4 from master

comment:17 Changed 6 weeks ago by Tim Graham <timograham@…>

In 744a44df:

[2.0.x] Refs #28462 -- Fixed 'invalid escape sequence' warning on Python 3.6+.

Backport of e9bd1a3e12df132527a8d8bea95858e856ac7be4 from master

comment:18 Changed 5 weeks ago by Tim Graham <timograham@…>

In 56c5c159:

[1.11.x] Fixed #28462 -- Decreased memory usage with ModelAdmin.list_editable.

Regression in 917cc288a38f3c114a5440f0749b7e5e1086eb36.

Backport of b18650a2634890aa758abae2f33875daa13a9ba3 from master

Note: See TracTickets for help on using tickets.
Back to Top