ModelAdmin.list_editable unusably slow and memory intensive with large datasets

[Ben Cole]

Since 1.10 list_editable on ModelAdmin is unusable for Models with a large-ish dataset.

The problem is caused by a recent change to how the FormSet is generated for the admin. Previously it was generated from the ChangeList result list, but it has been changed to use the admin's get_queryset which will return more than the current "pageful" of results (potentially the entire dataset) causing Django to generate a form for each instance. This results in Django consuming all available RAM and in some cases the python instance crashing. My personal laptop became unresponsive and I had to force power off.

See ​https://github.com/django/django/commit/917cc288a38f3c114a5440f0749b7e5e1086eb36#commitcomment-23412084

[Tim Graham]

​PR (without a test)

[Pablo Martín]

+1 Same error. We have overwritten the ModelAdmin for this problem. I think solve a strange problem and getting a memory problem is not a good solution. I prefer a 500 error one time every three month that a serious memory problem every day.

This commit ​https://github.com/django/django/commit/917cc288a38f3c114a5440f0749b7e5e1086eb36#commitcomment-23412084 does not make any sense.

[Tim Graham]

I don't think reverting is a good idea as that reintroduces possible data loss.

[Rubén Gómez]

I just made a PR to solve this problem: ​https://github.com/django/django/pull/9820

[AdamDonna]

Has tests

​https://github.com/django/django/pull/9920

I'm proposing a backport to 1.11

[Tim Graham]

"Ready for checkin" is set by the patch reviewer, not the author.

[Carlton Gibson]

Hi AdamDonna. I'm going to mark this "Patch needs improvement" for now, pending the extra testing outlined on the ticket.

• Please uncheck when that's ready
• @carltongibson me on GitHub if you want/need me to input/help.

Thanks for your effort!

[Tim Graham <timograham@…>]

In b18650a2:

Fixed #28462 -- Decreased memory usage with ModelAdmin.list_editable.

Regression in 917cc288a38f3c114a5440f0749b7e5e1086eb36.

[Tim Graham <timograham@…>]

In 10a6d444:

[2.1.x] Fixed #28462 -- Decreased memory usage with ModelAdmin.list_editable.

Regression in 917cc288a38f3c114a5440f0749b7e5e1086eb36.

Backport of b18650a2634890aa758abae2f33875daa13a9ba3 from master

[Tim Graham <timograham@…>]

In 22c7c2d:

[2.0.x] Fixed #28462 -- Decreased memory usage with ModelAdmin.list_editable.

Regression in 917cc288a38f3c114a5440f0749b7e5e1086eb36.

Backport of b18650a2634890aa758abae2f33875daa13a9ba3 from master

[Tim Graham <timograham@…>]

In e9bd1a3:

Refs #28462 -- Fixed 'invalid escape sequence' warning on Python 3.6+.

[Tim Graham <timograham@…>]

In 38bdac47:

[2.1.x] Refs #28462 -- Fixed 'invalid escape sequence' warning on Python 3.6+.

Backport of e9bd1a3e12df132527a8d8bea95858e856ac7be4 from master

[Tim Graham <timograham@…>]

In 744a44df:

[2.0.x] Refs #28462 -- Fixed 'invalid escape sequence' warning on Python 3.6+.

Backport of e9bd1a3e12df132527a8d8bea95858e856ac7be4 from master

[Tim Graham <timograham@…>]

In 56c5c159:

[1.11.x] Fixed #28462 -- Decreased memory usage with ModelAdmin.list_editable.

Regression in 917cc288a38f3c114a5440f0749b7e5e1086eb36.

Backport of b18650a2634890aa758abae2f33875daa13a9ba3 from master