Opened 9 months ago

Closed 9 months ago

Last modified 9 months ago

#35159 closed Bug (fixed)

dumpdata crashes on querysets with prefetch_related().

Reported by: Andrea F Owned by: Mariusz Felisiak
Component: Core (Management commands) Version: 5.0
Severity: Release blocker Keywords: dumpdata chunk_size iterator prefetch_related
Cc: Jacob Walls, Raphael Kimmig, Simon Charette Triage Stage: Accepted
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

The ./manage.py dumpdata management command calls iterator() on the QuerySets of the models, but it does not pass chunk_size(https://github.com/django/django/blob/main/django/core/management/commands/dumpdata.py#L222). On model QuerySets that include prefetch_related, this now (with 5.0) causes this error while running dumpdata:

CommandError: Unable to serialize database: chunk_size must be provided when using QuerySet.iterator() after prefetch_related().

Proposed fix is to pass a suitable chunk_size parameter to iterator() within handle() in dumpdata.py.

Change History (7)

comment:1 by yushan, 9 months ago

Owner: changed from nobody to yushan
Status: newassigned

comment:2 by Mariusz Felisiak, 9 months ago

Cc: Jacob Walls Raphael Kimmig Simon Charette added
Easy pickings: unset
Severity: NormalRelease blocker
Summary: dumpdata does not work in 5.0.1 on QuerySets with prefetch_relateddumpdata crashes on querysets with prefetch_related().
Triage Stage: UnreviewedAccepted

Thanks for the report. Does the following patch work for you?

  • django/core/management/commands/dumpdata.py

    diff --git a/django/core/management/commands/dumpdata.py b/django/core/management/commands/dumpdata.py
    index cc183517e3..01ff8974dd 100644
    a b class Command(BaseCommand):  
    219219                    if count_only:
    220220                        yield queryset.order_by().count()
    221221                    else:
    222                         yield from queryset.iterator()
     222                        chunk_size = (
     223                            2000 if queryset._prefetch_related_lookups else None
     224                        )
     225                        yield from queryset.iterator(chunk_size=chunk_size)
    223226
    224227        try:
    225228            self.stdout.ending = None

Regression in 139135627650ed6aaaf4c755b82c3bd43f2b8f51 (#29984).

comment:3 by Mariusz Felisiak, 9 months ago

Owner: changed from yushan to Mariusz Felisiak

comment:4 by Mariusz Felisiak, 9 months ago

comment:5 by Mariusz Felisiak, 9 months ago

Has patch: set

comment:6 by GitHub <noreply@…>, 9 months ago

Resolution: fixed
Status: assignedclosed

In 38eaf2f2:

Fixed #35159 -- Fixed dumpdata crash when base querysets use prefetch_related().

Regression in 139135627650ed6aaaf4c755b82c3bd43f2b8f51
following deprecation in edbf930287cb72e9afab1f7208c24b1146b0c4ec.

Thanks Andrea F for the report.

comment:7 by Mariusz Felisiak <felisiak.mariusz@…>, 9 months ago

In 7453d6a:

[5.0.x] Fixed #35159 -- Fixed dumpdata crash when base querysets use prefetch_related().

Regression in 139135627650ed6aaaf4c755b82c3bd43f2b8f51
following deprecation in edbf930287cb72e9afab1f7208c24b1146b0c4ec.

Thanks Andrea F for the report.
Backport of 38eaf2f21a2398a8dd8444f6df3723898cb5fe2a from main

Note: See TracTickets for help on using tickets.
Back to Top