Opened 3 years ago

Closed 3 years ago

Last modified 3 years ago

#17634 closed Cleanup/optimization (fixed)

MultiValueDict.appendlist is ineffective

Reported by: mila Owned by: nobody
Component: Core (Other) Version: 1.3
Severity: Normal Keywords:
Cc: miloslav.pojman@…, charette.s@… Triage Stage: Ready for checkin
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: yes UI/UX: no

Description

Current MultiValueDict.appendlist implementation uses + operator in order to merge two lists -- which is pretty ineffective.

When I replaced a list merge by a simple append call I got aprox. 20% execution time of the original implementation:

    def appendlist(self, key, value):
        super(MultiValueDict, self).setdefault(key, []).append(value)

Many calls to appendlist method are possible if the MultiValueDict is being built incrementally (for example when reading response from a remote API).

Attachments (2)

master...ticket-17634-multivalue-dict-optimisation.diff (592 bytes) - added by charettes 3 years ago.
master...ticket-17634-multivalue-dict-optimisation.diff.1 (1.3 KB) - added by charettes 3 years ago.
With tests from claudep

Download all attachments as: .zip

Change History (11)

comment:1 Changed 3 years ago by mila

  • Cc miloslav.pojman@… added
  • Needs documentation unset
  • Needs tests unset
  • Patch needs improvement unset

comment:2 Changed 3 years ago by carljm

  • Component changed from Uncategorized to Core (Other)
  • Triage Stage changed from Unreviewed to Accepted

comment:3 Changed 3 years ago by charettes

  • Cc charette.s@… added
  • Has patch set

Added the diff which is also reachable on github.

All tests passing using test_sqlite settings with this patch applied on r17419.

comment:4 Changed 3 years ago by claudep

  • Needs tests set

I suggest to add a test specific to appendlist in regressiontests/utils/datastructures.py (class MultiValueDictTests).

comment:5 Changed 3 years ago by claudep

djangobench results on MultiValueDict.appendlist with current patch:

Min: 0.000008 -> 0.000004: 2.0625x faster
Avg: 0.000008 -> 0.000005: 1.8077x faster
Significant (t=48.456398)

Changed 3 years ago by charettes

With tests from claudep

comment:6 Changed 3 years ago by charettes

  • Easy pickings set
  • Needs tests unset

Added a patch with test provided by claudep.

comment:7 Changed 3 years ago by claudep

  • Triage Stage changed from Accepted to Ready for checkin

Thanks, looks fine now.

comment:8 Changed 3 years ago by aaugustin

  • Resolution set to fixed
  • Status changed from new to closed

In [17464]:

(The changeset message doesn't reference this ticket)

comment:7 Changed 3 years ago by aaugustin

In [17807]:

[1.3.X] Fixed #17634 -- Optimized the performance of MultiValueDict by using append instead of copy and by minimizing the number of dict lookups. Backport of r17464 from trunk.

Note: See TracTickets for help on using tickets.
Back to Top