Opened 5 months ago

Closed 4 months ago

Last modified 4 months ago

#35467 closed Cleanup/optimization (fixed)

Prefer urlsplit() over urlparse()

Reported by: Adam Johnson Owned by: Jake Howard
Component: Utilities Version: dev
Severity: Normal Keywords:
Cc: Triage Stage: Ready for checkin
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description (last modified by Adam Johnson)

Many places in Django use [urlparse()](https://docs.python.org/3/library/urllib.parse.html#urllib.parse.urlparse), which supports the rarely-used “path parameter” syntax (not to be confused with query parameters). The urlsplit() function is similar but does not parse such path parameters, which makes it a bit faster.

I think most or all calls to urlparse() can be replaced with urlsplit(), and similarly urlunparse() with urlunsplit(). This may make a small but measurable performance difference in common paths, such as in CsrfViewMiddleware or the test Client.

See more in this Anthony Sottile video: https://www.youtube.com/watch?v=ABJvdsIANds , where he reports a 3% import time improvement on the Stripe project.

Change History (8)

comment:1 by Adam Johnson, 5 months ago

Description: modified (diff)

comment:2 by Sarah Boyce, 5 months ago

Triage Stage: UnreviewedAccepted

Accepting for someone to make updates and confirm with benchmarks. 👍

comment:3 by Jake Howard, 5 months ago

Owner: changed from nobody to Jake Howard
Status: newassigned

From some basic testing, it looks like this should have a nice improvement:

In [1]: import urllib.parse

In [2]: %timeit urllib.parse.urlparse("https://example.com")
1.52 µs ± 37.9 ns per loop (mean ± std. dev. of 7 runs, 1,000,000 loops each)

In [3]: %timeit urllib.parse.urlsplit("https://example.com")
258 ns ± 1.14 ns per loop (mean ± std. dev. of 7 runs, 1,000,000 loops each)

The difference stays about constant, even with longer URLs. The specifics will obviously vary by hardware and Python version (the above is 3.12.3), but ~6x improvement is definitely worthwhile.

comment:4 by Jake Howard, 5 months ago

Has patch: set
Last edited 5 months ago by Jake Howard (previous) (diff)

comment:5 by Natalia Bidart, 4 months ago

Needs tests: set
Patch needs improvement: set

comment:6 by Jake Howard, 4 months ago

Needs tests: unset
Patch needs improvement: unset

Comments addressed / replied-to in PR.

comment:7 by Mariusz Felisiak, 4 months ago

Resolution: fixed
Status: assignedclosed

comment:8 by Natalia Bidart, 4 months ago

Triage Stage: AcceptedReady for checkin
Note: See TracTickets for help on using tickets.
Back to Top