Opened 4 years ago

Closed 4 years ago

Last modified 4 years ago

#17516 closed Uncategorized (wontfix)

Memoize _os.safe_join()

Reported by: Kronuz Owned by: nobody
Component: Uncategorized Version: 1.3
Severity: Normal Keywords:
Cc: Triage Stage: Unreviewed
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

I found out safe_join() was taking a big chunk of time in my project (I have many apps and the load_template_source() calls use safe_join).

Since safe_join() does a lot of repetitive things when it joins paths in the same platform, and since the number of joined paths can be sort of limited, I made a patch to memoize it.

Before:

Stats:
         1372899 function calls (1333476 primitive calls) in 1.744 seconds

   Ordered by: internal time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
    16170    0.165    0.000    0.240    0.000 posixpath.py:312(normpath)
    15288    0.148    0.000    0.148    0.000 {posix.getcwdu}
     8085    0.073    0.000    0.073    0.000 {open}
     1018    0.051    0.000    0.051    0.000 {method 'recv' of '_socket.socket' objects}
    23375    0.048    0.000    0.068    0.000 posixpath.py:60(join)
   191264    0.046    0.000    0.046    0.000 {isinstance}
     8085    0.042    0.000    0.566    0.000 _os.py:28(safe_join)
    80087    0.035    0.000    0.035    0.000 {method 'startswith' of 'unicode' objects}
    16170    0.035    0.000    0.482    0.000 _os.py:18(abspathu)
   2468/1    0.034    0.000    1.533    1.533 base.py:740(render)
48746/40126    0.033    0.000    0.348    0.000 {getattr}
   200177    0.033    0.000    0.033    0.000 {method 'append' of 'list' objects}
8710/7896    0.029    0.000    0.062    0.000 base.py:373(__setattr__)
6884/6587    0.028    0.000    0.342    0.000 base.py:667(_resolve_lookup)
45540/43923    0.028    0.000    0.087    0.000 encoding.py:54(force_unicode)
      147    0.025    0.000    0.662    0.005 app_directories.py:52(load_template_source)

After:

Stats:
         904358 function calls (864935 primitive calls) in 1.123 seconds

   Ordered by: internal time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
     1017    0.050    0.000    0.050    0.000 {method 'recv' of '_socket.socket' objects}
     8085    0.048    0.000    0.048    0.000 {open}
   158630    0.040    0.000    0.040    0.000 {isinstance}
   2468/1    0.032    0.000    0.888    0.888 base.py:740(render)
48746/40126    0.032    0.000    0.141    0.000 {getattr}
8710/7896    0.028    0.000    0.063    0.000 base.py:373(__setattr__)
6884/6587    0.027    0.000    0.135    0.000 base.py:667(_resolve_lookup)
    30091    0.020    0.000    0.029    0.000 {hasattr}
29370/27753    0.019    0.000    0.070    0.000 encoding.py:54(force_unicode)
       18    0.019    0.001    0.019    0.001 {method 'execute' of 'psycopg2._psycopg.cursor' objects}
     6618    0.018    0.000    0.020    0.000 __init__.py:160(__getattr__)

Attachments (1)

#17516-memoized_safe_join.diff (776 bytes) - added by Kronuz 4 years ago.

Download all attachments as: .zip

Change History (3)

Changed 4 years ago by Kronuz

comment:1 Changed 4 years ago by anonymous

  • Needs documentation unset
  • Needs tests unset
  • Patch needs improvement unset
  • Resolution set to wontfix
  • Status changed from new to closed

I'm wontfixing this. If safe join is that common in your application you can memoize it yourself, I do not think it is sufficiently common in django code as to justify the memory increase (remember: memoization is a memory/speed tradeoff) for all users.

comment:2 Changed 4 years ago by Alex

That was me.

Note: See TracTickets for help on using tickets.
Back to Top