Opened 7 years ago

Closed 5 years ago

Last modified 5 years ago

#12844 closed Cleanup/optimization (wontfix)

improve lazy loading and resource usage

Reported by: Waldemar Kornewald Owned by: nobody
Component: Uncategorized Version: master
Severity: Normal Keywords:
Cc: Triage Stage: Design decision needed
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

This is a general problem, but it's particularly noticeable on App Engine. Django loads a lot of modules up-front even if they're not needed. This makes the first request on a new app server instance very slow because the user has to wait for Django to do its imports. This can take five seconds if you also enable the admin UI. The attached patch is the first step towards more lazy loading and it can shave off 0.5-1s if you have a simple app. The following components are loaded on-demand:

  • forms
  • messages
  • CSRF
  • most of the file upload handling code (that can still be improved, though)
  • management

Overall, I think it's 42 modules less for very simple sites.

Personally, I'd like to see Django as a more serious alternative to small frameworks and REST solutions because I love the ORM, but there's too much overhead for simple sites that only use a small fraction of Django's features. Independent of that, I think Django's resource usage should be improved, anyway.

Attachments (1)

lazy.patch (12.3 KB) - added by Waldemar Kornewald 7 years ago.
patch against r12411

Download all attachments as: .zip

Change History (11)

Changed 7 years ago by Waldemar Kornewald

Attachment: lazy.patch added

patch against r12411

comment:1 Changed 7 years ago by Waldemar Kornewald

Has patch: set
Needs documentation: unset
Needs tests: unset
Patch needs improvement: unset

comment:2 Changed 7 years ago by Alex Gaynor

Triage Stage: UnreviewedDesign decision needed

I'm am violently -1 on this patch. a) it complicates code, b) it slows down execution, c) it caters to the fact that GAE has a model that almost no one else has (seriously, we chucked CGI a while ago ;)), d) it's far better solved by using a custom import hook in your wsgi file (the same way mercurial does) in order to achieve lazy loading.

comment:3 Changed 7 years ago by Waldemar Kornewald

Well, I agree that GAE is very stupid in this case. :)

Still, (a) and (b) are hardly a huge problem. Those imports should be very fast once the respective modules have been loaded for the first time.

From your (c) and (d) I get the impression that you might have misunderstood the GAE issue. This is only about the very first request to a fresh app server instance (cold start). After the cold start all subsequent requests are fast because they reuse the main() import hook. So, the problem is that the very first request has to wait for Django to load and that takes a lot of time.

comment:4 Changed 7 years ago by Alex Gaynor

Yes, the purpose of the mercurial import hook is to automate lazy loading of modules (when you import something it basically returns a "fake" module, and only looking up the attribute on the module actually triggers the import): http://sayspy.blogspot.com/2009/06/lazy-imports-in-25-lines-of-code.html

comment:5 Changed 7 years ago by Waldemar Kornewald

Ah, sorry. Then it seems I misunderstood (GAE provides some other import hook for the wsgi handler module). Thanks for the link. I'll take a look and see how much it can help. The biggest problem will probably be django.forms because django.db.models.fields not only imports that module, but also accesses its attributes because it uses a field class as a default parameter. We could at least move that module access into the formfield function.

comment:6 Changed 6 years ago by Waldemar Kornewald

Cc: wkornewald@… added

comment:7 Changed 5 years ago by Luke Plant

Type: Cleanup/optimization

comment:8 Changed 5 years ago by Luke Plant

Severity: Normal

comment:9 Changed 5 years ago by Aymeric Augustin

Easy pickings: unset
Resolution: wontfix
Status: newclosed
UI/UX: unset

Closing for the reasons given by Alex.

Moving imports in functions and methods is messy, it goes against established conventions, and it can create subtle import loops. This isn't an acceptable solution.

comment:10 Changed 5 years ago by Waldemar Kornewald

Cc: wkornewald@… removed
Note: See TracTickets for help on using tickets.
Back to Top