Opened 5 years ago

Closed 4 years ago

Last modified 4 years ago

#12844 closed Cleanup/optimization (wontfix)

improve lazy loading and resource usage

Reported by: wkornewald Owned by: nobody
Component: Uncategorized Version: master
Severity: Normal Keywords:
Cc: Triage Stage: Design decision needed
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

This is a general problem, but it's particularly noticeable on App Engine. Django loads a lot of modules up-front even if they're not needed. This makes the first request on a new app server instance very slow because the user has to wait for Django to do its imports. This can take five seconds if you also enable the admin UI. The attached patch is the first step towards more lazy loading and it can shave off 0.5-1s if you have a simple app. The following components are loaded on-demand:

  • forms
  • messages
  • CSRF
  • most of the file upload handling code (that can still be improved, though)
  • management

Overall, I think it's 42 modules less for very simple sites.

Personally, I'd like to see Django as a more serious alternative to small frameworks and REST solutions because I love the ORM, but there's too much overhead for simple sites that only use a small fraction of Django's features. Independent of that, I think Django's resource usage should be improved, anyway.

Attachments (1)

lazy.patch (12.3 KB) - added by wkornewald 5 years ago.
patch against r12411

Download all attachments as: .zip

Change History (11)

Changed 5 years ago by wkornewald

patch against r12411

comment:1 Changed 5 years ago by wkornewald

  • Has patch set
  • Needs documentation unset
  • Needs tests unset
  • Patch needs improvement unset

comment:2 Changed 5 years ago by Alex

  • Triage Stage changed from Unreviewed to Design decision needed

I'm am violently -1 on this patch. a) it complicates code, b) it slows down execution, c) it caters to the fact that GAE has a model that almost no one else has (seriously, we chucked CGI a while ago ;)), d) it's far better solved by using a custom import hook in your wsgi file (the same way mercurial does) in order to achieve lazy loading.

comment:3 Changed 5 years ago by wkornewald

Well, I agree that GAE is very stupid in this case. :)

Still, (a) and (b) are hardly a huge problem. Those imports should be very fast once the respective modules have been loaded for the first time.

From your (c) and (d) I get the impression that you might have misunderstood the GAE issue. This is only about the very first request to a fresh app server instance (cold start). After the cold start all subsequent requests are fast because they reuse the main() import hook. So, the problem is that the very first request has to wait for Django to load and that takes a lot of time.

comment:4 Changed 5 years ago by Alex

Yes, the purpose of the mercurial import hook is to automate lazy loading of modules (when you import something it basically returns a "fake" module, and only looking up the attribute on the module actually triggers the import): http://sayspy.blogspot.com/2009/06/lazy-imports-in-25-lines-of-code.html

comment:5 Changed 5 years ago by wkornewald

Ah, sorry. Then it seems I misunderstood (GAE provides some other import hook for the wsgi handler module). Thanks for the link. I'll take a look and see how much it can help. The biggest problem will probably be django.forms because django.db.models.fields not only imports that module, but also accesses its attributes because it uses a field class as a default parameter. We could at least move that module access into the formfield function.

comment:6 Changed 4 years ago by wkornewald

  • Cc wkornewald@… added

comment:7 Changed 4 years ago by lukeplant

  • Type set to Cleanup/optimization

comment:8 Changed 4 years ago by lukeplant

  • Severity set to Normal

comment:9 Changed 4 years ago by aaugustin

  • Easy pickings unset
  • Resolution set to wontfix
  • Status changed from new to closed
  • UI/UX unset

Closing for the reasons given by Alex.

Moving imports in functions and methods is messy, it goes against established conventions, and it can create subtle import loops. This isn't an acceptable solution.

comment:10 Changed 4 years ago by wkornewald

  • Cc wkornewald@… removed
Note: See TracTickets for help on using tickets.
Back to Top