Changes between Initial Version and Version 1 of LoadBalancing

06/30/06 19:11:17 (16 years ago)
James Crasta <jcrasta@…>

Added initial version of document.


  • LoadBalancing

    v1 v1  
     1= Load Balancing and Django =
     3== NOTE: THIS DOCUMENT IS INCOMPLETE.  It is currently undergoing heavy editing, and I make no account for its accuracy as of this moment. -James Crasta ==
     5It all started with this IRC conversation on 6/30:
     713:55 < mattmcc> There isn't really much about load balancing that's Django-specific.
     813:55 < brantley> Mike: I don't think so.
     913:55 < Crast> mikearagua: no, but that's not really a django-specific topic anyway, it's more a webserver-config and DNS
     1013:56 < brantley> Personally, if I were to do that, I'd setup a specific box as the database-box, everything else would pretty much fall into place.
     1113:56 < mikearagua> yeah, i know, i just assume there would be particularities when doing it with a django app
     1213:56 < mattmcc> Not really.
     1313:56 < Crast> I have loadbalanced django using fastcgi and a single lighttpd in front, no proxying, and it worked... but I had no reason for it except for as proof-of-concept
     1413:57 < Crast> lighttpd does a decent job of load-balancing fcgi
     1513:57 < brantley> Crast, was that on multiple machines?
     1613:57 < Crast> brantley: yessuh, though admittedley one of them was the lighttpd machine itself
     1713:58 < Crast> I took that one out of the loop and substituted a laptop just to see how things would change
     1813:58 < brantley> Hrm, I don't understand, exactly...  Lighttpd sent the requests to fastcgi's on different servers?
     1913:58 < Crast> and the database server just ran on the lighttpd machine
     2013:59 < Crast> brantley: yep.  you can specify multiple fastCGI servers to a single path using lighttpd
     2113:59 < brantley> Damn, I didn't know that.
     2213:59 < brantley> That's pretty sweet.
     2313:59 < brantley> Yeah, Mike.  Do that.
     2413:59 < benbangert> Crast: lighty has some issues dealing with dead FastCGI processes
     2513:59 < Crast> therefore you don't get all the ugliness of HTTP proxying
     2613:59 < aurynn> yay
     2713:59 < Crast> benbangert: I don't use lighty to spawn the processes though, since they're not even running on the same machine
     2813:59 < mikearagua> i was mostly thinking about issues with session state
     2914:00 < benbangert> ie, when a Fast CGI process stops responding, it takes awhile before lightty will shut it down and restart it
     3014:00 < aurynn> automatic cpu underclocking, as well as load-based rampup fills me with complete joy.
     3114:00 < Crast> benbangert: however, you are correct
     3214:00 < benbangert> Crast: yea, but it doesn't seem to do a very good job noticing when a FCGI connection has stopped answering requests
     3314:00 < brantley> Err, I think the sessions are handled in the database with cookies, it should be fine.
     3414:00 < benbangert> I heard pound is really slick
     3514:00 < benbangert>
     3614:01 < Crast> benbangert: that's due to the natural timeout of the protocol though, it doesn't "not notice"  it just allows sufficient time to complete the request
     3714:01 < mikearagua> brantley: that's exactly what i wanted to know
     3814:01 < mikearagua> thanks. 
     3914:01 < brantley> I can't give you an exact affirmative, you should probably ask on django-users.
     4014:01 < brantley> But I'm pretty sure that's how it works.
     4114:01 < Crast> yeah, sessions are stored in the DB
     4214:02 < Crast> serialized python dicts more or less
     4314:02 < brantley> And hooked up through cookies.
     4414:02 < brantley> And as long as the browser thinks it's the same server, it'll work fine.
     4514:02 < Crast> yep
     4614:02 < mikearagua> it seemed that way to me but i could have been missing something
     4714:02 < brantley> Also: I would create a specific server that is dedicated to serving media.
     4814:03 < mikearagua> maybe i should start messing with lighttpd.  i use apache mostly
     4914:03 < brantley> With big fast hard-drives.
     5014:03 < Crast> well you do have a possible concurrency issue if two processes are using a session at the same time, whicever one writes second writes the finished session, however that'll happen even without load balancing, and it's really a non-issue
     5114:03 < brantley> That's really the database that needs to care about that, it seems to me.
     5214:03 < brantley> The first "load-balancing" should be separating the application server from the media serving.
     5314:04 < brantley> Then you go from there.
     5414:04 < Crast> I don't mind writing up a doc of my experimental results
     5514:04 < mikearagua> that would be cool
     5614:04 < brantley> That would be very progressive for enterprise Django.
     5714:04 < mikearagua> i could add to the doc as i go.  i would like to do some testing with load balancing before i deploy my app
     5814:05 < Crast> so I'll put the doc on the wiki then
     5914:05 < mattmcc> Serving the static media on a separate box is definitely the first step.  Optimizations for that server will be quite different from the ones serving up python.
     6014:05 < mikearagua> yeah, that's fairly easy though
     6114:06 < mikearagua> there are lots of docs describing how to serve static files fast
     6214:07 < mikearagua> especially with apache.  there are some issues with having dojo on a separate domain name though that i've run into so for the time being i'm running both off the same machine
     6314:07 < Crast> benbangert: there's a "disable-time" option to fastcgi servers, perhaps that needs to be tweaked for loadbalancing to be effective
     6414:07 < Crast> (on lighttpd, that is)
     67There are a few approaches to load balancing, and in many cases, they can be combined (some large websites use more than one of these strategies)
     69 1. '''Round Robin DNS''': Involves setting up multiple DNS records for the same hosting domain.
     70   * ''Advantages'': Very simple to set up, mostly does not require any special server setup at all.  Machines serving the site can be as geographically dispersed as you wish.  You can also round-robin more than just web requests, so even things like FTP, IRC, and many other services can be load-balanced this way.
     71   * ''Disadvantages'': The round-robin DNS is picked by your local DNS server serving you recursion results (usually your ISP for most end-users).  There's no nice way to "switch" users down to another server if one server is overloaded or down, and making changes to the DNS can take days to propagate across DNS caches.  It's very difficult to track if users are unable to access your site, since they may be sending requests nowhere.
     72 2. '''Reverse Proxying'''  Using a front-end server or dedicated hardware device in front of multiple web serving.
     73   * ''Advantages'': This allows a web site to be served by multiple independent web servers.  Most load balancer solutions include tools for monitoring server load and appropriately sending requests to the least-busy servers, and performing proper failover.  With appropriate hardware this can scale to very large proportions, and achieve very good uptime / consistent performance.
     74   * ''Disadvantages'': Set-up is tricky, and load-balancer hardware is often expensive.  In most set-ups, the static files have to be mirrored to all the web-servers, and every machine has to be running a full webserver stack.  Some issues pop up with things like some HTTP meta-information being lost due to the proxying.
     76 3. '''distributing dynamic content-generating processes'''
     77   * ''Advantages'':
     78   * ''Disadvantages'':
     80== Example "traditional" Load-balanced network layout ==
     81Obligatory ascii art diagram follows:
     83   ----------
     84((( INTERNET )))
     85   ----+-----
     86       |
     87       |
     88       |
     89  +-----------+
     90  |  Gateway  |-----+
     91  +-----------+     |
     92                    |
     93               +-----------------+  +-------------------+    +----------------+   
     94               |  Load Balancer  |  |  Database Server  |    |  Web Server 1  |
     95               +-----------------+  +-------------------+    +----------------+
     96                      |                      |                       |
     97Internal network -->  +----------------------+-----------------------+---------
     98                      |                      |
     99               +----------------+     +----------------+
     100               |  Web Server 2  |     |  Web Server 3  |     (more web servers)
     101               +----------------+     +----------------+
     104== About this document ==
     106This document will describe in detail one possible method of achieving load balancing with django, including discussion on solving the "static file problem."  This solution involves using the LigHTTPd web-server, FastCGI, and anywhere between two and infinity web servers.   Also, a potential solution for reverse proxying using Apache and mod_python will also be discussed, but in lesser detail.  Some performance numbers will be put out based on my independent study I have conducted using a variety of my own machines encompassing multiple platforms and operating systems, giving a general idea of the performance of load balancing.   
     108This document will also discuss potential pitfalls of using a load-balanced setup, how they occur and how to avoid them.   It is by no means comprehensive, but usually your issues will fall into one of a few categories, most of which can be solved rather simply.
Back to Top