Opened 17 years ago

Closed 15 years ago

Last modified 12 years ago

#3357 closed Uncategorized (wontfix)

Make Django's server optionally multithreaded

Reported by: treborhudson@… Owned by: Adrian Holovaty
Component: django-admin.py runserver Version: dev
Severity: Normal Keywords: post10
Cc: treborhudson@…, mir@…, flori@…, Ryan Witt, Almad Triage Stage: Design decision needed
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

See discussion here:
http://groups.google.com/group/django-developers/browse_thread/thread/04552616b81c115b

I've started a patch to be attached to this message momentarily...

Attachments (5)

threading.patch (2.1 KB ) - added by treborhudson@… 17 years ago.
threading.2.patch (2.1 KB ) - added by Rob Hudson <treborhudson@…> 17 years ago.
Same as last, but added "New in Development Version" to docs.
threading.3.patch (2.0 KB ) - added by Camille Harang 16 years ago.
Updated for 7409
devserver_multithread_1.0.patch (2.0 KB ) - added by Ryan Witt 15 years ago.
Multithread patch for release 1.0
devserver_multithread_trunk_r9532.patch (2.0 KB ) - added by Ryan Witt 15 years ago.
Multithreaded devserver patch for 1.0.1, 1.0.2 and trunk r9532

Download all attachments as: .zip

Change History (34)

by treborhudson@…, 17 years ago

Attachment: threading.patch added

comment:1 by anonymous, 17 years ago

Has patch: set

comment:2 by Graham King <graham@…>, 17 years ago

This patch works for me, both in single and multi-threaded modes.

comment:3 by Rob Hudson <treborhudson@…>, 17 years ago

The one thing this patch may need is a limit on the number of threads allowed, or maybe a settings.MAX_THREADS setting?

comment:4 by Graham King <graham@…>, 17 years ago

I think that would over-engineering this - it's just a single-user development server.

comment:5 by Rob Hudson <treborhudson@…>, 17 years ago

True. You're not likely to get DOS'd on localhost. :)

comment:6 by Michael Radziej <mir@…>, 17 years ago

Triage Stage: UnreviewedReady for checkin

Nice patch! Extra kudos for putting a link to the discussion thread into the ticket, that's great for us triagers!

I put this to Ready-for-checking based on Adrian's statement on the list.

by Rob Hudson <treborhudson@…>, 17 years ago

Attachment: threading.2.patch added

Same as last, but added "New in Development Version" to docs.

comment:7 by Jacob, 17 years ago

Resolution: wontfix
Status: newclosed

The problem with this is that Django isn't even close to being thread-safe :(

As such, I'm marking this WONTFIX since this could likely just lead towards nasty threading bugs.

comment:8 by Michael Radziej <mir@…>, 17 years ago

Resolution: wontfix
Status: closedreopened

Jacob, I think there must be a mis-understanding or something. As far as I know, Django is thread-safe. An enquiry in django-developers gave the same result. Therefore, I'm re-opening the ticket. Can you please explain what you mean with Django "not being thread-safe"?

If it really isn't thread-safe, threaded mode for fastcgi should not be offered, either.

comment:9 by Jacob, 17 years ago

Resolution: wontfix
Status: reopenedclosed

Let me rephrase slight: threading is *HARD*, and I can't be sure that it is threadsafe.

Add to that the fact that I worry this patch will encourage people to try to deploy against runserver, and I'm gonna turf this one again.

comment:10 by mir@…, 17 years ago

What a pity. There are people who need a threaded server for development for their JavaScript stuff, and others use fastcgi on MS Windows in threaded mode.

comment:11 by mir@…, 17 years ago

Cc: mir@… added

comment:12 by Graham King <graham@…>, 17 years ago

I've been using this happily for a while now, and would really like to keep it. The patch makes it off by default, and we could document that there might be threading issues. I think the pros outweight the cons.
If someone is running the dev server for a production site, despite all the warnings in the docs, they're probably already beyond our help - I don't think this patch will affect that.

If this doesn't go into Django, Rob, could you put the patch on the Wiki so I can still use it in future ?

Thanks.

comment:13 by Matthias Urlichs <smurf@…>, 17 years ago

Umm, to second what mir said: Some people *need* that feature.

The fix for threading problems (which may or may not still exist) is to find and fix them, not to make threading harder to use (or worse).

Besides, there are other "quality" deployment scenarios for Django which depend on threading (e.g. running Django as a WSGI server under Twisted).

People deploying against runserver? Sorry, that's a non-argument. By the same token, you might want to disallow gigabit network interfaces because somebody might saturate somebody's DSL with it. (Just to pick an equally-silly example.)

comment:14 by Michael Axiak, 16 years ago

Resolution: wontfix
Status: closedreopened
Triage Stage: Ready for checkinDesign decision needed

I'm reopening for one more consideration. I don't want to open this, as I agree that the debug server should be used only for debug. *However*, I have just been working on #2070 and when DEVELOPING upload handlers, one will need to handle more than one request at once.

As an example, consider an AJAX Upload widget you're working on. The file is being uploaded, while your want requests to be handled to do something during the upload. This works on any production-ready deployment (via forking or threads), however people will complain because this won't work on the development server. I could tell them "Well of course it won't work, the development server only has a single process and single thread", but it still won't stop the influx of complains from this non-intuitive behavior ("I thought the development server should work the same? This code isn't working on the development server, why should I think it's going to work in production?").

While I don't think adding threads is necessarily the right solution, I do think that the development server should handle multiple simultaneous requests *at some level* whether it be by forking, threading, or some other method. I just can't help but think there are other features/code/projects that will use multiple simultaneous requests, and people will just assume Django is broken because it Just Won't Work on their development server they just spent 3 minutes untarring and running.

by Camille Harang, 16 years ago

Attachment: threading.3.patch added

Updated for 7409

comment:15 by Camille Harang, 16 years ago

Same for me axiak, i need multithreads to develop AJAX with #2070, this patch seemed to conflict with new code so i uploaded the same patch but for rev 7409. BTW sometimes i would like to manage multithreads to prevent race conditions (handle a threaded request knowing if another request is aldready established on a given page), is there any clean way to know it? I developed a tiny solution based on sockets and DB, it works but i am maybe wrong...

comment:16 by Marc Fargas, 16 years ago

Keywords: post10 added
Triage Stage: Design decision neededAccepted

Moving to accepted as the arguments are valid altought the concerns raised by jacob (people deploying over runserver and thread safety).
with a bit of luck will discover bugs in thread safety and we can always blame users for deploying over the development server :)

But surely we can leave this for post-1.0

comment:17 by flosch, 16 years ago

Cc: flori@… added

comment:18 by James Bennett, 15 years ago

Triage Stage: AcceptedDesign decision needed

Something that's been wontfix'd twice by one of Django's lead devs sure as heck isn't "Accepted".

by Ryan Witt, 15 years ago

Multithread patch for release 1.0

by Ryan Witt, 15 years ago

Multithreaded devserver patch for 1.0.1, 1.0.2 and trunk r9532

comment:19 by Ryan Witt, 15 years ago

Cc: Ryan Witt added

It's now post-1.0 and I have a bunch of students in need of a multithreading dev server for a project, so I updated the patches to work with 1.0, 1.0.1, 1.0.2 and latest trunk.

The benefits to developers of a multithreaded test environment are pretty strong, as long as people are not deploying the devserver and developers are aware of the concurrency issues.

comment:20 by Almad, 15 years ago

Cc: Almad added

I'd also like to add that single-threaded server makes some tests unmakable. Like, go check 401 with urllib2.

comment:21 by cnlwsu@…, 15 years ago

Is this going to be fixed? Its difficult to have to set up an apache production environment just to test my long-polling widgets.

comment:22 by Malcolm Tredinnick, 15 years ago

Resolution: wontfix
Status: reopenedclosed

Until there is clear consensus from the core developers that this is worth pursuing (in other words, a discussion on django-dev with positive confirmation, not just lack of negative confirmation on the grounds that we've been through it 100 times before), I hope nobody will reopen this ticket. Our position has been made very clear and there has not been the necessary groundwork laid for anybody to just reopen this. The policy is clearly documented, please respect that.

comment:23 by anonymous, 15 years ago

Why implement this as a patch? Why not just create a new custom management command called "runthreaded" (as opposed to the default "runserver") and distribute that as a separate application?

comment:24 by James Aylett, 15 years ago

As per the previous suggestion, see <http://github.com/jaylett/django_concurrent_test_server/tree/master>, a very simple (and in some ways utterly evil) app that does this, based on this patch and the runserver command; hence, consider it under the same license as Django itself.

Drop in INSTALLED_APPS, set either USE_MULTITHREADED_SERVER or USE_MULTIFORKED_SERVER to True, and ./manage.py runconcurrentserver. No warranty, but works for me on the basis of a couple of hour's use (still several times longer than it took to write). Use the forking version if threads scare you or you know your user code isn't thread safe.

comment:25 by trevorcroft, 13 years ago

Allowing something like:
manage.py runthreadedserver

Is logically equivalent to allowing something like:
manage.py runserver

I think everyone will agree that you should keep runserver in django, therefore everyone (who is logical) would agree you could have runthreadedserver.

The arguments for disallowing one are identical to disallowing the other (fear of stupid users abusing it, and the fact that issues arise during development that are not real issues), and the arguments for allowing one are identical for allowing the other (its a great development tool).

comment:26 by egenix_viktor, 13 years ago

Severity: Normal
Type: Uncategorized

Workaround for multi-threaded development, if you really need that. It is useful if you have to debug long polling (Comet) connections or stream content to the client, which would need more than one HTTP connection open at the same time. Create a new main script in your Django project directory with the following contents and use that to run your project inside the debugger (like Wing IDE):

""" Start Django in multithreaded mode

It is NOT for deployment. It allows for debugging Django
while serving multiple requests at once in multi-threaded mode.

"""

import os, sys

HOST = '127.0.0.1'
PORT = 8000

DJANGO_PROJECT_DIR = os.path.dirname(__file__)
PARENT_DIR, DJANGO_PROJECT_NAME = DJANGO_PROJECT_DIR.rsplit(os.sep, 1)
 
sys.path.append(DJANGO_PROJECT_DIR)
os.environ['DJANGO_SETTINGS_MODULE'] = '%s.settings' % DJANGO_PROJECT_NAME

import django
import django.core.handlers.wsgi

application = django.core.handlers.wsgi.WSGIHandler()

if __name__ == '__main__':
    import wsgiref, SocketServer
    from wsgiref import simple_server
    class ThreadedWSGIServer(SocketServer.ThreadingMixIn, simple_server.WSGIServer):
        pass
    httpd = simple_server.make_server(
        HOST, PORT, application, server_class=ThreadedWSGIServer)
    print 'Serving on port %s:%d' % (HOST, PORT)
    httpd.serve_forever()

comment:27 by egenix_viktor, 13 years ago

I've also found that it is highly useful to set allow_reuse_address = True inside ThreadedWSGIServer on UNIX:

    class ThreadedWSGIServer(SocketServer.ThreadingMixIn, simple_server.WSGIServer):
        allow_reuse_address = True

Otherwise is is raising "socket already in use' errors all the time you want to start Django this way from the second try.

comment:28 by wesc, 12 years ago

Easy pickings: unset
UI/UX: unset

You can also patch this by inserting this code right before the __main__ block in manage.py:

import SocketServer
import django.core.servers.basehttp
django.core.servers.basehttp.WSGIServer = \
    type('WSGIServer',
         (SocketServer.ThreadingMixIn,
          django.core.servers.basehttp.WSGIServer,
          object),
         {})

comment:29 by anonymous, 12 years ago

Note also #16099 reversed the wontfix on this, the threading mixin and an option to disable its use were added in r16427.

Note: See TracTickets for help on using tickets.
Back to Top