Django

Code

Ticket #3357 (closed: wontfix)

Opened 3 years ago

Last modified 9 months ago

Make Django's server optionally multithreaded

Reported by: treborhudson@gmail.com Assigned to: adrian
Milestone: Component: django-admin.py runserver
Version: SVN Keywords: post10
Cc: treborhudson@gmail.com, mir@noris.de, flori@n-schlachter.de, onecreativenerd, Almad Triage Stage: Design decision needed
Has patch: 1 Needs documentation: 0
Needs tests: 0 Patch needs improvement: 0

Description

See discussion here: http://groups.google.com/group/django-developers/browse_thread/thread/04552616b81c115b

I've started a patch to be attached to this message momentarily...

Attachments

threading.patch (2.1 kB) - added by treborhudson@gmail.com on 01/23/07 23:10:56.
threading.2.patch (2.1 kB) - added by Rob Hudson <treborhudson@gmail.com> on 01/26/07 17:35:30.
Same as last, but added "New in Development Version" to docs.
threading.3.patch (2.0 kB) - added by Camille Harang on 04/11/08 07:18:38.
Updated for 7409
devserver_multithread_1.0.patch (2.0 kB) - added by onecreativenerd on 11/26/08 05:21:35.
Multithread patch for release 1.0
devserver_multithread_trunk_r9532.patch (2.0 kB) - added by onecreativenerd on 11/26/08 05:22:37.
Multithreaded devserver patch for 1.0.1, 1.0.2 and trunk r9532

Change History

01/23/07 23:10:56 changed by treborhudson@gmail.com

  • attachment threading.patch added.

01/23/07 23:17:16 changed by anonymous

  • needs_better_patch changed.
  • has_patch set to 1.
  • needs_tests changed.
  • needs_docs changed.

01/24/07 07:49:51 changed by Graham King <graham@darkcoding.net>

This patch works for me, both in single and multi-threaded modes.

01/24/07 11:26:12 changed by Rob Hudson <treborhudson@gmail.com>

The one thing this patch may need is a limit on the number of threads allowed, or maybe a settings.MAX_THREADS setting?

01/25/07 03:25:15 changed by Graham King <graham@darkcoding.net>

I think that would over-engineering this - it's just a single-user development server.

01/25/07 13:12:18 changed by Rob Hudson <treborhudson@gmail.com>

True. You're not likely to get DOS'd on localhost. :)

01/26/07 01:40:03 changed by Michael Radziej <mir@noris.de>

  • stage changed from Unreviewed to Ready for checkin.

Nice patch! Extra kudos for putting a link to the discussion thread into the ticket, that's great for us triagers!

I put this to Ready-for-checking based on Adrian's statement on the list.

01/26/07 17:35:30 changed by Rob Hudson <treborhudson@gmail.com>

  • attachment threading.2.patch added.

Same as last, but added "New in Development Version" to docs.

02/25/07 23:11:08 changed by jacob

  • status changed from new to closed.
  • resolution set to wontfix.

The problem with this is that Django isn't even close to being thread-safe :(

As such, I'm marking this WONTFIX since this could likely just lead towards nasty threading bugs.

02/27/07 15:28:11 changed by Michael Radziej <mir@noris.de>

  • status changed from closed to reopened.
  • resolution deleted.

Jacob, I think there must be a mis-understanding or something. As far as I know, Django is thread-safe. An enquiry in django-developers gave the same result. Therefore, I'm re-opening the ticket. Can you please explain what you mean with Django "not being thread-safe"?

If it really isn't thread-safe, threaded mode for fastcgi should not be offered, either.

02/28/07 09:41:47 changed by jacob

  • status changed from reopened to closed.
  • resolution set to wontfix.

Let me rephrase slight: threading is *HARD*, and I can't be sure that it is threadsafe.

Add to that the fact that I worry this patch will encourage people to try to deploy against runserver, and I'm gonna turf this one again.

02/28/07 09:48:17 changed by mir@noris.de

What a pity. There are people who need a threaded server for development for their JavaScript? stuff, and others use fastcgi on MS Windows in threaded mode.

02/28/07 09:57:07 changed by mir@noris.de

  • cc changed from treborhudson@gmail.com to treborhudson@gmail.com, mir@noris.de.

02/28/07 11:04:20 changed by Graham King <graham@darkcoding.net>

I've been using this happily for a while now, and would really like to keep it. The patch makes it off by default, and we could document that there might be threading issues. I think the pros outweight the cons. If someone is running the dev server for a production site, despite all the warnings in the docs, they're probably already beyond our help - I don't think this patch will affect that.

If this doesn't go into Django, Rob, could you put the patch on the Wiki so I can still use it in future ?

Thanks.

03/01/07 12:36:25 changed by Matthias Urlichs <smurf@smurf.noris.de>

Umm, to second what mir said: Some people *need* that feature.

The fix for threading problems (which may or may not still exist) is to find and fix them, not to make threading harder to use (or worse).

Besides, there are other "quality" deployment scenarios for Django which depend on threading (e.g. running Django as a WSGI server under Twisted).

People deploying against runserver? Sorry, that's a non-argument. By the same token, you might want to disallow gigabit network interfaces because somebody might saturate somebody's DSL with it. (Just to pick an equally-silly example.)

03/21/08 12:56:46 changed by axiak

  • status changed from closed to reopened.
  • resolution deleted.
  • stage changed from Ready for checkin to Design decision needed.

I'm reopening for one more consideration. I don't want to open this, as I agree that the debug server should be used only for debug. *However*, I have just been working on #2070 and when DEVELOPING upload handlers, one will need to handle more than one request at once.

As an example, consider an AJAX Upload widget you're working on. The file is being uploaded, while your want requests to be handled to do something during the upload. This works on any production-ready deployment (via forking or threads), however people will complain because this won't work on the development server. I could tell them "Well of course it won't work, the development server only has a single process and single thread", but it still won't stop the influx of complains from this non-intuitive behavior ("I thought the development server should work the same? This code isn't working on the development server, why should I think it's going to work in production?").

While I don't think adding threads is necessarily the right solution, I do think that the development server should handle multiple simultaneous requests *at some level* whether it be by forking, threading, or some other method. I just can't help but think there are other features/code/projects that will use multiple simultaneous requests, and people will just assume Django is broken because it Just Won't Work on their development server they just spent 3 minutes untarring and running.

04/11/08 07:18:38 changed by Camille Harang

  • attachment threading.3.patch added.

Updated for 7409

04/11/08 07:28:29 changed by Camille Harang

Same for me axiak, i need multithreads to develop AJAX with #2070, this patch seemed to conflict with new code so i uploaded the same patch but for rev 7409. BTW sometimes i would like to manage multithreads to prevent race conditions (handle a threaded request knowing if another request is aldready established on a given page), is there any clean way to know it? I developed a tiny solution based on sockets and DB, it works but i am maybe wrong...

06/15/08 13:56:47 changed by telenieko

  • keywords set to post10.
  • stage changed from Design decision needed to Accepted.

Moving to accepted as the arguments are valid altought the concerns raised by jacob (people deploying over runserver and thread safety). with a bit of luck will discover bugs in thread safety and we can always blame users for deploying over the development server :)

But surely we can leave this for post-1.0

07/28/08 15:22:23 changed by flosch

  • cc changed from treborhudson@gmail.com, mir@noris.de to treborhudson@gmail.com, mir@noris.de, flori@n-schlachter.de.

11/16/08 01:12:06 changed by ubernostrum

  • stage changed from Accepted to Design decision needed.

Something that's been wontfix'd twice by one of Django's lead devs sure as heck isn't "Accepted".

11/26/08 05:21:35 changed by onecreativenerd

  • attachment devserver_multithread_1.0.patch added.

Multithread patch for release 1.0

11/26/08 05:22:37 changed by onecreativenerd

  • attachment devserver_multithread_trunk_r9532.patch added.

Multithreaded devserver patch for 1.0.1, 1.0.2 and trunk r9532

11/26/08 05:32:28 changed by onecreativenerd

  • cc changed from treborhudson@gmail.com, mir@noris.de, flori@n-schlachter.de to treborhudson@gmail.com, mir@noris.de, flori@n-schlachter.de, onecreativenerd.

It's now post-1.0 and I have a bunch of students in need of a multithreading dev server for a project, so I updated the patches to work with 1.0, 1.0.1, 1.0.2 and latest trunk.

The benefits to developers of a multithreaded test environment are pretty strong, as long as people are not deploying the devserver and developers are aware of the concurrency issues.

01/25/09 05:08:21 changed by Almad

  • cc changed from treborhudson@gmail.com, mir@noris.de, flori@n-schlachter.de, onecreativenerd to treborhudson@gmail.com, mir@noris.de, flori@n-schlachter.de, onecreativenerd, Almad.

I'd also like to add that single-threaded server makes some tests unmakable. Like, go check 401 with urllib2.

04/20/09 15:39:54 changed by cnlwsu@gmail.com

Is this going to be fixed? Its difficult to have to set up an apache production environment just to test my long-polling widgets.

04/26/09 14:29:04 changed by mtredinnick

  • status changed from reopened to closed.
  • resolution set to wontfix.

Until there is clear consensus from the core developers that this is worth pursuing (in other words, a discussion on django-dev with positive confirmation, not just lack of negative confirmation on the grounds that we've been through it 100 times before), I hope nobody will reopen this ticket. Our position has been made very clear and there has not been the necessary groundwork laid for anybody to just reopen this. The policy is clearly documented, please respect that.

04/26/09 15:50:50 changed by anonymous

Why implement this as a patch? Why not just create a new custom management command called "runthreaded" (as opposed to the default "runserver") and distribute that as a separate application?

06/25/09 14:40:27 changed by jaylett

As per the previous suggestion, see <http://github.com/jaylett/django_concurrent_test_server/tree/master>, a very simple (and in some ways utterly evil) app that does this, based on this patch and the runserver command; hence, consider it under the same license as Django itself.

Drop in INSTALLED_APPS, set either USE_MULTITHREADED_SERVER or USE_MULTIFORKED_SERVER to True, and ./manage.py runconcurrentserver. No warranty, but works for me on the basis of a couple of hour's use (still several times longer than it took to write). Use the forking version if threads scare you or you know your user code isn't thread safe.


Add/Change #3357 (Make Django's server optionally multithreaded)




Change Properties
Action