Opened 4 years ago

Closed 3 months ago

#20238 closed New feature (fixed)

Support multiple open connections in LiveServerTestCase

Reported by: jwxiao@… Owned by: Nadege
Component: Testing framework Version: master
Severity: Normal Keywords:
Cc: Triage Stage: Accepted
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no


Run a selenium test with LiveServerTestCase. Django starts StoppableWSGIServer. The StoppableWSGIServer is not multithread support. When the codes to be tested call urllib.urlopen to the same server (ie: StoppableWSGIServer) the selenium test will hang and fail because of time-out.

Change History (20)

comment:1 Changed 4 years ago by Julien Phalip

Resolution: needsinfo
Status: newclosed

Thank you for the report, however there is too little information here to identify if or where there is an issue. Could you please provide a detailed explanation of how to reproduce the issue you're facing, or even better, could you provide some code or a test case? Thanks!

comment:2 Changed 4 years ago by Julien Phalip

Just a clarification: please feel free to reopen this ticket if you can provide more detailed information.

comment:3 Changed 4 years ago by Joeri Bekker

Although this is not my ticket, I just ran into the same issue.

The problem occurs typically when, within a request/response cycle, another request is done to the same StoppableWSGIServer. Now, the first request can never finish because the second request is waiting for the StoppableWSGIServer to respond (but it never will because it's waiting for the first request...)

# view
def index(self, request):
    if request.GET.get('inner_request') == 'true':
        # Within a request/response, perform another request to the same live server.
        # This one get's stuck.
    return HttpResponse()

# test
class LiveServerTests(LiveServerTestCase):
    def test_request_within_request(self):
        response = requests.get(self.live_server_url, params={'inner_request': 'true'})

        # This code is never reached (or the live server throws a timeout)
        self.assertEqual(response.status_code, 200)

The use case is somewhat fictional.

Last edited 4 years ago by Joeri Bekker (previous) (diff)

comment:4 Changed 3 years ago by bdrennan@…

Resolution: needsinfo
Status: closednew

comment:5 Changed 3 years ago by Aymeric Augustin

Resolution: wontfix
Status: newclosed

Considering that:

  • "the use case is somewhat fictional",
  • the ticket was reopened without any additional information,
  • more threading will certainly create more race conditions on shutdown, especially when it comes to the database connections — it's taken months to eliminate most from LiveServerTestCase, and I'm sure there are still some left,

I don't think we should add threading without more compelling arguments.

comment:6 Changed 6 months ago by Aymeric Augustin

Resolution: wontfix
Status: closednew

I'm hitting this problem, which means I have a concrete use case to offer.

I have a view that generates a PDF document by converting a HTML document served by the same server (with phantomjs). It works as follows:

  • user makes a HTTP request for a PDF document
  • pdf_view() starts a phantomjs subprocess
  • phantomjs makes a HTTP request for the equivalent HTML document
  • html_view() renders a HTML template
  • phantomjs gets the HTML, converts it to PDF, and returns that
  • pdf_view() returns the PDF

I cannot use that view with the LiveServerTestCase because it's single threaded. The thread runs pdf_view(), waits for phantomjs, but there's no thread available to run html_view(), so the test blocks there.

comment:7 Changed 6 months ago by Aymeric Augustin

The following monkey-patch does the job for me.

import socketserver

from django.test import testcases

class ThreadedWSGIServer(socketserver.ThreadingMixIn, testcases.WSGIServer):
    Make Django's live server test case multi-threaded.



testcases.WSGIServer = ThreadedWSGIServer

I'm using the PostgreSQL database backend.

I don't think this proposal is sufficient for in-memory SQLite. A WSGIHandler subclass that replaces the database connection for each new thread by the database connection from the main thread is likely needed.

By the way, this piece of code in django.core.servers.basehttp:

    if threading:
        httpd_cls = type(str('WSGIServer'), (socketserver.ThreadingMixIn, WSGIServer), {})
        httpd_cls = WSGIServer

could be replaced with:

    httpd_cls = ThreadedWSGIServer if threading else WSGIServer

assuming the definition of ThreadedWSGIServer above.

There's no reason (that I can think of) to define a class dynamically like that here. Saving a couple lines isn't worth the obfuscation.

comment:8 Changed 6 months ago by Aymeric Augustin

jwxiao, I'm sorry for not understanding your report originally.

comment:9 Changed 6 months ago by Tim Graham

Summary: StoppableWSGIServer is not multithread supportSupport multiple open connections in LiveServerTestCase
Triage Stage: UnreviewedAccepted
Type: BugNew feature

This was also requested in #25970.

comment:10 Changed 6 months ago by Nadege

Owner: changed from nobody to Nadege
Status: newassigned

comment:11 Changed 5 months ago by Tim Graham

Support for multiple open connections is also needed if you want to write a test that uses multiple browser instances as demonstrated in #27665 (closed as duplicate).

comment:12 Changed 5 months ago by Claude Paroz

Has patch: set
Needs tests: set
Version: 1.4master

I simply put Aymeric suggestion to a patch here:
If someone has an idea for a test, please help.

comment:13 Changed 5 months ago by Aymeric Augustin

Nadege also prepared a patch but hasn't submitted a PR yet. I was planning to take a look at her work after the holidays.

comment:14 Changed 5 months ago by Claude Paroz

Oh, sorry, I didn't intend to duplicate her work.

comment:15 Changed 5 months ago by Nadege

Needs tests: unset

We actually made the sames changes, I added some tests as well.
They don't fail but hang whithout the patch.
I tried to do a test that could show potential database issue because aymeric sugested that might be a problem
but things seems good.

Patch is here

comment:16 Changed 5 months ago by Tim Graham

Unless you're motivated to solve the Python 2 failures, we can target this patch for Django 2.0 which drops support for Python 2. The 1.11 feature freeze is Friday.

comment:17 Changed 5 months ago by Nadege

I'm good with targeting Django 2.0

comment:18 Changed 4 months ago by Claude Paroz

Patch needs improvement: set

comment:19 Changed 4 months ago by Tim Graham

Patch needs improvement: unset

comment:20 Changed 3 months ago by Tim Graham <timograham@…>

Resolution: fixed
Status: assignedclosed

In bece8378:

Fixed #20238 -- Added threading support to LiveServerTestCase.

Note: See TracTickets for help on using tickets.
Back to Top