Opened 11 years ago

Closed 7 years ago

Last modified 3 years ago

#20238 closed New feature (fixed)

Support multiple open connections in LiveServerTestCase

Reported by: jwxiao@… Owned by: Nadege
Component: Testing framework Version: dev
Severity: Normal Keywords:
Cc: Triage Stage: Accepted
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

Run a selenium test with LiveServerTestCase. Django starts StoppableWSGIServer. The StoppableWSGIServer is not multithread support. When the codes to be tested call urllib.urlopen to the same server (ie: StoppableWSGIServer) the selenium test will hang and fail because of time-out.

Change History (21)

comment:1 by Julien Phalip, 11 years ago

Resolution: needsinfo
Status: newclosed

Thank you for the report, however there is too little information here to identify if or where there is an issue. Could you please provide a detailed explanation of how to reproduce the issue you're facing, or even better, could you provide some code or a test case? Thanks!

comment:2 by Julien Phalip, 11 years ago

Just a clarification: please feel free to reopen this ticket if you can provide more detailed information.

comment:3 by Joeri Bekker, 11 years ago

Although this is not my ticket, I just ran into the same issue.

The problem occurs typically when, within a request/response cycle, another request is done to the same StoppableWSGIServer. Now, the first request can never finish because the second request is waiting for the StoppableWSGIServer to respond (but it never will because it's waiting for the first request...)

# view
def index(self, request):
    if request.GET.get('inner_request') == 'true':
        # Within a request/response, perform another request to the same live server.
        # This one get's stuck.
        urllib.urlopen(request.build_absolute_uri())
    return HttpResponse()

# test
class LiveServerTests(LiveServerTestCase):
    def test_request_within_request(self):
        response = requests.get(self.live_server_url, params={'inner_request': 'true'})

        # This code is never reached (or the live server throws a timeout)
        self.assertEqual(response.status_code, 200)

The use case is somewhat fictional.

Last edited 11 years ago by Joeri Bekker (previous) (diff)

comment:4 by bdrennan@…, 10 years ago

Resolution: needsinfo
Status: closednew

comment:5 by Aymeric Augustin, 10 years ago

Resolution: wontfix
Status: newclosed

Considering that:

  • "the use case is somewhat fictional",
  • the ticket was reopened without any additional information,
  • more threading will certainly create more race conditions on shutdown, especially when it comes to the database connections — it's taken months to eliminate most from LiveServerTestCase, and I'm sure there are still some left,

I don't think we should add threading without more compelling arguments.

comment:6 by Aymeric Augustin, 7 years ago

Resolution: wontfix
Status: closednew

I'm hitting this problem, which means I have a concrete use case to offer.

I have a view that generates a PDF document by converting a HTML document served by the same server (with phantomjs). It works as follows:

  • user makes a HTTP request for a PDF document
  • pdf_view() starts a phantomjs subprocess
  • phantomjs makes a HTTP request for the equivalent HTML document
  • html_view() renders a HTML template
  • phantomjs gets the HTML, converts it to PDF, and returns that
  • pdf_view() returns the PDF

I cannot use that view with the LiveServerTestCase because it's single threaded. The thread runs pdf_view(), waits for phantomjs, but there's no thread available to run html_view(), so the test blocks there.

comment:7 by Aymeric Augustin, 7 years ago

The following monkey-patch does the job for me.

import socketserver

from django.test import testcases


class ThreadedWSGIServer(socketserver.ThreadingMixIn, testcases.WSGIServer):
    """
    Make Django's live server test case multi-threaded.

    See https://code.djangoproject.com/ticket/20238#comment:6

    """


testcases.WSGIServer = ThreadedWSGIServer

I'm using the PostgreSQL database backend.

I don't think this proposal is sufficient for in-memory SQLite. A WSGIHandler subclass that replaces the database connection for each new thread by the database connection from the main thread is likely needed.


By the way, this piece of code in django.core.servers.basehttp:

    if threading:
        httpd_cls = type(str('WSGIServer'), (socketserver.ThreadingMixIn, WSGIServer), {})
    else:
        httpd_cls = WSGIServer

could be replaced with:

    httpd_cls = ThreadedWSGIServer if threading else WSGIServer

assuming the definition of ThreadedWSGIServer above.

There's no reason (that I can think of) to define a class dynamically like that here. Saving a couple lines isn't worth the obfuscation.

comment:8 by Aymeric Augustin, 7 years ago

jwxiao, I'm sorry for not understanding your report originally.

comment:9 by Tim Graham, 7 years ago

Summary: StoppableWSGIServer is not multithread supportSupport multiple open connections in LiveServerTestCase
Triage Stage: UnreviewedAccepted
Type: BugNew feature

This was also requested in #25970.

comment:10 by Nadege, 7 years ago

Owner: changed from nobody to Nadege
Status: newassigned

comment:11 by Tim Graham, 7 years ago

Support for multiple open connections is also needed if you want to write a test that uses multiple browser instances as demonstrated in #27665 (closed as duplicate).

comment:12 by Claude Paroz, 7 years ago

Has patch: set
Needs tests: set
Version: 1.4master

I simply put Aymeric suggestion to a patch here: https://github.com/django/django/pull/7768
If someone has an idea for a test, please help.

comment:13 by Aymeric Augustin, 7 years ago

Nadege also prepared a patch but hasn't submitted a PR yet. I was planning to take a look at her work after the holidays.

comment:14 by Claude Paroz, 7 years ago

Oh, sorry, I didn't intend to duplicate her work.

comment:15 by Nadege, 7 years ago

Needs tests: unset

We actually made the sames changes, I added some tests as well.
They don't fail but hang whithout the patch.
I tried to do a test that could show potential database issue because aymeric sugested that might be a problem
but things seems good.

Patch is here https://github.com/django/django/pull/7832

comment:16 by Tim Graham, 7 years ago

Unless you're motivated to solve the Python 2 failures, we can target this patch for Django 2.0 which drops support for Python 2. The 1.11 feature freeze is Friday.

comment:17 by Nadege, 7 years ago

I'm good with targeting Django 2.0

comment:18 by Claude Paroz, 7 years ago

Patch needs improvement: set

comment:19 by Tim Graham, 7 years ago

Patch needs improvement: unset

comment:20 by Tim Graham <timograham@…>, 7 years ago

Resolution: fixed
Status: assignedclosed

In bece8378:

Fixed #20238 -- Added threading support to LiveServerTestCase.

comment:21 by Chris Jerdonek, 3 years ago

FYI, I just filed #32416, which lists a possible regression caused by this fix.

Note: See TracTickets for help on using tickets.
Back to Top