[patch] Add live test server support to test framework

path to resole this enhancement

Adding patch to subject.

Whoops.

fixed comment typo and improper print statement

Un-assigning from myself since this needs to be reviewed by a commiter.

How does this relate to the unit test system that has evolved since then?

I haven't looked at the code base since I wrote this patch. So my short answer would be that it doesn't relate at all and probably breaks.

If the new unittest system includes support for running a live server and making real requests then this patch is obsolete.

If the new unittest system doesn't have list server support then I should probably rewrite this patch to provide the support within the new framework.

Wow, Mikeal, that was a fast reply. Could you, as time permits, look whether this is still needed and put a little comment in the ticket?

Or perhaps Russell could take a short look, he should be able to spot it without looking ;-) I don't know the unit test so well. I'm only doing triage here.

The testing documentation doesn't seem to have changed since I wrote this patch.

http://www.djangoproject.com/documentation/testing/

Regardless I think I could do this a little more elegantly now. I also read about support for cherrypy's wsgi when installed which I think would be important to add since that big hack to kill the server isn't necessary using the cherrypy server. Eww, and I'm raising a string instead of a proper exception in there.

I'll try to find some time this week or next to work on this.

runtestserver command for manage.py

With the core-management.patch it would be possible to start a test server like './manage.py runserver' with the command './manage.py runtestserver'. It would be setup a test-db like with './manage test' and loaded fixtures called 'initial_data' and 'test_data'.

testing contrib to flush database during tests

Together with the core-management.patch it would be possible to flush a running live test server between tests.

Confirmed that it worksforme on SVN HEAD of unicode branch.

Just, changes made on django/test/utils.py by http_test_case.2.diff patch has to be reverted, as database syssetm changed meantime.

I'd be very very glad to see this patch being checked in ASAP as I'm using it on all my projects already.

All patches in one, with reverted utils.py

I believe this was fixed in [5912] with the addition of manage.py testserver.

This is a little different to manage.py testserver. The testserver command is about getting a server that uses test data; this ticket is about starting a server during a test case so that a test case can poke AJAX-style methods, or write a view that in turn makes HTTP calls. This is needed if you are going to run Selenium-style tests, as the test case needs to interact with a live running server.

Is there a way to see this one in trunk?

Anything needs to be done, or any way how to speed it up?

The patch attached to this bug won't work in the current trunk.

I can write another one but after seeing what happened to the last patch I'd like to know that someone is actually going to review it before it becomes outdated like the last one.

Sorry, i was trying to apply the wrong patch.

Haven't tried the newest set.

Replying to Mikeal Rogers <mikeal@osafoundation.org>:

The patch attached to this bug won't work in the current trunk. I can write another one but after seeing what happened to the last patch I'd like to know that someone is actually going to review it before it becomes outdated like the last one.

The requirements for this patch are the same as every other patch, as documented here:

http://www.djangoproject.com/documentation/contributing/#patch-style

In short:

1. Your patch needs to encompass a single idea. In this case, adding a TestCase? with a test server. The :memory: change is probably a reasonable change, but is unrelated to the ticket purpose. It should be logged and processed as separate issue.
2. Your patch needs tests. A regression test that demonstrates that the new test case works is an absolute minimum. Unless a patch is trivial, or I need the feature/fix myself, or it's difficult/impossible to test the feature (e.g., testing command line options to ./manage.py), I won't even bother looking at a patch until it has tests. I'm too busy to spend my days trying to work out how other peoples code should work. A test is how you prove you are serious about using my time.
3. Your patch needs documentation. If you have tests, documentation can wait until after an initial review - but if you don't have a test, then you're going to need to document how your feature works.
4. (Generally required, but not so much in this case since this ticket is on my watch list) - the triage team needs to review and promote the patch.

I originally wrote this a long time ago (at least in web years). I've learned a lot since then and have mde my way through a dozen test frameworks and developed a few myself. I took a few hours out of my day today and try to re-assess the situation.

IMHO TestCase? is not the right place for this. TestCase? can only define setup and teardown for a single unittest class, this needs to define something that can be a product of the test environment that encompasses multiple test classes and suites.

It's been a while since I monkeyed around with unittest, but having done this in the past I know it's not easy and is one of the primary reasons people don't like unittest.

I also haven't looked at the test infrastructure for django in a long time, just a quick glance shows that a lot seems to have changed, so I don't know if this is even possible.

Eventually I'm going to need to integrate windmill (http://windmill.osafoundation.org) tests in to my own django project. When that work is finished I'll send a proposal to the list to see if it would be a valued contribution, but doing this right will probably require a new collector based framework with more acutely defined setup/teardown.

In the meantime, if anyone wants to take on this work again ( adding a TestCase? with live server support ) they are more than welcome to it, you can email me personally if you have any implementation specific questions. I'm more than happy to help I'm just not up for wrangling unittest anymore.

Replying to Mikeal Rogers <mikeal@osafoundation.org>:

I'm more than happy to help I'm just not up for wrangling unittest anymore.

Well, it's not a problem to implement nose et al via settings.TEST_RUNNER option. However, even if SeleniumTestCase? (or equivalent) is discovered, connected to selenium proxy and browser is started successfully, tests are still not usable as they cannot connect to live server :]

(IMHO biggest flaw in unittest is absence of SkipTest?...)

Well, the last patch seems not to work in current trunk version. The structure of django/core has changed, and I see no obvious way of implementing the patch.

I've created my own patch which seems to do the job. At least it works for me. Probably you'll need Python 2.5 to run it and, of course, Linux (since the fork() method is not implemented in Windows, as I believe). I tested it only with SQLite and really don't know if it'll work with other backends.

The patch does a simple thing: after initializing environment and creating a test database, it forks and in child process a Django server is started. Being a child, it inherits the environment. In parent process, a test is started (which can include both Django TestCase and Selenium methods), and when it finishes the child is killed and the parent returns the result. The patch consists of two files, one for django/test/simple.py and another for django/test/utils.py.

One little thing to mention: I couldn't manage to make this idea work when I used :memory: database (which is default for SQLite in tests), thus I changed it to file. Then it worked.

Using this method you probably will not have to use testserver (see [5912]).

Any comments would be much appreciated.

The second part of the patch

Sorry, but I can't figure out why diff for the first file isn't displayed.

I have something working that will add this functionality to TestCase?. I'll clean up my patch, write some tests, and get something here today or tomorrow.

I've added this functionality to the TestCase? class.

The problem Mikeal mentioned with regards to in-memory databases (http://tinyurl.com/5p3g2y) was not a problem with the database, only in the attempt to access an in-memory database within a separate thread. Because of the threading, we have to special case in-memory databases.

Tests and documentation changes are included.

Could somebody clarify for me what this does that can't be handled by, or built on top of, django-admin.py/manage.py testserver? Since that's implemented as a management command, it can be called from Python code as needed, which (if this feature is still needed to address a problem with testserver) looks like it'd greatly simplify this patch.

Yeah, directly calling testserver would greatly simplify things. The main problem with that, however, is that testserver isn't stoppable. After running a test, I need to be able to stop the server, reload fixtures, etc.

I agree though, I'm not happy with how much code I'm repeating. Suggestions welcome.

I'm working on improving the way this fails when problems starting a test server are encountered. I'll have a better patch shortly.

fixes way server handles error on startup

refactor and cleanup

One thing to note about my patch, while it's on my mind:

In order to be able to stop the running test webserver, I need to set a socket timeout on waiting for requests. eg:

    def server_bind(self):
        """ Sets timeout to 1 second. """
        WSGIServer.server_bind(self)
        self.socket.settimeout(1)

But what happens when you're waiting on a request that always takes longer than 1 second? You'll continually timeout? You'll fail on a selenium wait? Decidedly something quote unquote not good.

I don't know a clean solution to this. I need some kind of timeout. Otherwise, there's no way to stop mid handle_request(). And then no way to kill the server after a test is done. It'll just hang on that handle_request(). But any timeout is subject to problems if it needs to handle a request that's going to take longer.

        # Loop until we get a stop event
        while not self._stopevent.isSet():
            httpd.handle_request()

And since there's no way to kill a python thread, there's no other way to stop this server.

We could have this timeout as a parameter and default to 1s. That way at least if people run up against the problem they can up the timeout and fix it.

Nevermind. The patch handles this.

That server_bind timeout is just for receiving requests. Once the server grabs a request it sets the socket for handling it to None.

            sock, address = self.socket.accept()
            sock.settimeout(None)
            return (sock, address)

So requests can take however long.

Actually. The timeout should be a lot smaller then. 0.001s working fine for me.

Updated devin's patch with last changes in testing environment

I borrowed the code from the latest patch here to add Django support to windmill.

http://trac.getwindmill.com/browser/trunk/windmill/authoring/djangotest.py

It hasn't been fully documented yet but the simplest case was written up for the email list.

http://groups.google.com/group/windmill-dev/browse_thread/thread/85f23d2a0d4e99