test --failfast doesn't stop on errors in _pre_setup()

[slinkp@…]

--failfast stops on the first failure, but doesn't stop on the first error.
I think it should.

[anonymous]

It does in my experience:

python manage.py test --failfast
Creating test database for alias 'default'...
........................................................................................................................................................................................................................................................................................................................E
======================================================================
ERROR: test_active_for_...[details omitted]
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/kmtracey/projects/[details omitted]
AttributeError: 'module' object has no attribute [details omitted]

----------------------------------------------------------------------
Ran 313 tests in 11.486s

FAILED (errors=1)
Destroying test database for alias 'default'...

(This is with roughly 1.3 level.)

[anonymous]

(A full test run here would have run 512 tests, not stopped at 313.)

[Paul Winkler]

Sorry, I mis-reported. I've updated the title, though Trac won't let me edit the description (maybe I wasn't signed in before).

The real problem was that if TestCase._pre_setup() raises an exception, all tests are still run but reported as errors.

I did notice that exceptions raised by TestCase.setUp() *will* exit on the first error with --failfast.
So far, _pre_setup() is the only place I've observed this problem with failfast.

FWIW, the way that I discovered this was by having a multi-database configuration that worked on 1.2.3 but apparently wasn't compatible with the introduction of the TEST_DEPENDENCIES setting in 1.2.4.

Demonstration: Drop this in your app's tests.py and run tests:

class SimpleTest(TestCase):


    def _pre_setup(self):
        super(SimpleTest, self)._pre_setup()
        raise Exception("oh nooo")

    def test_aaa_ok(self):
        pass

    def test_bbb_ok(self):
        pass

    def test_ccc_nameerror(self):
        return foo

    def test_ddd_zerodivisionerror(self):
        return 3 / 0

    def test_eee_failing(self):
        self.assertEqual(1, 2)

    def test_zzz_ok(self):
        pass

I see 6 errors reported, even with --failfast:

$ django-admin.py test --failfast testapp

... snip ...

Ran 0 tests in 0.001s

FAILED (errors=6)
Destroying test database 'default'...

[Paul Winkler]

Re-closing. I just tested upgrading to Django 1.3 and it has apparently fixed this problem already.