Opened 4 years ago

Closed 4 years ago

#15689 closed (fixed)

test --failfast doesn't stop on errors in _pre_setup()

Reported by: slinkp@… Owned by: nobody
Component: Testing framework Version: 1.2
Severity: Keywords:
Cc: Triage Stage: Unreviewed
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: UI/UX:

Description

--failfast stops on the first failure, but doesn't stop on the first error.
I think it should.

Change History (4)

comment:1 Changed 4 years ago by anonymous

  • Needs documentation unset
  • Needs tests unset
  • Patch needs improvement unset
  • Resolution set to worksforme
  • Status changed from new to closed

It does in my experience:

python manage.py test --failfast
Creating test database for alias 'default'...
........................................................................................................................................................................................................................................................................................................................E
======================================================================
ERROR: test_active_for_...[details omitted]
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/kmtracey/projects/[details omitted]
AttributeError: 'module' object has no attribute [details omitted]

----------------------------------------------------------------------
Ran 313 tests in 11.486s

FAILED (errors=1)
Destroying test database for alias 'default'...

(This is with roughly 1.3 level.)

comment:2 Changed 4 years ago by anonymous

(A full test run here would have run 512 tests, not stopped at 313.)

comment:3 Changed 4 years ago by slinkp

  • Resolution worksforme deleted
  • Status changed from closed to reopened
  • Summary changed from test --failfast doesn't stop on errors to test --failfast doesn't stop on errors in _pre_setup()

Sorry, I mis-reported. I've updated the title, though Trac won't let me edit the description (maybe I wasn't signed in before).

The real problem was that if TestCase._pre_setup() raises an exception, all tests are still run but reported as errors.

I did notice that exceptions raised by TestCase.setUp() *will* exit on the first error with --failfast.
So far, _pre_setup() is the only place I've observed this problem with failfast.

FWIW, the way that I discovered this was by having a multi-database configuration that worked on 1.2.3 but apparently wasn't compatible with the introduction of the TEST_DEPENDENCIES setting in 1.2.4.

Demonstration: Drop this in your app's tests.py and run tests:

class SimpleTest(TestCase):


    def _pre_setup(self):
        super(SimpleTest, self)._pre_setup()
        raise Exception("oh nooo")

    def test_aaa_ok(self):
        pass

    def test_bbb_ok(self):
        pass

    def test_ccc_nameerror(self):
        return foo

    def test_ddd_zerodivisionerror(self):
        return 3 / 0

    def test_eee_failing(self):
        self.assertEqual(1, 2)

    def test_zzz_ok(self):
        pass

I see 6 errors reported, even with --failfast:

$ django-admin.py test --failfast testapp

... snip ...

Ran 0 tests in 0.001s

FAILED (errors=6)
Destroying test database 'default'...

comment:4 Changed 4 years ago by slinkp

  • Resolution set to fixed
  • Status changed from reopened to closed

Re-closing. I just tested upgrading to Django 1.3 and it has apparently fixed this problem already.

Note: See TracTickets for help on using tickets.
Back to Top