Make unittest ignore Django frames in tracebacks

[Adam Johnson]

Make unittest ignore Django frames in tracebacks

The module containing Django’s custom assert methods (django.test.testcases) does not have the __unittest attribute. This makes unit test unnecessarily display frames in those methods when tests fail, and opens the debugger there when using --pdb.

unittest ignores stack frames in modules with __unittest defined (​source, ​example). If add this in django.test.testcases, we get the same behaviour.

Take these two failing tests:

from django.test import SimpleTestCase


class ExampleTests(SimpleTestCase):
    def test_1_unittest_assert(self):
        x = 0
        self.assertTrue(x)

    def test_2_django_assert(self):
        y = "[]"
        self.assertJSONEqual("{}", y)

Running them gives:

./manage.py test
Found 2 test(s).
System check identified no issues (0 silenced).
FF
======================================================================
FAIL: test_1_unittest_assert (example.tests.ExampleTests.test_1_unittest_assert)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/.../example/tests.py", line 7, in test_1_unittest_assert
    self.assertTrue(x)
AssertionError: 0 is not true

======================================================================
FAIL: test_2_django_assert (example.tests.ExampleTests.test_2_django_assert)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/.../example/tests.py", line 11, in test_2_django_assert
    self.assertJSONEqual("{}", y)
  File "/.../django/test/testcases.py", line 1020, in assertJSONEqual
    self.assertEqual(data, expected_data, msg=msg)
AssertionError: {} != []

----------------------------------------------------------------------
Ran 2 tests in 0.001s

FAILED (failures=2)

Notice how the only frame shown for test_1_unittest_assert is in the test itself, whilst test_2_django_assert contains the frame from within Django. This is useless noise for the user, which is why unittest hides its own frames

Also, when running tests with --pdb, pdb opens within the test for test_1_unittest_assert.
We can run x to check the value of variable x and c to continue:

$ ./manage.py test --pdb
Found 2 test(s).
System check identified no issues (0 silenced).
F
Opening PDB: AssertionError('0 is not true')
> /.../example/tests.py(7)test_1_unittest_assert()
-> self.assertTrue(x)
(Pdb) x
0
(Pdb) c

But in test_2_django_assert, pdb opens within the assertJSONEqual method.
We need to run u (up) to traverse the stack to the test method before we can do any meaningful debugging.
(Some assert methods require multiple u commands.)

F
Opening PDB: AssertionError('{} != []')
> /.../django/test/testcases.py(1020)assertJSONEqual()
-> self.assertEqual(data, expected_data, msg=msg)
(Pdb) u
> /.../example/tests.py(11)test_2_django_assert()
-> self.assertJSONEqual("{}", y)
(Pdb) y
'[]'
(Pdb) c

[Natalia Bidart]

Thank you Adam for this ticket!

I agree with your analysis. For posterity, the ​referenced implementation looks like this:

    def _is_relevant_tb_level(self, tb):
        return '__unittest' in tb.tb_frame.f_globals

[nessita <124304+nessita@…>]

In 95827452:

Fixed #35622 -- Made unittest ignore Django assertions in traceback frames.

Co-authored-by: Natalia <124304+nessita@…>