Document transaction leak possiblity in TestCase

[Jonas Haag]

Coming from ​https://groups.google.com/forum/#!topic/django-developers/t1xaSmNMuKQ, I found a bug in the test database setup. This is probably related to #24080 but I'm not 100% sure.

The error can be shown with a specific series of skipped and non-skipped tests. I attached a minimal example project that demonstrates the issue.

This used to work in Django 1.7, and doesn't work in any Django versions later than 1.7. It is caused, according to a Git bisect, by ​https://github.com/django/django/commit/da9fe5c717c179a9e881a40efd3bfe36a21bf4a6.

[Jonas Haag]

Minimal test project

[Tim Graham]

If you raise SkipTest in setUpClass() after calling super(), then I'd think the database setup in TestCase.setUpClass() happens but not the teardown in TestCase.tearDownClass? If so, I'm not sure there's much Django can do about the issue. Any suggestions? Can you raise SkipTest before calling super()?

[Carl Meyer]

Without digging into it more I don't know if this is something that would need to be fixed in Django or in stdlib unittest, but ISTM that ideally even if SkipTest is raised after setUpClass, the testing framework ought to be responsible to make sure that tearDownClass is still called regardless.

[Jonas Haag]

I agree with carljm.

[Jonas Haag]

From the Python docs:

If an exception is raised during a setUpClass then the tests in the class are not run and the tearDownClass is not run. Skipped classes will not have setUpClass or tearDownClass run. If the exception is a SkipTest exception then the class will be reported as having been skipped instead of as an error.

Looks like nothing's wrong with Django here.

[Tim Graham]

How about a ​documentation patch to raise awareness?

[Tim Graham <timograham@…>]

In a5619f7e:

Fixed #25860 -- Documented a transaction leak possiblity in TestCase.

Thanks Jonas Haag for report and review.

[Tim Graham <timograham@…>]

In 35b5e11:

[1.9.x] Fixed #25860 -- Documented a transaction leak possiblity in TestCase.

Thanks Jonas Haag for report and review.

Backport of a5619f7ed3bc2ca07b428fa0abfaaff088378f48 from master