Django's test suite does not pass in reverse order

[Tim Graham]

runtests.py now has a --reverse flag to run tests in reverse order (#23742). Unfortunately, Django's test suite doesn't pass this way which means some tests have side effects. We should see if we can fix this.

[Diego Guimarães]

initial patch

[Diego Guimarães]

This initial patch fixes most of the errors when running the Django test suite with --reverse. The common problems I've found so far, was basically that some tests were performing operations that in a reverse way triggered side effects, like a smtp sever that was stopped first and had issues with Thread when tried to start it again, and cache that was closed by a previous test, etc.

There is still one test failing with issues handling staticfiles cache.

FAIL: test_template_tag_denorm (staticfiles_tests.tests.TestCollectionCachedStorage)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/diego/djangoproject/django/tests/staticfiles_tests/tests.py", line 526, in test_template_tag_denorm
      self.assertIn(b'url("img/relative.acae32e4532b.png', content)
      AssertionError: 'url("img/relative.acae32e4532b.png' not found in '@import url("../cached/styles.bb84a0240107.css");\nbody {\n    background: #d3d6d8 url("img/relative.deploy12345.png");\n}\n'

To reproduce this error: ./runtests.py --reverse staticfiles_tests.tests

[Tim Graham]

I saw your paste in IRC that you had a proposed fix for the last issue. Any chance you could make a pull request with the fixes? That makes review a lot easier. Thanks!

[Diego Guimarães]

Yes, sure. I'll create the pull request and update the ticket.

[Diego Guimarães]

Pull request created ​https://github.com/django/django/pull/3679

[Tim Graham <timograham@…>]

In d05709821cf6ff77e00871d6fac3eb43356d44d1:

Refs #23947 -- Made a modeladmin test cleanup after itself.

[Tim Graham <timograham@…>]

In e32a8a99d99c3d6c3015e220f6719f6ddc48b4f1:

Refs #23947 -- Isolated a mail test.

[Tim Graham <timograham@…>]

In 106dde91e4dc30766db00d4ea21b4f954f49e83e:

Refs #23947 -- Fixed introspection test that had a side effect.

[Tim Graham <timograham@…>]

In 40c60efecc0fc73f0b2320b44d684586b52ee799:

Refs #23947 -- Isolated some cache tests.

Thanks Diego Guimarãesi and Florian Apolloner.

[Tim Graham <timograham@…>]

In 9f427617e4559012e1c2fd8fce46cbe225d8515d:

Refs #23947 -- Worked around a bug in Python that prevents deprecation warnings from appearing in tests.

[Diego Guimarães]

This new patch tries to solve the staticfiles test problem. I added a tearDown to the TestHashedFiles mixin, where it clears the hashed_files after each test, avoiding side effects and unexpected changes among them. Is this approach valid?

[Tim Graham <timograham@…>]

In fd60e6c8878986a102f0125d9cdf61c717605cf1:

Refs #23947 -- Prevented staticfiles tests from having side effects.

[Tim Graham]

SQLite is passing in reverse, but it looks like there is more work to do on PostgreSQL. I see the following failures:

admin_views.tests.AdminViewDeletedObjectsTest.test_generic_relations
generic_relations_regress.tests.GenericRelationTests.test_annotate
migrations.test_operations.OperationTests.test_alter_field_m2m
queries.tests.SubqueryTests.test_related_sliced_subquery

[Diego Guimarães]

Ok, I'll setup the test suite to run with PostgresSQL and debug those tests.

[Diego Guimarães]

Created a new PR with the fixes to PostgreSQL ​https://github.com/django/django/pull/3726

[Tim Graham <timograham@…>]

In d8182f294aa0a8dfb3ea4611a9adb7a748a36bf4:

Refs #23947 -- Fixed admin_views test execution order dependency.

[Tim Graham <timograham@…>]

In 222699d224ee7851e1cab5ecc029cf83836b882b:

Refs #23947 -- Fixed queries test execution order dependency.

Specify an id to avoid conflict with objects created in setUpTestData.

[Tim Graham <timograham@…>]

In c17d821fa74b01205f3a2fe42a96dfe804de3df3:

Refs #23947 -- Improved migrations tests table cleanup.

Copied technique from schema tests.

[Tim Graham]

I think we're finished here for now. Thanks Diego!

[Tim Graham <timograham@…>]

In 5c43fd48258eb977e6e611310e12a921d8470944:

Isolated some cache tests; refs #23947.

This reverts a change made in 40c60efecc0fc73f0b2320b44d684586b52ee799
which was incorrect and caused CacheKeyWarnings.