SitesFrameworkTests fails when called after ContentTypesViewsSiteRelTests

[Matthijs Kooijman]

When running the test suite, I noticed that I sometimes, got a test failure:

======================================================================
FAIL: test_clear_site_cache (sites_tests.tests.SitesFrameworkTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python3.7/unittest/case.py", line 59, in testPartExecutor
    yield
  File "/usr/lib/python3.7/unittest/case.py", line 615, in run
    testMethod()
  File "/home/matthijs/docs/src/upstream/django/django/test/utils.py", line 370, in inner
    return func(*args, **kwargs)
  File "/home/matthijs/docs/src/upstream/django/tests/sites_tests/tests.py", line 162, in test_clear_site_cache
    self.assertEqual(models.SITE_CACHE, {})
  File "/usr/lib/python3.7/unittest/case.py", line 839, in assertEqual
    assertion_func(first, second, msg=msg)
  File "/usr/lib/python3.7/unittest/case.py", line 1138, in assertDictEqual
    self.fail(self._formatMessage(msg, standardMsg))
  File "/usr/lib/python3.7/unittest/case.py", line 680, in fail
    raise self.failureException(msg)
AssertionError: {2: <Site: example2.com>, 'example2.com': <Site: example2.com>} != {}
- {2: <Site: example2.com>, 'example2.com': <Site: example2.com>}
+ {}

This was a bit hard to catch, since it only occurred when running the full test suite, with parallelization, using mysql. Running just this single test, or running without parallelization did not show the problem. I suspect that when running without paralellization, or not on mysql, the timings of the tests were subtly different, so the assignment of tests over processes was slightly different, preventing this problem from manifesting.

By adding some debug output, I was able to produce a sequence of two tests, when run without parallelization, produce the same symptom, also with sqlite:

./runtests.py contenttypes_tests.test_views.ContentTypesViewsSiteRelTests.test_shortcut_view_with_site_m2m sites_tests.tests.SitesFrameworkTests.test_clear_site_cache --parallel 1

Looking at the test_clear_site_cache, it starts by asserting that the SITE_CACHE is empty. However, there is not really anything to guarantee that it is. Originally, I thought that the [pre_save handler from contrib.sites](​https://github.com/django/django/blob/22ce5d0031bd795ade081394043833e82046016c/django/contrib/sites/models.py#L103-L120) would clear the cache when the [test setup saved a Site object](​https://github.com/django/django/blob/22ce5d0031bd795ade081394043833e82046016c/tests/sites_tests/tests.py#L25-L26), but then I realized that this only clears the saved Site from the cache, not everything.

So, it seems the problem is that test_clear_site_cache assumes that the SITE_CACHE is empty at the start of the test, but there is nothing to guarantee this. It does not seem reasonable to expect all tests to clean up the SITE_CACHE after they run (since most tests should not even need to be aware that this cache exists at all), so then I guess test_clear_site_cache (or its parent class) should clear the cache before starting?

I won't be creating a PR for this one, since I really need to get back to actual productive work on my original application...

[Claude Paroz]

Thanks, great work for finding the reproducible case!

[Claude Paroz]

​PR which clears the cache before *and* after tests.

[Mariusz Felisiak <felisiak.mariusz@…>]

In 5660267e:

Fixed #31121 -- Cleared Site cache in SitesFrameworkTests.

Thanks Matthijs Kooijman for the report and analysis.