A test in test_runner fails in isolation

[Yosuke Yasuda]

I run tests with Python 3.4.3 but failed like this.

% ./runtests.py ./test_runner
Testing against Django installed in '/Users/yasudayousuke/oss_hack/django_dev/django_parent/django'
Creating test database for alias 'default'...
Creating test database for alias 'other'...
.....F...........ss.......................
======================================================================
FAIL: test_duplicates_ignored (test_runner.test_discover_runner.DiscoverRunnerTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/yasudayousuke/oss_hack/django_dev/django_parent/tests/test_runner/test_discover_runner.py", line 133, in test_duplicates_ignored
    self.assertEqual(single, dups)
AssertionError: 259 != 260

----------------------------------------------------------------------
Ran 42 tests in 4.421s

FAILED (failures=1, skipped=2)
Destroying test database for alias 'default'...
Destroying test database for alias 'other'...

[Tim Graham]

Bisected to ad0be620aece4dde2f94383426c284f4afef9fbc

[Claude Paroz]

I think I've at least identified the issue. When test_discover_runner is run and gis_tests is not in current test labels, the discovery process inside the test fails with the error:

RuntimeError: Model class gis_tests.geo3d.models.City3D doesn't declare an explicit app_label and either isn't in an application in INSTALLED_APPS or else was imported before its application was loaded.

Then that discovery result is not considered as duplicate, hence the test failure.

I don't know right now what's the best resolution would be, apart from recreating an embedded app structure inside test_runner or in a ../test_discovery_sample-like dir, which might be overkill just for that test.

[Claude Paroz]

Maybe with that patch:

tests/test_runner/test_discover_runner.py

@@ -121 +121 @@
         """
         Tests shouldn't be discovered twice when discovering on overlapping paths.
         """
-        single = DiscoverRunner().build_suite(["gis_tests"]).countTestCases()
-        dups = DiscoverRunner().build_suite(
-            ["gis_tests", "gis_tests.geo3d"]).countTestCases()
+        with self.modify_settings(INSTALLED_APPS={'append': 'gis_tests.geo3d'}):
+            suite = DiscoverRunner().build_suite(["gis_tests.geo3d", "gis_tests.geo3d"])
+            single = DiscoverRunner().build_suite(["gis_tests"]).countTestCases()
+            dups = DiscoverRunner().build_suite(
+                ["gis_tests", "gis_tests.geo3d"]).countTestCases()
         self.assertEqual(single, dups)
 
     def test_reverse(self):

Joseph, still interested to test that and cook a patch?

[Joseph Gordon]

Yes, still interested. I'll test it out.

[Joseph Gordon]

Looks like it fixed the assertion error. Is the next step to create a pull request (this is my first time interacting with the Django project)?

[Claude Paroz]

Yes, please.

[Joseph Gordon]

Made pull request:

​https://github.com/django/django/pull/5050

[Tim Graham <timograham@…>]

In 199a02d1:

Fixed #25110 -- Fixed a test_runner test isolation regression.

Thanks claudep for the patch.