django test command runs wrong tests if test module has no tests

[Chris Jerdonek]

The Django test command seems to behave incorrectly if passed a test module that has no tests.

For example, if passed test label foo.bar corresponding to module foo.bar and foo.bar has no tests, the command seems to discover and run all tests in foo.*, which is more than it should.

This behavior made it much harder to troubleshoot the fact that one of my test modules mistakenly had no tests. If given a module with no tests, Django should report back with a message like "0 tests found in ---" or simply run no tests.

[Preston Timmons]

Hi Chris,

The behavior you describe is a bug in how Python's unittest discovery works, but it isn't an issue with Django's test runner than I'm aware of.

There's even a test to ensure that:

​https://github.com/django/django/blob/master/tests/test_runner/test_discover_runner.py#L92

Are you using a custom test runner by any chance?

I can't reproduce the problem in my own testing. If you can give me a case where it discovers incorrectly, I'll take another look.

[Chris Jerdonek]

Thanks for the response. No, I'm not using a custom test runner. But it looks like this was fixed in 1.7? I'm using 1.6 as the bug report indicates. And indeed, I don't have the fix when I manually checked the Django code I'm using.

[Preston Timmons]

Although we added the test case in the 1.7 cycle, this wasn't a bug in 1.6 that I'm aware of. That's the version I'm using to try to reproduce what you're seeing.

I'm using the following layout to test:

foo/
  __init__.py
  bar.py
  car.py

The bar.py file is empty and car.py has:

import unittest

from django.test import TestCase


class SampleTest(unittest.TestCase):

    def test_one(self):
        assert True


class DjangoTest(TestCase):

    def test_one(self):
        assert True

Running python manage.py test foo.bar returns:

----------------------------------------------------------------------
Ran 0 tests in 0.000s

OK

Is this a correct interpretation of the scenario your describing?

[Chris Jerdonek]

Thanks for putting the test case together! I ran it and got 0 tests, too. But then I realized it's not discovering tests because the filenames don't match the default test pattern. When I renamed the files to test_bar.py and test_car.py, I was able to reproduce the problem. (Sorry, I probably should have said foo.test_bar in my original post above.)

[Preston Timmons]

Yes, I can reproduce this now. That explains why the existing test reports back without error as well.

[Preston Timmons]

After some further testing I realized this is fixed in 1.7. That's because we now only run discovery if the test label is a module or directory.

[Chris Jerdonek]

So just to clarify, what changes need to be made in what versions? From the comments, it seems the test needs to be fixed in 1.7 so that it has a chance of failing, and the bug can be fixed in versions prior to 1.7.

(By the way, I don't quite understand the explanation for why it's working in 1.7. In the test case, the test label is a module, which you say will cause discovery to be run. But you also said that Python's unittest discovery is what has the bug?)

[Carl Meyer]

I don't think the severity of this bug justifies backporting, so the relevant question is what still needs fixing in 1.7.

[Preston Timmons]

Sorry for not being clear. I meant package, not module.

No change is needed for 1.7. The behavior described was fixed at the same time as #21206.

When the test runner is given a label, two things happen.

1) Tests are first loaded using unittest loadTestsFromName
2) If none are found, and the path is a package or folder, unittest discovery is run.

Before #21206 we didn't check if a path was a package or folder. That's what exposed the buggy unittest discovery behavior.