Running tagged tests hides any with syntax errors

[Kryštof Řeháček]

When I execute tests with syntax error without any argument then it shows SyntaxError: invalid syntax as expected but when I execute tagged tests, then it shows that it ran 0 tests and it is OK and even error. So when tests with syntax executed with the --tag="some_tag" argument and they have syntax error then they complete with OK status.

[Tim Graham]

The problem may be in django.test.runner.filter_tests_by_tags().

[Daniel Hones]

It looks like this behavior is ultimately due to Python's unittest TestLoader and not Django's test framework. In case of a SyntaxError when loading a test suite, it hits ​this line, which creates a _FailedTest object from it without raising an error. Then the django.test.runner.filter_tests_by_tags() sees a _FailedTest object which won't have any tags and so it won't be run at all. Since tests are tagged using a decorator, tests in a file with a syntax error won't have any tags recognized for them anyway because the file cannot be evaluated. I don't think there's a reasonable fix for this and I'm not sure it should even be considered a bug, although the behavior is a bit misleading.

[Chris Jerdonek]

I think a possible solution for this would be to make it so that unittest.loader._FailedTest objects always match when tags are provided. That would prevent the issue of tests falsely passing due to e.g. syntax errors.

[Chris Jerdonek]

PR: ​https://github.com/django/django/pull/14195

[Mariusz Felisiak <felisiak.mariusz@…>]

In 038940cf:

Fixed #29127 -- Prevented DiscoverRunner from hiding tagged test with syntax errors.

This mades _FailedTest objects always match tags in DiscoverRunner.