Opened 5 years ago

Closed 17 months ago

#29127 closed Bug (fixed)

Running tagged tests hides any with syntax errors

Reported by: Kryštof Řeháček Owned by: Chris Jerdonek
Component: Testing framework Version: 2.0
Severity: Normal Keywords: tests, tagged-tests, SyntaxError
Cc: Triage Stage: Ready for checkin
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

When I execute tests with syntax error without any argument then it shows SyntaxError: invalid syntax as expected but when I execute tagged tests, then it shows that it ran 0 tests and it is OK and even error. So when tests with syntax executed with the --tag="some_tag" argument and they have syntax error then they complete with OK status.

Change History (9)

comment:1 Changed 5 years ago by Kryštof Řeháček

Summary: manage.py executting tagged tests with syntax error does not show themmanage.py executing tagged tests with syntax error does not show them

comment:2 Changed 5 years ago by Tim Graham

Summary: manage.py executing tagged tests with syntax error does not show themRunning tagged tests hides any with syntax errors
Triage Stage: UnreviewedAccepted

The problem may be in django.test.runner.filter_tests_by_tags().

comment:3 Changed 4 years ago by Daniel Hones

Owner: changed from nobody to Daniel Hones
Status: newassigned

comment:4 Changed 4 years ago by Daniel Hones

Owner: Daniel Hones deleted
Status: assignednew

It looks like this behavior is ultimately due to Python's unittest TestLoader and not Django's test framework. In case of a SyntaxError when loading a test suite, it hits this line, which creates a _FailedTest object from it without raising an error. Then the django.test.runner.filter_tests_by_tags() sees a _FailedTest object which won't have any tags and so it won't be run at all. Since tests are tagged using a decorator, tests in a file with a syntax error won't have any tags recognized for them anyway because the file cannot be evaluated. I don't think there's a reasonable fix for this and I'm not sure it should even be considered a bug, although the behavior is a bit misleading.

comment:5 Changed 17 months ago by Chris Jerdonek

I think a possible solution for this would be to make it so that unittest.loader._FailedTest objects always match when tags are provided. That would prevent the issue of tests falsely passing due to e.g. syntax errors.

comment:6 Changed 17 months ago by Chris Jerdonek

Owner: set to Chris Jerdonek
Status: newassigned

comment:7 Changed 17 months ago by Chris Jerdonek

Has patch: set

comment:8 Changed 17 months ago by Mariusz Felisiak

Triage Stage: AcceptedReady for checkin

comment:9 Changed 17 months ago by Mariusz Felisiak <felisiak.mariusz@…>

Resolution: fixed
Status: assignedclosed

In 038940cf:

Fixed #29127 -- Prevented DiscoverRunner from hiding tagged test with syntax errors.

This mades _FailedTest objects always match tags in DiscoverRunner.

Note: See TracTickets for help on using tickets.
Back to Top