Opened 7 years ago

Closed 4 years ago

#27391 closed New feature (fixed)

Support unittest.TestCase.debug() method.

Reported by: Pavel Savchenko Owned by: nobody
Component: Testing framework Version: dev
Severity: Normal Keywords: unittest SimpleTestCase debug
Cc: me@… Triage Stage: Ready for checkin
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

Introduction

The documentation for unittest.TestCase.debug states that invoking it would:

Run the test without collecting the result. This allows exceptions raised by the test to be propagated to the caller, and can be used to support running tests under a debugger.

In essence, debug() invokes the test method same as TestCase.run() does, without the added functionality of catching exceptions and related cruft.

Side effects

A recent change in pytest (see references below) brought to our attention the fact that Django does not implement a debug method, and performs important stuff in TestCase.__call__ (besides calling run()), such as calling _pre_startup and _post_teardown methods.

Thus, since debug() is unimplemented, running debug on a django TestCase (for example for debugging purposes), results unintended behavior, such as certain methods not being called. Most importantly _fixture_callback and _fixture_teardown which also perform database rollback (or call flush).

References:
https://github.com/pytest-dev/pytest/pull/1890
https://github.com/pytest-dev/pytest/issues/1932
https://github.com/pytest-dev/pytest-django/pull/406
https://github.com/python/cpython/blob/master/Lib/unittest/case.py#L651-L658

Change History (17)

comment:1 by Tim Graham, 7 years ago

Component: UncategorizedTesting framework
Type: UncategorizedNew feature

Would this have benefit outside of the usage of third-party test runners such as pytest? If so, what does it look like?

comment:2 by Pavel Savchenko, 7 years ago

Well, not that I know of yet, but basically, any test runner that allows unittests may assume "debug" can be used for exception debugging; and right now this cannot be done without issues when using Django's SimpleTestCase derivatives.

Regardless, the patch should be fairly simple 10 line method and I can submit that. It's invoking the test that I'm having trouble with, how to invoke "debug" as part of the regular test suite, maybe we can use call_command...

Last edited 7 years ago by Pavel Savchenko (previous) (diff)

comment:3 by Tim Graham, 7 years ago

Triage Stage: UnreviewedAccepted

comment:4 by Pavel Savchenko, 7 years ago

As for other test runners, here's now nose2 invokes a debugger (IIRC pytest does something similar):
https://github.com/nose-devs/nose2/blob/master/nose2/plugins/debugger.py#L41-L60

This means that the post_mortem is invoked only after the exception was already caught in SimpleTestCase.__call__ and, for example, the database was already rolled back. The test frameworks simple have no real way to examine the state of the application at the time of the error (but only "after", when it may be too late).

comment:5 by Pavel Savchenko, 7 years ago

Has patch: set

Added test case (demonstrating usage expectations) and resolution in PR 7436

comment:6 by Adam Johnson, 7 years ago

Cc: me@… added

comment:7 by Tim Graham, 7 years ago

Patch needs improvement: set

I think the test could be a bit cleaner. I sent some edits to the PR, though I broke some things too.

comment:8 by Tim Graham, 7 years ago

Patch needs improvement: unset

comment:9 by Tim Martin, 6 years ago

Patch needs improvement: set

comment:10 by Asif Saifuddin Auvi, 6 years ago

Needs documentation: set
Version: 1.10master

in reply to:  10 comment:11 by Pavel Savchenko, 6 years ago

Patch needs improvement: unset

Replying to Asif Saifuddin Auvi:

Patch was improved as per Tim Martin's notes in commit 7ec5fae. I added a sketch for release note mention in this comment.

Is there other documentation you had in mind?

comment:12 by Tim Graham, 6 years ago

Needs documentation: unset

comment:13 by Tim Graham, 6 years ago

Patch needs improvement: set

comment:14 by Mariusz Felisiak, 4 years ago

Patch needs improvement: unset

comment:15 by Mariusz Felisiak <felisiak.mariusz@…>, 4 years ago

In dc8cd2fe:

Refs #27391 -- Added more tests for SimpleTestCase.

comment:16 by Mariusz Felisiak, 4 years ago

Summary: Support unittest.TestCase.debug() methodSupport unittest.TestCase.debug() method.
Triage Stage: AcceptedReady for checkin

comment:17 by Mariusz Felisiak <felisiak.mariusz@…>, 4 years ago

Resolution: fixed
Status: newclosed

In 1711c509:

Fixed #27391 -- Implemented SimpleTestCase.debug().

debug() should bubbled up exceptions if occurring in test, but behave
the same as run() when no exceptions occurred.

Note: See TracTickets for help on using tickets.
Back to Top