Code

Opened 7 years ago

Closed 6 years ago

#3971 closed (wontfix)

Verbose testing framework does not say which test it is about to execute

Reported by: mtredinnick Owned by: nobody
Component: Testing framework Version: master
Severity: Keywords:
Cc: Triage Stage: Accepted
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: UI/UX:

Description

Some of the changes I'm making at the moment have the side-effect of breaking tests very early in their execution (when table flushing fails, for example). The problem is, even when running with -v2, I get no indication of which test it was trying to run. The verbose output only seems shows output that didn't cause the harness itself to fail.

I'm not sure how feasible it is to implement, but it would be nice for times to like this to have a -v3 option that said something like "About to attempt modeltests.model_forms.models", so that if I break the harness itself, I know where to look. I had to resort to running each directory in tests/modeltests/ one at a time until I saw the breakage (tests aren't run in any particularly predictable order, that I can see). Not the end of the world, but time consuming and a little tiring the third or fourth time it happened.

Attachments (1)

3971.diff (2.6 KB) - added by mtredinnick 7 years ago.
Updated patch to current svn

Download all attachments as: .zip

Change History (5)

comment:1 Changed 7 years ago by Simon G. <dev@…>

  • Needs documentation unset
  • Needs tests unset
  • Patch needs improvement unset
  • Triage Stage changed from Unreviewed to Accepted

comment:2 Changed 7 years ago by russellm

A vanilla unit test under -v2 *does* report the name of the test before the test is executed. The problem you are seeing is caused by the way we have overloaded TestCase.run() to call the flush and fixture load code before calling the baseclass run() implementation. As a result, the flush is called before the 'start test' debug message is printed.

I've attached a patch which I think will fix the problem. Could you try it out with one of your failing tests and let me know if it works?

Interesting side effect - this patch can be cleaned up to remove the need for django.test.TestCase. This approach would allow any TestCase instance with a 'fixtures' member variable to operate as a 'flush and load' test case. I'm not sure how I feel about this; do you have any opinions?

Changed 7 years ago by mtredinnick

Updated patch to current svn

comment:3 Changed 7 years ago by mtredinnick

Finally remembered to test this. It doesn't work as I would have thought, but it's possible my particular stress testing isn't worth solving.

I simulated my previous problems by putting "raise Exception" into core/management.py on line 732 (just before the call to load_data() in the flush() function). Then, running all the tests under -v2 exhibits the expected explosion, but we aren't told what failed.

Thinking about this a bit more, I'm not sure it's worth going to extreme lengths to account for this behaviour. Detecting that some bozo has stuck a monkey wrench deep in the guts of management.py might not be easy.

In any case, I've updated the patch to apply against recent trunk, since there were some changes in the interim and a bit of manual patching was required.

comment:4 Changed 6 years ago by russellm

  • Resolution set to wontfix
  • Status changed from new to closed

Closing this off; Malcolm has questioned the need for a fix, and we're no longer overloading TestCase.run() anyway.

Add Comment

Modify Ticket

Change Properties
<Author field>
Action
as closed
as The resolution will be set. Next status will be 'closed'
The resolution will be deleted. Next status will be 'new'
Author


E-mail address and user name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.