Code

Opened 7 years ago

Closed 4 years ago

#4788 closed (duplicate)

django selftests should skip tests bound to fail

Reported by: mir@… Owned by: kkubasik
Component: Testing framework Version: master
Severity: Keywords:
Cc: devin@…, sciyoshi@…, jb0t Triage Stage: Accepted
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: UI/UX:

Description

There are some cases where selftests must fail, e.g. for mysql with a transaction enabled storage engine, the serialization selftests don't work because mysql tests foreign key constraints prematurely.

Ideally, the test frame work should allow to skip tests properly, so that at the end of the tests you get a bottom line like:

200 tests passed; 14 skipped; 1 failure

See also this discussion

Attachments (5)

skipped_test_deco_r8069.diff (2.4 KB) - added by devin 6 years ago.
added decorators for skipping tests
skipped_test_deco_r8075.diff (7.6 KB) - added by devin 6 years ago.
skipped_test_deco_r8127.diff (11.0 KB) - added by devin 6 years ago.
skip test decorators. changes to auth tests. and decorator tests and documentation.
testing-patches.diff (14.2 KB) - added by sciyoshi 6 years ago.
Updated patch for Django >= 1.0
testing-patches-1.0.1.diff (14.2 KB) - added by sciyoshi 6 years ago.
Updated for Django 1.0.1

Download all attachments as: .zip

Change History (23)

comment:1 Changed 7 years ago by mtredinnick

  • Needs documentation unset
  • Needs tests unset
  • Patch needs improvement unset

Hooking "skipped" tests into Python's unittest framework is a little tricky. I tried to do it in the past for an entirely different project and was only partially successful (and not in any way that I would include in Django). But it will be interesting to see what you come up with.

comment:2 Changed 7 years ago by Simon G. <dev@…>

  • Triage Stage changed from Unreviewed to Accepted

Accepted as per MT's comment above.

comment:3 Changed 7 years ago by russellm

  • Owner changed from nobody to russellm
  • Status changed from new to assigned

For reference - the mysql tests aren't the only ones affected by this; strictly, the YAML parser tests should register as non-run/fail if you don't have pyYAML installed, and the markup tests should variously have known failures if docutils, markdown or texttile are not available. At the moment, YAML tests are skipped, and the markup tests become no-ops if their dependent libraries are not available. This leads to bugs like #5362 - my first impression with this test was to mark it 'worksforme', until I realized that the test was hiding the underlying problem.

comment:4 Changed 7 years ago by russellm

  • Owner russellm deleted
  • Status changed from assigned to new

comment:5 Changed 7 years ago by Michael Radziej <mir@…>

I'm quoting Russell's idea from django-developers so that it doesn't get lost:

However, my
preferred solution would fix this at the output layer, rather than the
test layer - i.e., let the tests run and fail, but filter the output
against a list of known failures so that the failures are reported in
the final output as "X tests passed (with Y known and acceptable
failures)" rather than the current flood of stack traces.

comment:6 Changed 7 years ago by adrian

  • Owner set to nobody

comment:7 Changed 6 years ago by devin

  • Owner changed from nobody to devin

comment:8 Changed 6 years ago by devin

  • Cc devin@… added

Changed 6 years ago by devin

added decorators for skipping tests

comment:9 Changed 6 years ago by devin

  • Has patch set
  • milestone set to post-1.0
  • Needs documentation set
  • Needs tests set
  • Patch needs improvement set

I've added a patch that defines decorators for conditionally skipping tests and a usage of it per Jason and Russell's discussion in IRC about #7611.

When tests are skipped, I'm raising an Exception SkippedTest. Now ideally these would be handled differently than other errors, but that'd require practically rewriting unittest as mtredinnick mentioned.

What we could do is filter this out in the output layer per Russell's idea. Check errors vs. SkippedTest and count those as a separate category in the output. Then we'd have to roll our own TestRunner instead of using unittest.TextTestRunner. Which would overlap a bit with #7884.

That's the direction I'm leaning, but I thought I'd bring the topic up now to see if that's the direction we need to be headed.

comment:10 Changed 6 years ago by devin

  • Patch needs improvement unset

I've implemented what I've mentioned before. Including decorators extending conditional_skip for the three cases mentioned in http://groups.google.com/group/django-developers/browse_thread/thread/f68628091d75f5c1

More could/should be added as needed.

I'm going to write up docs and tests for this shortly.

Changed 6 years ago by devin

Changed 6 years ago by devin

skip test decorators. changes to auth tests. and decorator tests and documentation.

comment:11 Changed 6 years ago by devin

  • Needs documentation unset
  • Needs tests unset

I've finished up the patch. Though this somewhat disagrees with TestCase's urls attribute.

I'm not quite sure how we should handle that. It seems requiring the views is a better solution than adding the urls for the reasons brought up in #7611.

comment:12 Changed 6 years ago by sciyoshi

  • Cc sciyoshi@… added

Attaching an updated patch for Django >= 1.0. I've also made the urls use reverse lookups instead of the hardcoded paths since those views may be included at different URLs in projects.

Changed 6 years ago by sciyoshi

Updated patch for Django >= 1.0

comment:13 Changed 6 years ago by telenieko

From the diff:

871 New in Django development version

That should be ".. versionadded: 1.1", look here, that paragraph was changed just before 1.0 went out in [8843]

Changed 6 years ago by sciyoshi

Updated for Django 1.0.1

comment:14 Changed 6 years ago by jb0t

  • Cc jb0t added

comment:15 Changed 5 years ago by anonymous

  • milestone post-1.0 deleted

Milestone post-1.0 deleted

comment:16 Changed 5 years ago by kkubasik

  • Owner changed from devin to kkubasik

comment:17 Changed 5 years ago by kkubasik

Incorperated version into gsoc-testing branch.

comment:18 Changed 4 years ago by russellm

  • Resolution set to duplicate
  • Status changed from new to closed

I'm going to close this in favor of #12991; unittest2 provides test skipping as a native feature, and the (soon to be in trunk) patch for that ticket includes introducing test skipping calls.

Add Comment

Modify Ticket

Change Properties
<Author field>
Action
as closed
as The resolution will be set. Next status will be 'closed'
The resolution will be deleted. Next status will be 'new'
Author


E-mail address and user name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.