Opened 9 years ago

Closed 6 years ago

#4788 closed (duplicate)

django selftests should skip tests bound to fail

Reported by: mir@… Owned by: Kevin Kubasik
Component: Testing framework Version: master
Severity: Keywords:
Cc: devin@…, sciyoshi@…, Jay Hargis Triage Stage: Accepted
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: UI/UX:

Description

There are some cases where selftests must fail, e.g. for mysql with a transaction enabled storage engine, the serialization selftests don't work because mysql tests foreign key constraints prematurely.

Ideally, the test frame work should allow to skip tests properly, so that at the end of the tests you get a bottom line like:

200 tests passed; 14 skipped; 1 failure

See also this discussion

Attachments (5)

skipped_test_deco_r8069.diff (2.4 KB) - added by devin 8 years ago.
added decorators for skipping tests
skipped_test_deco_r8075.diff (7.6 KB) - added by devin 8 years ago.
skipped_test_deco_r8127.diff (11.0 KB) - added by devin 8 years ago.
skip test decorators. changes to auth tests. and decorator tests and documentation.
testing-patches.diff (14.2 KB) - added by Samuel Cormier-Iijima 8 years ago.
Updated patch for Django >= 1.0
testing-patches-1.0.1.diff (14.2 KB) - added by Samuel Cormier-Iijima 8 years ago.
Updated for Django 1.0.1

Download all attachments as: .zip

Change History (23)

comment:1 Changed 9 years ago by Malcolm Tredinnick

Needs documentation: unset
Needs tests: unset
Patch needs improvement: unset

Hooking "skipped" tests into Python's unittest framework is a little tricky. I tried to do it in the past for an entirely different project and was only partially successful (and not in any way that I would include in Django). But it will be interesting to see what you come up with.

comment:2 Changed 9 years ago by Simon G. <dev@…>

Triage Stage: UnreviewedAccepted

Accepted as per MT's comment above.

comment:3 Changed 9 years ago by Russell Keith-Magee

Owner: changed from nobody to Russell Keith-Magee
Status: newassigned

For reference - the mysql tests aren't the only ones affected by this; strictly, the YAML parser tests should register as non-run/fail if you don't have pyYAML installed, and the markup tests should variously have known failures if docutils, markdown or texttile are not available. At the moment, YAML tests are skipped, and the markup tests become no-ops if their dependent libraries are not available. This leads to bugs like #5362 - my first impression with this test was to mark it 'worksforme', until I realized that the test was hiding the underlying problem.

comment:4 Changed 9 years ago by Russell Keith-Magee

Owner: Russell Keith-Magee deleted
Status: assignednew

comment:5 Changed 9 years ago by Michael Radziej <mir@…>

I'm quoting Russell's idea from django-developers so that it doesn't get lost:

However, my
preferred solution would fix this at the output layer, rather than the
test layer - i.e., let the tests run and fail, but filter the output
against a list of known failures so that the failures are reported in
the final output as "X tests passed (with Y known and acceptable
failures)" rather than the current flood of stack traces.

comment:6 Changed 9 years ago by Adrian Holovaty

Owner: set to nobody

comment:7 Changed 8 years ago by devin

Owner: changed from nobody to devin

comment:8 Changed 8 years ago by devin

Cc: devin@… added

Changed 8 years ago by devin

added decorators for skipping tests

comment:9 Changed 8 years ago by devin

Has patch: set
milestone: post-1.0
Needs documentation: set
Needs tests: set
Patch needs improvement: set

I've added a patch that defines decorators for conditionally skipping tests and a usage of it per Jason and Russell's discussion in IRC about #7611.

When tests are skipped, I'm raising an Exception SkippedTest. Now ideally these would be handled differently than other errors, but that'd require practically rewriting unittest as mtredinnick mentioned.

What we could do is filter this out in the output layer per Russell's idea. Check errors vs. SkippedTest and count those as a separate category in the output. Then we'd have to roll our own TestRunner instead of using unittest.TextTestRunner. Which would overlap a bit with #7884.

That's the direction I'm leaning, but I thought I'd bring the topic up now to see if that's the direction we need to be headed.

comment:10 Changed 8 years ago by devin

Patch needs improvement: unset

I've implemented what I've mentioned before. Including decorators extending conditional_skip for the three cases mentioned in http://groups.google.com/group/django-developers/browse_thread/thread/f68628091d75f5c1

More could/should be added as needed.

I'm going to write up docs and tests for this shortly.

Changed 8 years ago by devin

Changed 8 years ago by devin

skip test decorators. changes to auth tests. and decorator tests and documentation.

comment:11 Changed 8 years ago by devin

Needs documentation: unset
Needs tests: unset

I've finished up the patch. Though this somewhat disagrees with TestCase's urls attribute.

I'm not quite sure how we should handle that. It seems requiring the views is a better solution than adding the urls for the reasons brought up in #7611.

comment:12 Changed 8 years ago by Samuel Cormier-Iijima

Cc: sciyoshi@… added

Attaching an updated patch for Django >= 1.0. I've also made the urls use reverse lookups instead of the hardcoded paths since those views may be included at different URLs in projects.

Changed 8 years ago by Samuel Cormier-Iijima

Attachment: testing-patches.diff added

Updated patch for Django >= 1.0

comment:13 Changed 8 years ago by Marc Fargas

From the diff:

871 New in Django development version

That should be ".. versionadded: 1.1", look here, that paragraph was changed just before 1.0 went out in [8843]

Changed 8 years ago by Samuel Cormier-Iijima

Attachment: testing-patches-1.0.1.diff added

Updated for Django 1.0.1

comment:14 Changed 8 years ago by Jay Hargis

Cc: Jay Hargis added

comment:15 Changed 8 years ago by (none)

milestone: post-1.0

Milestone post-1.0 deleted

comment:16 Changed 7 years ago by Kevin Kubasik

Owner: changed from devin to Kevin Kubasik

comment:17 Changed 7 years ago by Kevin Kubasik

Incorperated version into gsoc-testing branch.

comment:18 Changed 6 years ago by Russell Keith-Magee

Resolution: duplicate
Status: newclosed

I'm going to close this in favor of #12991; unittest2 provides test skipping as a native feature, and the (soon to be in trunk) patch for that ticket includes introducing test skipping calls.

Note: See TracTickets for help on using tickets.
Back to Top