django selftests should skip tests bound to fail

[mir@…]

There are some cases where selftests must fail, e.g. for mysql with a transaction enabled storage engine, the serialization selftests don't work because mysql tests foreign key constraints prematurely.

Ideally, the test frame work should allow to skip tests properly, so that at the end of the tests you get a bottom line like:

200 tests passed; 14 skipped; 1 failure

See also ​this discussion

[Malcolm Tredinnick]

Hooking "skipped" tests into Python's unittest framework is a little tricky. I tried to do it in the past for an entirely different project and was only partially successful (and not in any way that I would include in Django). But it will be interesting to see what you come up with.

[Simon G. <dev@…>]

Accepted as per MT's comment above.

[Russell Keith-Magee]

For reference - the mysql tests aren't the only ones affected by this; strictly, the YAML parser tests should register as non-run/fail if you don't have pyYAML installed, and the markup tests should variously have known failures if docutils, markdown or texttile are not available. At the moment, YAML tests are skipped, and the markup tests become no-ops if their dependent libraries are not available. This leads to bugs like #5362 - my first impression with this test was to mark it 'worksforme', until I realized that the test was hiding the underlying problem.

[Michael Radziej <mir@…>]

I'm quoting Russell's idea from django-developers so that it doesn't get lost:

However, my
preferred solution would fix this at the output layer, rather than the
test layer - i.e., let the tests run and fail, but filter the output
against a list of known failures so that the failures are reported in
the final output as "X tests passed (with Y known and acceptable
failures)" rather than the current flood of stack traces.

[devin]

added decorators for skipping tests

[devin]

I've added a patch that defines decorators for conditionally skipping tests and a usage of it per Jason and Russell's discussion in IRC about #7611.

When tests are skipped, I'm raising an Exception SkippedTest. Now ideally these would be handled differently than other errors, but that'd require practically rewriting unittest as mtredinnick mentioned.

What we could do is filter this out in the output layer per Russell's idea. Check errors vs. SkippedTest and count those as a separate category in the output. Then we'd have to roll our own TestRunner instead of using unittest.TextTestRunner. Which would overlap a bit with #7884.

That's the direction I'm leaning, but I thought I'd bring the topic up now to see if that's the direction we need to be headed.

[devin]

I've implemented what I've mentioned before. Including decorators extending conditional_skip for the three cases mentioned in ​http://groups.google.com/group/django-developers/browse_thread/thread/f68628091d75f5c1

More could/should be added as needed.

I'm going to write up docs and tests for this shortly.

[devin]

skip test decorators. changes to auth tests. and decorator tests and documentation.

[devin]

I've finished up the patch. Though this somewhat disagrees with TestCase's urls attribute.

I'm not quite sure how we should handle that. It seems requiring the views is a better solution than adding the urls for the reasons brought up in #7611.

[Samuel Cormier-Iijima]

Attaching an updated patch for Django >= 1.0. I've also made the urls use reverse lookups instead of the hardcoded paths since those views may be included at different URLs in projects.

[Samuel Cormier-Iijima]

Updated patch for Django >= 1.0

[Marc Fargas]

From the diff:

871 New in Django development version

That should be ".. versionadded: 1.1", look ​here, that paragraph was changed just before 1.0 went out in [8843]

[Samuel Cormier-Iijima]

Updated for Django 1.0.1

[(none)]

Milestone post-1.0 deleted

[Kevin Kubasik]

Incorperated version into gsoc-testing branch.

[Russell Keith-Magee]

I'm going to close this in favor of #12991; unittest2 provides test skipping as a native feature, and the (soon to be in trunk) patch for that ticket includes introducing test skipping calls.