#32178 closed New feature (fixed)
Allow database backends to skip tests and mark expected failures
Reported by: | Tim Graham | Owned by: | Hasan Ramezani |
---|---|---|---|
Component: | Database layer (models, ORM) | Version: | dev |
Severity: | Normal | Keywords: | |
Cc: | Triage Stage: | Accepted | |
Has patch: | yes | Needs documentation: | no |
Needs tests: | no | Patch needs improvement: | no |
Easy pickings: | no | UI/UX: | no |
Description
Currently Django's built-in database backends have special privileges in the test suite when it comes to test skipping such as:
tests/model_fields/test_integerfield.py: @unittest.skipIf(connection.vendor == 'sqlite', "SQLite doesn't have a constraint.")
While DatabaseFeatures
attributes are generally preferred to connection.vendor
checking, I think allowing backends to skip certain tests has its place. Adding features flags that are likely to apply to only a single database is a bit cumbersome, especially for third-party backends.
Django should provide more generic way to skip tests and mark expected failures. For example, see django-cockroachdb's logic which was inspired by django-mssql-backend. In a more refined version, expected_failures
and skip_classes
would be attributes of DatabaseFeatures
.
That logic uses a RUNNING_COCKROACH_BACKEND_TESTS
environment variable to decide whether or not the Django test suite is running. There might be a better way to determine that. At least the variable name would be more generic.
Change History (19)
comment:1 by , 4 years ago
Triage Stage: | Unreviewed → Accepted |
---|
comment:2 by , 4 years ago
comment:4 by , 4 years ago
Hi Tim,
I created a PR based on django-cockroachdb
example. Please check the PR if you have time and if you confirm it, I will continue with moving skip tests to database creation files.
I added expected_failures_tests
and skip_tests
(which can be used for skipping both test class and function) and added a mark_tests
method similar to the one that you did for django-cockroachdb
.
comment:5 by , 4 years ago
Has patch: | set |
---|---|
Owner: | changed from | to
Status: | new → assigned |
comment:6 by , 4 years ago
Patch needs improvement: | set |
---|
I'll mark this as RFC when I'm done iterating with Hasan. (Meanwhile, let's leave "needs improvement" checked so it doesn't appear in the review queue for others.)
comment:7 by , 4 years ago
Patch needs improvement: | unset |
---|
comment:8 by , 4 years ago
Needs documentation: | set |
---|---|
Patch needs improvement: | set |
comment:9 by , 4 years ago
Needs documentation: | unset |
---|---|
Patch needs improvement: | unset |
comment:10 by , 4 years ago
Triage Stage: | Accepted → Ready for checkin |
---|
comment:14 by , 4 years ago
I have a comment / question about this feature. I was analyzing a test by inspecting the code, and I couldn't figure out why it was passing on SQLite because it used to have "skip" code which is no longer there. And then I came across this ticket via the history.
Would it be possible to have at least a code convention of including a code comment at the test site when the test is included in django_test_skips
? Otherwise, the average reader wouldn't know to check there for the test's name. (I would call this behavior "magical" because it's not explicit at the code site.) Also, even for people that know about it, it's not something that's easy to check while e.g. browsing the code on the web.
comment:15 by , 4 years ago
I sympathize but would you give third-party backends the same privilege? If so, it may be better to scrap this feature and allow third-parties to contribute their skips directly in the test suite. If not, this feature could at least be documented in the contributing guide.
comment:17 by , 4 years ago
Triage Stage: | Ready for checkin → Accepted |
---|
What can we do to skip a single test? I checked the
django-cockroachdb
and there you have askip_tests
which can be used to skip a single test(If I got it right).