Opened 10 years ago

Closed 10 years ago

Last modified 10 years ago

#23947 closed Cleanup/optimization (fixed)

Django's test suite does not pass in reverse order

Reported by: Tim Graham Owned by: Diego Guimarães
Component: Core (Other) Version: dev
Severity: Normal Keywords:
Cc: diegaogs@… Triage Stage: Accepted
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

runtests.py now has a --reverse flag to run tests in reverse order (#23742). Unfortunately, Django's test suite doesn't pass this way which means some tests have side effects. We should see if we can fix this.

Attachments (2)

23947.diff (10.3 KB ) - added by Diego Guimarães 10 years ago.
initial patch
static_files.diff (575 bytes ) - added by Diego Guimarães 10 years ago.

Download all attachments as: .zip

Change History (23)

comment:1 by Diego Guimarães, 10 years ago

Cc: diegaogs@… added
Owner: changed from nobody to Diego Guimarães
Status: newassigned

by Diego Guimarães, 10 years ago

Attachment: 23947.diff added

initial patch

comment:2 by Diego Guimarães, 10 years ago

Has patch: set
Patch needs improvement: set

This initial patch fixes most of the errors when running the Django test suite with --reverse. The common problems I've found so far, was basically that some tests were performing operations that in a reverse way triggered side effects, like a smtp sever that was stopped first and had issues with Thread when tried to start it again, and cache that was closed by a previous test, etc.

There is still one test failing with issues handling staticfiles cache.

FAIL: test_template_tag_denorm (staticfiles_tests.tests.TestCollectionCachedStorage)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/diego/djangoproject/django/tests/staticfiles_tests/tests.py", line 526, in test_template_tag_denorm
      self.assertIn(b'url("img/relative.acae32e4532b.png', content)
      AssertionError: 'url("img/relative.acae32e4532b.png' not found in '@import url("../cached/styles.bb84a0240107.css");\nbody {\n    background: #d3d6d8 url("img/relative.deploy12345.png");\n}\n'

To reproduce this error: ./runtests.py --reverse staticfiles_tests.tests

comment:3 by Tim Graham, 10 years ago

I saw your paste in IRC that you had a proposed fix for the last issue. Any chance you could make a pull request with the fixes? That makes review a lot easier. Thanks!

comment:4 by Diego Guimarães, 10 years ago

Yes, sure. I'll create the pull request and update the ticket.

comment:5 by Diego Guimarães, 10 years ago

comment:6 by Tim Graham <timograham@…>, 10 years ago

In d05709821cf6ff77e00871d6fac3eb43356d44d1:

Refs #23947 -- Made a modeladmin test cleanup after itself.

comment:7 by Tim Graham <timograham@…>, 10 years ago

In e32a8a99d99c3d6c3015e220f6719f6ddc48b4f1:

Refs #23947 -- Isolated a mail test.

comment:8 by Tim Graham <timograham@…>, 10 years ago

In 106dde91e4dc30766db00d4ea21b4f954f49e83e:

Refs #23947 -- Fixed introspection test that had a side effect.

comment:9 by Tim Graham <timograham@…>, 10 years ago

In 40c60efecc0fc73f0b2320b44d684586b52ee799:

Refs #23947 -- Isolated some cache tests.

Thanks Diego Guimarãesi and Florian Apolloner.

comment:10 by Tim Graham <timograham@…>, 10 years ago

In 9f427617e4559012e1c2fd8fce46cbe225d8515d:

Refs #23947 -- Worked around a bug in Python that prevents deprecation warnings from appearing in tests.

by Diego Guimarães, 10 years ago

Attachment: static_files.diff added

comment:11 by Diego Guimarães, 10 years ago

This new patch tries to solve the staticfiles test problem. I added a tearDown to the TestHashedFiles mixin, where it clears the hashed_files after each test, avoiding side effects and unexpected changes among them. Is this approach valid?

comment:12 by Tim Graham <timograham@…>, 10 years ago

In fd60e6c8878986a102f0125d9cdf61c717605cf1:

Refs #23947 -- Prevented staticfiles tests from having side effects.

comment:13 by Tim Graham, 10 years ago

Has patch: unset
Patch needs improvement: unset

SQLite is passing in reverse, but it looks like there is more work to do on PostgreSQL. I see the following failures:

admin_views.tests.AdminViewDeletedObjectsTest.test_generic_relations
generic_relations_regress.tests.GenericRelationTests.test_annotate
migrations.test_operations.OperationTests.test_alter_field_m2m
queries.tests.SubqueryTests.test_related_sliced_subquery

comment:14 by Diego Guimarães, 10 years ago

Ok, I'll setup the test suite to run with PostgresSQL and debug those tests.

comment:15 by Diego Guimarães, 10 years ago

Created a new PR with the fixes to PostgreSQL https://github.com/django/django/pull/3726

comment:16 by Diego Guimarães, 10 years ago

Has patch: set

comment:17 by Tim Graham <timograham@…>, 10 years ago

In d8182f294aa0a8dfb3ea4611a9adb7a748a36bf4:

Refs #23947 -- Fixed admin_views test execution order dependency.

comment:18 by Tim Graham <timograham@…>, 10 years ago

In 222699d224ee7851e1cab5ecc029cf83836b882b:

Refs #23947 -- Fixed queries test execution order dependency.

Specify an id to avoid conflict with objects created in setUpTestData.

comment:19 by Tim Graham <timograham@…>, 10 years ago

In c17d821fa74b01205f3a2fe42a96dfe804de3df3:

Refs #23947 -- Improved migrations tests table cleanup.

Copied technique from schema tests.

comment:20 by Tim Graham, 10 years ago

Resolution: fixed
Status: assignedclosed

I think we're finished here for now. Thanks Diego!

comment:21 by Tim Graham <timograham@…>, 10 years ago

In 5c43fd48258eb977e6e611310e12a921d8470944:

Isolated some cache tests; refs #23947.

This reverts a change made in 40c60efecc0fc73f0b2320b44d684586b52ee799
which was incorrect and caused CacheKeyWarnings.

Note: See TracTickets for help on using tickets.
Back to Top