Opened 3 years ago

Closed 3 years ago

Last modified 2 years ago

#32746 closed Bug (invalid)

Unexpected test failures in 3.3.2 when running the whole testsuite

Reported by: Jakub Kulík Owned by: nobody
Component: Testing framework Version: 3.2
Severity: Normal Keywords:
Cc: Triage Stage: Unreviewed
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no


We recently introduced Django 3.3.2 to Solaris (with Python 3.9) and we are hitting unexpected test failures when running the full test suite on SPARC. When I rerun those tests one by one however, they are successful. My guess is that parts of the test suite are not independent and affect each other.

Errors we are seeing are (although I am not sure that they are always the same and they changed since 3.3.1):

ERROR: test_output_normal (test_runner.test_debug_sql.TestDebugSQL)
ERROR: test_output_verbose (test_runner.test_debug_sql.TestDebugSQL)
ERROR: test_setupclass_exception (test_runner.test_debug_sql.TestDebugSQL)
ERROR: test_lazy_base_url_init (file_storage.tests.FileSystemStorageTests)
FAIL: test_incorrect_timezone (settings_tests.tests.SettingsTests)
FAILED (failures=1, errors=4, skipped=1146, expected failures=4)

Running the test suite with --reverse results in many more failures (attached).

This happens on SPARC only, Intel test results are clean (both with and without --reverse). Our other Django 2.2.21 with Python 3.7 is clean as well on both platforms.

I am not sure how to continue/investigate the issue further, but I am happy to do so if guided a little.

I have yet to try Django 3.1 or Python 3.7 (but runtime is highly unlike the problem).

Attachments (1)

reverse-failures.txt (71.0 KB ) - added by Jakub Kulík 3 years ago.
Test failures when running with --reverse

Download all attachments as: .zip

Change History (7)

by Jakub Kulík, 3 years ago

Attachment: reverse-failures.txt added

Test failures when running with --reverse

comment:1 by Mariusz Felisiak, 3 years ago

Resolution: needsinfo
Status: newclosed

I cannot reproduce these failures, all tests works for me when run in reverse, see also Can you provide a subset of these that causes failures for you? flags passed to, Python version, database, etc.

comment:2 by Jakub Kulík, 3 years ago

I see this issue only with Django 3.2 on SPARC (with both Python 3.7 & 3.9 and both with and without --reverse). Intel in this case is without issues, I also don't see this in Django 3.1 or 3.0 with all combinations of Python runtime (3.7 and 3.9), architecture (SPARC and Intel), and order (--reverse or not).

In each case, tests are executed with the following command:
/usr/bin/env LC_ALL=en_US.UTF-8 /usr/bin/python3.x ./ --settings test_sqlite --parallel=1 --verbosity=3 (--reverse)

comment:3 by Mariusz Felisiak, 3 years ago

Thanks for extra details, unfortunately I still cannot reproduce these failures so it must be something Solaris-specific.

comment:4 by Jakub Kulík, 3 years ago

Unfortunately, it seems so... Do you have any pointers/ideas on how I should debug this? I'll try to find and fix the issue.

in reply to:  4 comment:5 by Mariusz Felisiak, 3 years ago

Replying to Jakub Kulík:

Unfortunately, it seems so... Do you have any pointers/ideas on how I should debug this? I'll try to find and fix the issue.

It would help to find test classes (or even a set of tests) that are reproducing this issue without running the entire test suite. Maybe test_runner is enough.

comment:6 by Jakub Kulík, 2 years ago

Resolution: needsinfoinvalid

I am sorry, I just found the issue and it's entirely on our side.

For posterity: Django test suite can be fragile in a sense that when one test fails, many unrelated may follow, and that's what happened here. test_incorrect_timezone is the only one that is actually failing and the cause is our internal Solaris issue with /usr/share/zoneinfo. Once patched, the test suite is again 100% green.

Note: See TracTickets for help on using tickets.
Back to Top