Unexpected test failures in 3.3.2 when running the whole testsuite

[Jakub Kulík]

We recently introduced Django 3.3.2 to Solaris (with Python 3.9) and we are hitting unexpected test failures when running the full test suite on SPARC. When I rerun those tests one by one however, they are successful. My guess is that parts of the test suite are not independent and affect each other.

Errors we are seeing are (although I am not sure that they are always the same and they changed since 3.3.1):

ERROR: test_output_normal (test_runner.test_debug_sql.TestDebugSQL)
ERROR: test_output_verbose (test_runner.test_debug_sql.TestDebugSQL)
ERROR: test_setupclass_exception (test_runner.test_debug_sql.TestDebugSQL)
ERROR: test_lazy_base_url_init (file_storage.tests.FileSystemStorageTests)
FAIL: test_incorrect_timezone (settings_tests.tests.SettingsTests)
FAILED (failures=1, errors=4, skipped=1146, expected failures=4)

Running the test suite with --reverse results in many more failures (attached).

This happens on SPARC only, Intel test results are clean (both with and without --reverse). Our other Django 2.2.21 with Python 3.7 is clean as well on both platforms.

I am not sure how to continue/investigate the issue further, but I am happy to do so if guided a little.

I have yet to try Django 3.1 or Python 3.7 (but runtime is highly unlike the problem).

[Jakub Kulík]

Test failures when running with --reverse

[Mariusz Felisiak]

I cannot reproduce these failures, all tests works for me when run in reverse, see also ​https://djangoci.com/job/main-reverse/. Can you provide a subset of these that causes failures for you? flags passed to runtests.py, Python version, database, etc.

[Jakub Kulík]

I see this issue only with Django 3.2 on SPARC (with both Python 3.7 & 3.9 and both with and without --reverse). Intel in this case is without issues, I also don't see this in Django 3.1 or 3.0 with all combinations of Python runtime (3.7 and 3.9), architecture (SPARC and Intel), and order (--reverse or not).

In each case, tests are executed with the following command:
/usr/bin/env LC_ALL=en_US.UTF-8 /usr/bin/python3.x ./runtests.py --settings test_sqlite --parallel=1 --verbosity=3 (--reverse)

[Mariusz Felisiak]

Thanks for extra details, unfortunately I still cannot reproduce these failures so it must be something Solaris-specific.

[Jakub Kulík]

Unfortunately, it seems so... Do you have any pointers/ideas on how I should debug this? I'll try to find and fix the issue.

[Mariusz Felisiak]

Replying to Jakub Kulík:

Unfortunately, it seems so... Do you have any pointers/ideas on how I should debug this? I'll try to find and fix the issue.

It would help to find test classes (or even a set of tests) that are reproducing this issue without running the entire test suite. Maybe test_runner is enough.

[Jakub Kulík]

I am sorry, I just found the issue and it's entirely on our side.

For posterity: Django test suite can be fragile in a sense that when one test fails, many unrelated may follow, and that's what happened here. test_incorrect_timezone is the only one that is actually failing and the cause is our internal Solaris issue with /usr/share/zoneinfo. Once patched, the test suite is again 100% green.