id	summary	reporter	owner	description	type	status	component	version	severity	resolution	keywords	cc	stage	has_patch	needs_docs	needs_tests	needs_better_patch	easy	ui_ux
33264	DiscoverRunner doesn't counts unexpectedSuccess as an error	Baptiste Mispelon	Baptiste Mispelon	"I noticed today that my CI test suite was green even though its console output printed:
{{{
FAILED (expected failures=13, unexpected successes=1)
}}}

It turns out that Django's default `DiscoverRunner` does not count ""unexpected successes"" as errors when deciding the return code of its `test` management command. [1]

So even though it prints ""FAILED"", the command returns with an exit code of 0 which my CI (correctly) interprets as a success.

The fix seems quite simple:
{{{#!diff
diff --git a/django/test/runner.py b/django/test/runner.py
index 09ac4e142a..2e36514922 100644
--- a/django/test/runner.py
+++ b/django/test/runner.py
@@ -875,7 +875,7 @@ class DiscoverRunner:
         teardown_test_environment()
 
     def suite_result(self, suite, result, **kwargs):
-        return len(result.failures) + len(result.errors)
+        return len(result.failures) + len(result.errors) + len(result.unexpectedSuccesses)
 
     def _get_databases(self, suite):
         databases = {}
}}}

But I wonder if that would be considered backwards incompatible.
I also think this is a pretty serious issue and I wonder if backports would be considered for this.

[1] https://github.com/django/django/blob/60503cc747eeda7c61bab02b71f8f55a733a6eea/django/test/runner.py#L877-L878"	Bug	closed	Testing framework	3.2	Normal	fixed			Ready for checkin	1	0	0	0	0	0