assertWarnsMessage() also ignores entire warning category

[Mike Edmunds]

assertWarnsMessage(category, message) implicitly does the equivalent of ignore_warnings(category). As a result, tests for particular deprecation warnings can overlook other, unrelated deprecation warnings coming from the code being tested.

You would expect the second test case below to cause an error for RemovedInDjango70Warning: Unrelated deprecation, but it actually passes:

class SomeTests(SimpleTestCase):
    def test_this_errors_as_expected(self):
        # ERROR: RemovedInDjango70Warning: Unrelated deprecation
        warnings.warn("Unrelated deprecation", RemovedInDjango70Warning)

    def test_so_this_should_also_error_but_does_not(self):
        with self.assertWarnsMessage(RemovedInDjango70Warning, "Expected deprecation"):
            warnings.warn("Expected deprecation", RemovedInDjango70Warning)
            # This gets ignored.
            warnings.warn("Unrelated deprecation", RemovedInDjango70Warning)

    def test_duplicate_warnings_should_maybe_also_error(self):
        with self.assertWarnsMessage(RemovedInDjango70Warning, "Expected deprecation"):
            warnings.warn("Expected deprecation", RemovedInDjango70Warning)
            # This also gets ignored. Maybe it shouldn't?
            warnings.warn("Expected deprecation", RemovedInDjango70Warning)

The problem is assertWarnsMessage() and assertRaisesMessage() try to share SimpleTestCase._assertFooMessage(), which wraps assertWarns() and assertRaises() respectively. But those methods aren't really parallel: there can be only one error raised, but execution can continue after a warning. assertWarns(category) captures all warnings in the category.

[Mike Edmunds]

Here's a test case that could be added to tests/test_utils/tests.py to verify a fix:

class AssertWarnsMessageTests(SimpleTestCase):
    ...

    def test_does_not_ignore_entire_category(self):
        with (
            self.assertRaisesMessage(RemovedInNextVersionWarning, "Unexpected"),
            self.assertWarnsMessage(RemovedInNextVersionWarning, "Expected"),
        ):
            warnings.warn("Expected", RemovedInNextVersionWarning)
            warnings.warn("Unexpected", RemovedInNextVersionWarning)