assertWarnsMessage() only checks the first warning for a matching message

[Mike Edmunds]

assertWarnsMessage(category, message) fails if the first warning issued in category does not contain message, even if some later warning does match.

As a result, test cases depend on the specific order warnings are issued. This can become confusing when there are several related deprecations going in at once, or when a feature being tested uses some other feature that also issues a warning.

For example, you might expect this test to pass, but it fails with AssertionError: 'Deprecated in MyApp' not found in 'Deprecated in Django':

class MyAppTests(SimpleTestCase):
    def test_deprecated_in_my_app(self):
        with self.assertWarnsMessage(DeprecationWarning, "Deprecated in MyApp"):
            # MyApp calls something in Django that happens to be deprecated:
            warnings.warn("Deprecated in Django", RemovedInNextVersionWarning)
            # Then MyApp issues the warning being tested:
            warnings.warn("Deprecated in MyApp", DeprecationWarning)

If you switch the order of the two warnings, the test passes. (And the second warning is then ignored—see #37072.)

There is a workaround by combining a very carefully crafted ignore_warnings() context with assertWarnsMessage(). (It requires knowing that the semi-documented ignore_warnings() also accepts a message argument. And that it interprets it quite differently from assertWarnsMessage().)

[Artyom Kotovskiy]

Would love to work on it in case it gets accepted.

[Jacob Walls]

Ouch, great find. This is supposed to be a non-regex version of the stdlib assertWarnsRegex, which isn't order-sensitive.

[Mike Edmunds]

I would suggest trying to fix #37072 at the same time—they're closely related. I outlined a possible approach in a comment there. There are 3-4 existing tests that seem to unintentionally depend on either this or #37072's behavior, so will need to be updated as part of the fix.

[Artyom Kotovskiy]

Pr -> ​https://github.com/django/django/pull/21228