CookieStorage for contrib.messages crashes after upgrade to django 3.2

[Jan Pieter Waagmeester]

After upgrading to django 3.2, a previously stored cookie for contrib.messages crashes in

​https://github.com/django/django/blob/d6314c4c2ef647efe0d12450214fc5b4a4055290/django/contrib/messages/storage/cookie.py#L175

Django Version: 3.2
Python Version: 3.8.2

Traceback (most recent call last):
  File "/home/obs/virtualenv/lib/python3.8/site-packages/django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File "/home/obs/virtualenv/lib/python3.8/site-packages/django/utils/deprecation.py", line 119, in __call__
    response = self.process_response(request, response)
  File "/home/obs/virtualenv/lib/python3.8/site-packages/django/contrib/messages/middleware.py", line 23, in process_response
    unstored_messages = request._messages.update(response)
  File "/home/obs/virtualenv/lib/python3.8/site-packages/django/contrib/messages/storage/base.py", line 127, in update
    messages = self._loaded_messages + self._queued_messages
  File "/home/obs/virtualenv/lib/python3.8/site-packages/django/contrib/messages/storage/base.py", line 79, in _loaded_messages
    messages, all_retrieved = self._get()
  File "/home/obs/virtualenv/lib/python3.8/site-packages/django/contrib/messages/storage/fallback.py", line 25, in _get
    messages, all_retrieved = storage._get()
  File "/home/obs/virtualenv/lib/python3.8/site-packages/django/contrib/messages/storage/cookie.py", line 86, in _get
    messages = self._decode(data)
  File "/home/obs/virtualenv/lib/python3.8/site-packages/django/contrib/messages/storage/cookie.py", line 175, in _decode
    return self.signer.unsign_object(data, serializer=MessageSerializer)
  File "/home/obs/virtualenv/lib/python3.8/site-packages/django/core/signing.py", line 195, in unsign_object
    data = b64_decode(base64d)
  File "/home/obs/virtualenv/lib/python3.8/site-packages/django/core/signing.py", line 68, in b64_decode
    return base64.urlsafe_b64decode(s + pad)
  File "/usr/lib/python3.8/base64.py", line 133, in urlsafe_b64decode
    return b64decode(s)
  File "/usr/lib/python3.8/base64.py", line 87, in b64decode
    return binascii.a2b_base64(s)

Exception Type: Error at /user/login/
Exception Value: Invalid base64-encoded string: number of data characters (369) cannot be 1 more than a multiple of 4

(redacted) contents of the 'messages' cookie:

'[["__json_message",0,25,"Successfully signed in as '
 'admin@example.org."],["__json_message",0,25,"Successfully '
 'signed in as jieter."],["__json_message",0,25,"Ingelogd als '
 'admin@example.org."],["__json_message",0,25,"Ingelogd '
 'als '
 'admin@example.org."],["__json_message",0,20,"Bevestigingsmail '
 'verzonden naar test@example.nl."],["__json_message",0,25,"Ingelogd '
 'als '
 'test@example.nl."]]:1lTkj1:j_3PlpYSKiqPTMAB6_p2Q00eE8j6k7n0Sg_-_IpXG7Y')

[Mariusz Felisiak]

Jan thanks for this report, however I cannot reproduce this issue. Everything works when I will use your messages in ​messages_tests.test_cookie.CookieTests.test_legacy_encode_decode.

[Florian Apolloner]

We might need the original cookie as seen by the browser. Can you send it to security@… (for the lack of a better "trusted" address, we will handle the data with care).

[Florian Apolloner]

What confuses me here is that according to the traceback the signature was valid and it then failed in ​https://github.com/django/django/blob/b6475d7d7940f3ce575e0b0f2d83e517f899b4cf/django/core/signing.py#L195 -- but if the signature was valid in the first place the data should have been base64 encoded. The contents of the cookie seem to suggest otherwise.

[Jan Pieter Waagmeester]

I sent the contents of the cookie to security@

[Florian Apolloner]

Thanks, I think I understand what is going on. Out of pure luck we created test messages that properly decode from base64. They then fail json decoding and fall back to the old mechanism. I think we need to add binascii.Error to ​https://github.com/django/django/commit/2d6179c819010f6a9d00835d5893c4593c0b85a0#diff-6c5e944e6869d04c50af2ee02dd85c621177547eb7771d11066f452788d483f4R185

[Florian Apolloner]

Hi Jan, I have attached a diff that should fix your issue. Do you think you could apply that single change to your Django codebase and retry?

[Mariusz Felisiak]

Regression in 2d6179c819010f6a9d00835d5893c4593c0b85a0.

[Jan Pieter Waagmeester]

Replying to Florian Apolloner:

Hi Jan, I have attached a diff that should fix your issue. Do you think you could apply that single change to your Django codebase and retry?

Works like expected with the patch applied, great!

[Mariusz Felisiak]

​PR

[Mariusz Felisiak <felisiak.mariusz@…>]

In 4511d145:

Fixed #32643 -- Fixed decoding of messages in the pre-Django 3.2 format.

Thanks Jan Pieter Waagmeester for the report.

Regression in 2d6179c819010f6a9d00835d5893c4593c0b85a0.

[Mariusz Felisiak <felisiak.mariusz@…>]

In 539d005a:

[3.2.x] Fixed #32643 -- Fixed decoding of messages in the pre-Django 3.2 format.

Thanks Jan Pieter Waagmeester for the report.

Regression in 2d6179c819010f6a9d00835d5893c4593c0b85a0.

Backport of 4511d1459810037b91faa5b506e4f75c77aa72be from main.

[Adam Hooper]

Could Django please release a patch release that fixes this bug?

Unless I'm misunderstanding, I've run into this:

1. Upgrade to Django 3.2.
2. Test. Everything passes.
3. Deploy to production.
4. Users experience 500 errors.

In my case, I _happened_ to catch this when I deployed to staging. Pure luck.

But in general, this crash affect tons of existing sites and there is no workaround until 3.2.1 is released, right?

[Florian Apolloner]

and there is no workaround until 3.2.1 is released, right?

Setting a custom storage backend with the fix applied is one option. The other option is a custom Django package till 3.2.1 is released.