Changes between Version 1 and Version 2 of Ticket #37223
- Timestamp:
- Jul 21, 2026, 9:21:01 AM (3 hours ago)
Legend:
- Unmodified
- Added
- Removed
- Modified
-
Ticket #37223 – Description
v1 v2 3 3 `django.core.signing.JSONSerializer` encodes its JSON payload with latin-1 and decodes it with latin-1. With the default `ensure_ascii=True` serializer the output is pure ASCII, so latin-1 and UTF-8 are indistinguishable and the bug is invisible. But any serializer that emits raw UTF-8 (orjson, msgspec, `json.dumps(..., ensure_ascii=False)`) produces non-ASCII bytes for non-ASCII data, and `JSONSerializer.loads` silently mojibakes them when decoding as latin-1. No `BadSignature` and no `UnicodeDecodeError` is raised — the data is returned corrupted as if it were valid. 4 4 5 Both `SESSION_SERIALIZER` and `signing.dumps(serializer=...)` are documented extension points , so a user adopting a UTF-8-emitting serializer hits this. It was documented from the third-party-package side by Adam Johnson in django-orjson PR #15 ([https://github.com/adamchainz/django-orjson/pull/15]).5 Both `SESSION_SERIALIZER` and `signing.dumps(serializer=...)` are documented extension points. The scenario that bites is the rollback: a project runs a UTF-8-emitting serializer (e.g. `OrjsonSerializer` for `SESSION_SERIALIZER`), then switches back to the default — every token signed while the UTF-8 serializer was active comes back mojibaked, with no `BadSignature` and no `UnicodeDecodeError`. A staggered deploy across mixed patched/unpatched processes is the same class of risk, just less common. The django-orjson maintainer documented this as a one-way migration limitation in [https://github.com/adamchainz/django-orjson/pull/15 django-orjson PR #15] (warning boxes in `docs/sessions.rst` and `docs/signing.rst`, plus `xfail` tests for the cross-serializer case), rather than fixing it — the fix has to live in Django. This is a maintainer-documented limitation, not a user-reported data loss incident. 6 6 7 7 == How to reproduce == … … 45 45 * `django/contrib/sessions/backends/base.py` and `django/contrib/sessions/backends/signed_cookies.py` — use `signing.dumps` / `signing.loads`. 46 46 47 `django/contrib/messages/storage/cookie.py` is '''not''' covered by this fix: it has its own `MessagePartGatherSerializer.dumps` and `MessageSerializer.loads` that call `json.dumps(...).encode("latin-1")` and `json.loads(data.decode("latin-1"))` directly (lines 68 and 73), independent of `signing.JSONSerializer`. That is the same bug pattern on a separate code path and should be filed as a follow-up ticket. `django/contrib/admin` does not use `signing` directly.47 `django/contrib/messages/storage/cookie.py` is '''not''' covered by this fix: it has its own `MessagePartGatherSerializer.dumps` and `MessageSerializer.loads` that call `json.dumps(...).encode("latin-1")` and `json.loads(data.decode("latin-1"))` directly (lines 68 and 73), independent of `signing.JSONSerializer`. That is the same bug pattern on a separate code path. The two are intentionally kept as separate tickets and separate PRs: they touch different modules, share no code path, and can land and be reverted independently. Keeping them separate also keeps each diff minimal and the review focused on one contract at a time. `django/contrib/admin` does not use `signing` directly. 48 48 49 49 == Proposed direction == … … 61 61 Tests should be added to the existing signing test module (tests/signing/tests.py) and should cover: 62 62 63 * a regression test that feeds raw UTF-8 bytes (as a UTF-8-emitting external serializer would produce ) to loads and asserts the original value is returned, not the latin-1 mojibake — this is the test that fails before the fix and passes after;63 * a regression test that feeds raw UTF-8 bytes (as a UTF-8-emitting external serializer would produce — not bytes `JSONSerializer.dumps` could ever produce, since `ensure_ascii=True` escapes everything to ASCII) to loads and asserts the original value is returned, not the latin-1 mojibake — this is the test that fails before the fix and passes after; 64 64 * a round-trip of a non-ASCII payload (including combining marks and non-BMP characters) through dumps/loads. 65 65 … … 70 70 * [https://github.com/django/django/pull/21651 PR #21651] — Pull request implementing this fix. 71 71 * [https://github.com/django/new-features/issues/187 new-features #187] — Pluggable JSON serialization/deserialization backend (this fix removes the signing carve-out from step 1). 72 * [https://github.com/adamchainz/django-orjson/pull/15 django-orjson PR #15] — Documented the mojibake from the third-party-package side. 72 * [https://github.com/adamchainz/django-orjson/pull/15 django-orjson PR #15] — The django-orjson maintainer documented this as a one-way migration limitation (warning boxes + `xfail` tests for the cross-serializer case), not a user-reported data loss incident. Authoritative source for the mojibake behavior described here. 73 * Django's `contrib/sessions/backends/signed_cookies.py` — uses `signing.dumps(serializer=self.serializer)` where `self.serializer` is `SESSION_SERIALIZER`; sessions routinely contain non-ASCII data (user names, locale), so the mixing scenario is reachable from Django core, not only from third-party packages. 73 74 * Commit [58a086acfb] — introduced the latin-1 encode/decode step (Oct 2012).