Signing's JSONSerializer uses latin-1, causing silent mojibake with UTF-8-emitting serializers

[Paolo Melchiorre]

Summary

django.core.signing.JSONSerializer encodes its JSON payload with latin-1 and decodes it with latin-1. With the default ensure_ascii=True serializer the output is pure ASCII, so latin-1 and UTF-8 are indistinguishable and the bug is invisible. But any serializer that emits raw UTF-8 (orjson, msgspec, json.dumps(..., ensure_ascii=False)) produces non-ASCII bytes for non-ASCII data, and JSONSerializer.loads silently mojibakes them when decoding as latin-1. No BadSignature and no UnicodeDecodeError is raised — the data is returned corrupted as if it were valid.

Both SESSION_SERIALIZER and signing.dumps(serializer=...) are documented extension points. The scenario that bites is the rollback: a project runs a UTF-8-emitting serializer (e.g. OrjsonSerializer for SESSION_SERIALIZER), then switches back to the default — every token signed while the UTF-8 serializer was active comes back mojibaked, with no BadSignature and no UnicodeDecodeError. A staggered deploy across mixed patched/unpatched processes is the same class of risk, just less common. The django-orjson maintainer documented this as a one-way migration limitation in ​django-orjson PR #15 (warning boxes in docs/sessions.rst and docs/signing.rst, plus xfail tests for the cross-serializer case), rather than fixing it — the fix has to live in Django. This is a maintainer-documented limitation, not a user-reported data loss incident.

How to reproduce

JSONSerializer currently does:

# django/core/signing.py
class JSONSerializer:
    def dumps(self, obj):
        return json.dumps(obj, separators=(",", ":")).encode("latin-1")

    def loads(self, data):
        return json.loads(data.decode("latin-1"))

A payload containing héllo, written by a UTF-8-emitting serializer as the bytes h\xc3\xa9llo, is decoded by loads as latin-1 into héllo. The HMAC is over the bytes (which are intact), so the signature still verifies and the corrupted value is returned to the caller as if it were valid. A minimal reproduction:

from django.core.signing import JSONSerializer

# What a UTF-8-emitting serializer (e.g. orjson) would produce for {"text": "héllo"}
external_bytes = b'{"text":"h\xc3\xa9llo"}'

# Django's loads mojibakes it
assert JSONSerializer().loads(external_bytes) == {"text": "héllo"}  # silently corrupted
assert JSONSerializer().loads(external_bytes) != {"text": "héllo"}  # the real value is lost

Diagnosis

The latin-1 step was introduced in commit [58a086acfb] (Oct 2012, "Required serializer to use bytes in loads/dumps") as part of the Python 3 bytes-interface porting, chosen for consistency with the bytes interface rather than because latin-1 was required. With the default ensure_ascii=True serializer the output is always pure ASCII, so latin-1 and UTF-8 produce identical bytes and the bug is invisible — which is why it has gone unfixed since 2012. It surfaces the moment any non-default serializer is used.

The bug was surfaced while analyzing the pluggable JSON backend proposal (​#187), which had to carve core/signing.py out of the swappable backend because of this latin-1 step.

Scope

Only django/core/signing.py is in scope. Two contrib consumers are transitively covered because they use signing through the public API:

• django/contrib/sessions/serializers.py — JSONSerializer is a re-export of signing.JSONSerializer.
• django/contrib/sessions/backends/base.py and django/contrib/sessions/backends/signed_cookies.py — use signing.dumps / signing.loads.

django/contrib/messages/storage/cookie.py is not covered by this fix: it has its own MessagePartGatherSerializer.dumps and MessageSerializer.loads that call json.dumps(...).encode("latin-1") and json.loads(data.decode("latin-1")) directly (lines 68 and 73), independent of signing.JSONSerializer. That is the same bug pattern on a separate code path. The two are intentionally kept as separate tickets and separate PRs: they touch different modules, share no code path, and can land and be reverted independently. Keeping them separate also keeps each diff minimal and the review focused on one contract at a time. django/contrib/admin does not use signing directly.

Proposed direction

The natural fix is to switch the encode/decode step from latin-1 to utf-8 in both dumps and loads, leaving the separators=(",", ":") argument untouched so determinism is preserved. The exact shape of the change is left to the pull request.

Backwards compatibility

Fully backward-compatible. Existing signed tokens contain only ASCII bytes (ensure_ascii=True escapes every non-ASCII character to \uXXXX), and ASCII decodes identically under latin-1 and UTF-8, so every token already in the wild verifies and decodes to the same value. The compress=True path is covered by the same argument: existing compressed tokens are ASCII.

New tokens with non-ASCII data will contain raw UTF-8 bytes instead of \uXXXX escapes, so they are slightly shorter and round-trip correctly through the new loads.

Suggested test coverage

Tests should be added to the existing signing test module (tests/signing/tests.py) and should cover:

• a regression test that feeds raw UTF-8 bytes (as a UTF-8-emitting external serializer would produce — not bytes JSONSerializer.dumps could ever produce, since ensure_ascii=True escapes everything to ASCII) to loads and asserts the original value is returned, not the latin-1 mojibake — this is the test that fails before the fix and passes after;
• a round-trip of a non-ASCII payload (including combining marks and non-BMP characters) through dumps/loads.

The dumps encode step is not independently testable with the current JSONSerializer (ensure_ascii=True makes its output always ASCII), so no dumps-side regression test is warranted; that change is forward-looking and kept for symmetry with loads.

Related

• ​PR #21651 — Pull request implementing this fix.
• ​new-features #187 — Pluggable JSON serialization/deserialization backend (this fix removes the signing carve-out from step 1).
• ​django-orjson PR #15 — The django-orjson maintainer documented this as a one-way migration limitation (warning boxes + xfail tests for the cross-serializer case), not a user-reported data loss incident. Authoritative source for the mojibake behavior described here.
• Django's contrib/sessions/backends/signed_cookies.py — uses signing.dumps(serializer=self.serializer) where self.serializer is SESSION_SERIALIZER; sessions routinely contain non-ASCII data (user names, locale), so the mixing scenario is reachable from Django core, not only from third-party packages.
• Commit [58a086acfb] — introduced the latin-1 encode/decode step (Oct 2012).