id	summary	reporter	owner	description	type	status	component	version	severity	resolution	keywords	cc	stage	has_patch	needs_docs	needs_tests	needs_better_patch	easy	ui_ux
37223	Signing's JSONSerializer uses latin-1, causing silent mojibake with UTF-8-emitting serializers	Paolo Melchiorre	Paolo Melchiorre	"== Summary ==

`django.core.signing.JSONSerializer` encodes its JSON payload with latin-1 and decodes it with latin-1. With the default `ensure_ascii=True` serializer the output is pure ASCII, so latin-1 and UTF-8 are indistinguishable and the bug is invisible. But any serializer that emits raw UTF-8 (orjson, msgspec, `json.dumps(..., ensure_ascii=False)`) produces non-ASCII bytes for non-ASCII data, and `JSONSerializer.loads` silently mojibakes them when decoding as latin-1. No `BadSignature` and no `UnicodeDecodeError` is raised — the data is returned corrupted as if it were valid.

Both `SESSION_SERIALIZER` and `signing.dumps(serializer=...)` are documented extension points. The scenario that bites is the rollback: a project runs a UTF-8-emitting serializer (e.g. `OrjsonSerializer` for `SESSION_SERIALIZER`), then switches back to the default — every token signed while the UTF-8 serializer was active comes back mojibaked, with no `BadSignature` and no `UnicodeDecodeError`. A staggered deploy across mixed patched/unpatched processes is the same class of risk, just less common. The django-orjson maintainer documented this as a one-way migration limitation in [https://github.com/adamchainz/django-orjson/pull/15 django-orjson PR #15] (warning boxes in `docs/sessions.rst` and `docs/signing.rst`, plus `xfail` tests for the cross-serializer case), rather than fixing it — the fix has to live in Django. This is a maintainer-documented limitation, not a user-reported data loss incident.

== How to reproduce ==

`JSONSerializer` currently does:

{{{#!python
# django/core/signing.py
class JSONSerializer:
    def dumps(self, obj):
        return json.dumps(obj, separators=("","", "":"")).encode(""latin-1"")

    def loads(self, data):
        return json.loads(data.decode(""latin-1""))
}}}

A payload containing `héllo`, written by a UTF-8-emitting serializer as the bytes `h\xc3\xa9llo`, is decoded by `loads` as latin-1 into `héllo`. The HMAC is over the bytes (which are intact), so the signature still verifies and the corrupted value is returned to the caller as if it were valid. A minimal reproduction:

{{{#!python
from django.core.signing import JSONSerializer

# What a UTF-8-emitting serializer (e.g. orjson) would produce for {""text"": ""héllo""}
external_bytes = b'{""text"":""h\xc3\xa9llo""}'

# Django's loads mojibakes it
assert JSONSerializer().loads(external_bytes) == {""text"": ""héllo""}  # silently corrupted
assert JSONSerializer().loads(external_bytes) != {""text"": ""héllo""}  # the real value is lost
}}}

== Diagnosis ==

The latin-1 step was introduced in commit [58a086acfb] (Oct 2012, ""Required serializer to use bytes in loads/dumps"") as part of the Python 3 bytes-interface porting, chosen for consistency with the bytes interface rather than because latin-1 was required. With the default `ensure_ascii=True` serializer the output is always pure ASCII, so latin-1 and UTF-8 produce identical bytes and the bug is invisible — which is why it has gone unfixed since 2012. It surfaces the moment any non-default serializer is used.

The bug was surfaced while analyzing the pluggable JSON backend proposal ([https://github.com/django/new-features/issues/187 #187]), which had to carve `core/signing.py` out of the swappable backend because of this latin-1 step.

== Scope ==

Only `django/core/signing.py` is in scope. Two `contrib` consumers are transitively covered because they use `signing` through the public API:

 * `django/contrib/sessions/serializers.py` — `JSONSerializer` is a re-export of `signing.JSONSerializer`.
 * `django/contrib/sessions/backends/base.py` and `django/contrib/sessions/backends/signed_cookies.py` — use `signing.dumps` / `signing.loads`.

`django/contrib/messages/storage/cookie.py` is '''not''' covered by this fix: it has its own `MessagePartGatherSerializer.dumps` and `MessageSerializer.loads` that call `json.dumps(...).encode(""latin-1"")` and `json.loads(data.decode(""latin-1""))` directly (lines 68 and 73), independent of `signing.JSONSerializer`. That is the same bug pattern on a separate code path. The two are intentionally kept as separate tickets and separate PRs: they touch different modules, share no code path, and can land and be reverted independently. Keeping them separate also keeps each diff minimal and the review focused on one contract at a time. `django/contrib/admin` does not use `signing` directly.

== Proposed direction ==

The natural fix is to switch the encode/decode step from `latin-1` to `utf-8` in both `dumps` and `loads`, leaving the `separators=("","", "":"")` argument untouched so determinism is preserved. The exact shape of the change is left to the pull request.

== Backwards compatibility ==

Fully backward-compatible. Existing signed tokens contain only ASCII bytes (`ensure_ascii=True` escapes every non-ASCII character to `\uXXXX`), and ASCII decodes identically under latin-1 and UTF-8, so every token already in the wild verifies and decodes to the same value. The `compress=True` path is covered by the same argument: existing compressed tokens are ASCII.

New tokens with non-ASCII data will contain raw UTF-8 bytes instead of `\uXXXX` escapes, so they are slightly shorter and round-trip correctly through the new `loads`.

== Suggested test coverage ==

Tests should be added to the existing signing test module (tests/signing/tests.py) and should cover:

 * a regression test that feeds raw UTF-8 bytes (as a UTF-8-emitting external serializer would produce — not bytes `JSONSerializer.dumps` could ever produce, since `ensure_ascii=True` escapes everything to ASCII) to loads and asserts the original value is returned, not the latin-1 mojibake — this is the test that fails before the fix and passes after;
 * a round-trip of a non-ASCII payload (including combining marks and non-BMP characters) through dumps/loads.

The dumps encode step is not independently testable with the current JSONSerializer (ensure_ascii=True makes its output always ASCII), so no dumps-side regression test is warranted; that change is forward-looking and kept for symmetry with loads.

== Related ==

 * [https://github.com/django/django/pull/21651 PR #21651] — Pull request implementing this fix.
 * [https://github.com/django/new-features/issues/187 new-features #187] — Pluggable JSON serialization/deserialization backend (this fix removes the signing carve-out from step 1).
 * [https://github.com/adamchainz/django-orjson/pull/15 django-orjson PR #15] — The django-orjson maintainer documented this as a one-way migration limitation (warning boxes + `xfail` tests for the cross-serializer case), not a user-reported data loss incident. Authoritative source for the mojibake behavior described here.
 * Django's `contrib/sessions/backends/signed_cookies.py` — uses `signing.dumps(serializer=self.serializer)` where `self.serializer` is `SESSION_SERIALIZER`; sessions routinely contain non-ASCII data (user names, locale), so the mixing scenario is reachable from Django core, not only from third-party packages.
 * Commit [58a086acfb] — introduced the latin-1 encode/decode step (Oct 2012).
"	New feature	closed	Core (Other)	dev	Normal	needsnewfeatureprocess	signing latin1 utf8 json	Andy Miller	Unreviewed	1	0	0	0	0	0