Opened 62 minutes ago
#37442 assigned Bug
JSONL deserializer incorrectly treats Unicode line separators as record boundaries
| Reported by: | Patel Namraa | Owned by: | Patel Namraa |
|---|---|---|---|
| Component: | Core (Serialization) | Version: | dev |
| Severity: | Normal | Keywords: | |
| Cc: | Patel Namraa | Triage Stage: | Unreviewed |
| Has patch: | yes | Needs documentation: | no |
| Needs tests: | no | Patch needs improvement: | no |
| Easy pickings: | no | UI/UX: | no |
Description
The JSONL deserializer incorrectly uses str.splitlines() to split string and bytes input into records. splitlines() treats U+2028 (LINE SEPARATOR), U+2029 (PARAGRAPH SEPARATOR), and U+0085 (NEXT LINE) as line boundaries, even though these characters can occur unescaped inside JSON strings.
The JSONL serializer uses ensure_ascii=False, so these characters can be emitted directly in serialized JSON. As a result, data produced by Django's JSONL serializer cannot always be deserialized again when a serialized value contains one of these characters.
For example:
from django.core import serializers
from myapp.models import Author
author = Author(pk=1, name="Jane\u2028Doe")
data = serializers.serialize("jsonl", [author])
list(serializers.deserialize("jsonl", data))
This raises DeserializationError because the record is split at U+2028 instead of only at the JSONL record delimiter.
The same behavior occurs with U+2029 and U+0085, and with both str and bytes input.
The JSONL serializer uses \n as the record separator, so the deserializer should split only on \n rather than using splitlines().
A possible fix is:
stream_or_string = stream_or_string.split("\n")
This continues to support CRLF input because the resulting line retains a trailing \r, which is valid JSON whitespace.
I have added regression tests covering U+2028, U+2029, and U+0085 with both str and bytes input, as well as a test confirming that CRLF line endings continue to work.
The issue does not affect loading JSONL from a file object in the same way, since file iteration uses the actual newline delimiter. However, it affects callers passing serialized JSONL as a string or bytes object to serializers.deserialize().
I have reproduced the issue against Django's current development version.