#37442 assigned Bug

JSONL deserializer incorrectly treats Unicode line separators as record boundaries

Reported by: Patel Namraa Owned by: Patel Namraa
Component: Core (Serialization) Version: dev
Severity: Normal Keywords:
Cc: Patel Namraa Triage Stage: Unreviewed
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

The JSONL deserializer incorrectly uses str.splitlines() to split string and bytes input into records. splitlines() treats U+2028 (LINE SEPARATOR), U+2029 (PARAGRAPH SEPARATOR), and U+0085 (NEXT LINE) as line boundaries, even though these characters can occur unescaped inside JSON strings.

The JSONL serializer uses ensure_ascii=False, so these characters can be emitted directly in serialized JSON. As a result, data produced by Django's JSONL serializer cannot always be deserialized again when a serialized value contains one of these characters.

For example:

from django.core import serializers
from myapp.models import Author

author = Author(pk=1, name="Jane\u2028Doe")
data = serializers.serialize("jsonl", [author])
list(serializers.deserialize("jsonl", data))

This raises DeserializationError because the record is split at U+2028 instead of only at the JSONL record delimiter.

The same behavior occurs with U+2029 and U+0085, and with both str and bytes input.

The JSONL serializer uses \n as the record separator, so the deserializer should split only on \n rather than using splitlines().

A possible fix is:

stream_or_string = stream_or_string.split("\n")

This continues to support CRLF input because the resulting line retains a trailing \r, which is valid JSON whitespace.

I have added regression tests covering U+2028, U+2029, and U+0085 with both str and bytes input, as well as a test confirming that CRLF line endings continue to work.

The issue does not affect loading JSONL from a file object in the same way, since file iteration uses the actual newline delimiter. However, it affects callers passing serialized JSONL as a string or bytes object to serializers.deserialize().

I have reproduced the issue against Django's current development version.

Change History (0)

Note: See TracTickets for help on using tickets.
Back to Top