Opened 6 years ago

Last modified 6 years ago

#22088 assigned Bug

XML deserializer strips leading whitespace on loaddata

Reported by: Joseph-django@… Owned by: Martin Matusiak
Component: Core (Serialization) Version: 1.6
Severity: Normal Keywords: xml deserialization
Cc: numerodix@… Triage Stage: Accepted
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no


If an object instance has a character field and the value of that field starts with the tab character, loaddata removes that tab character when the loaded fixture is in XML format.

Note that the XML dump data does not strip this leading tab character. Also note that both the JSON dump and load data preserve the tab character.

I have not tested this with other whitespace characters. This can be easily reproduced by creating a simple model:

class Foobar(models.model)
  name = models.CharField(max_length=20)

And then creating an instance of that model (e.g, in Django Shell) with a name value of, e.g, "\tBaz" and then using the dumpdata with --format=xml.

Once the fixture has been generated, remove the existing instance (either by deleting it, or flushing the app data, or your preferred method) and then using loaddata to load the fixture. Note the instance's name no longer contains the tab character.

Change History (8)

comment:1 Changed 6 years ago by Martin Matusiak

Cc: numerodix@… added
Owner: changed from nobody to Martin Matusiak
Status: newassigned

comment:2 Changed 6 years ago by Martin Matusiak

I can reproduce it.

comment:3 Changed 6 years ago by Martin Matusiak

Trying with a few special characters:

Foobar.objects.create(name=' bar')

XML pretty printer corrupts this completely:

    <object pk="9" model="app.foobar">
        <field type="CharField" name="name"> bar</field>
    <object pk="10" model="app.foobar">
        <field type="CharField" name="name">bar</field>
    <object pk="11" model="app.foobar">
        <field type="CharField" name="name"bar</field>
    <object pk="12" model="app.foobar">
        <field type="CharField" name="name">
    <object pk="13" model="app.foobar">
        <field type="CharField" name="name">
    <object pk="14" model="app.foobar">
bar</field>eld type="CharField" name="name">
    <object pk="15" model="app.foobar">
        <field type="CharField" name="name">    bar</field>
    <object pk="16" model="app.foobar">
        <field type="CharField" name="name">

In terse mode it's more likely to be correct when loaded again, but clearly this needs fixing:

<object pk="9" model="app.foobar"><field type="CharField" name="name"> bar</field></object><object pk="10" model="app.foobar"><field type="CharField" name="name">bar</field></object><object pk="11" model="app.foobar"><field type="CharField" name="name"bar</field></object><object pk="12" model="app.foobar"><field type="CharField" name="name">
                               bar</field></object><object pk="13" model="app.foobar"><field type="CharField" name="name">
bar</field></object><object pk="15" model="app.foobar"><field type="CharField" name="name">     bar</field></object><object pk="16" model="app.foobar"><field type="CharField" name="name">

So that's dumpdata at fault, not loaddata.

comment:4 Changed 6 years ago by Martin Matusiak

Another experiment: hand edit the xml dump file using html escapes (1) to see if loaddata will load it correctly. No, also doesn't work.

Problem 1: The xml serializers (SimplerXMLGenerator/pulldom) do not round trip these characters.
Problem 2: Even if they did a tab character would be stripped due to:

value = field.to_python(getInnerText(field_node).strip())


comment:5 in reply to:  3 Changed 6 years ago by Daniele Procida

Triage Stage: UnreviewedAccepted

Replying to numerodix:

    <object pk="11" model="app.foobar">
        <field type="CharField" name="name"bar</field>

Wow. If that's what you're getting in the dumped text file, that's remarkable.

Last edited 6 years ago by Daniele Procida (previous) (diff)

comment:6 Changed 6 years ago by Martin Matusiak

What we could do is try to wrap any special characters in a CDATA section. So the xml would look like this:

<object pk="2" model="app.foobar"><field type="CharField" name="name"><![CDATA[\t]]>bar</field></object>

The deserializer then gives us the tab character escaped:


So we'd have to strip the escape.

But this feels very ad-hoc and messy.

comment:7 Changed 6 years ago by jarshwah

Shouldn't all straight up text (CharField/TextField) be wrapped in a CDATA though? Then there's no need to look for special characters at all. Are there any downsides to wrapping all text values in a CDATA? Unsure why the deserialiser would escape the tab though (I haven't looked into it), but that is sort of a separate - related - issue.

comment:8 Changed 6 years ago by Martin Matusiak

@smeaton, I think it probably should, yes. The downside would be that you're adding 12 bytes to every value even though most strings would not need it.

One could optimize that by wrapping only strings that need to be wrapped, according to a character range or something like that.

This serializer is also used by the syndication feed framework btw. It would be nice to fix the problem in both places at once.

Note: See TracTickets for help on using tickets.
Back to Top