Opened 18 months ago

Last modified 17 months ago

#22088 assigned Bug

XML deserializer strips leading whitespace on loaddata

Reported by: Joseph-django@… Owned by: numerodix
Component: Core (Serialization) Version: 1.6
Severity: Normal Keywords: xml deserialization
Cc: numerodix@… Triage Stage: Accepted
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no


If an object instance has a character field and the value of that field starts with the tab character, loaddata removes that tab character when the loaded fixture is in XML format.

Note that the XML dump data does not strip this leading tab character. Also note that both the JSON dump and load data preserve the tab character.

I have not tested this with other whitespace characters. This can be easily reproduced by creating a simple model:

class Foobar(models.model)
  name = models.CharField(max_length=20)

And then creating an instance of that model (e.g, in Django Shell) with a name value of, e.g, "\tBaz" and then using the dumpdata with --format=xml.

Once the fixture has been generated, remove the existing instance (either by deleting it, or flushing the app data, or your preferred method) and then using loaddata to load the fixture. Note the instance's name no longer contains the tab character.

Change History (8)

comment:1 Changed 17 months ago by numerodix

  • Cc numerodix@… added
  • Needs documentation unset
  • Needs tests unset
  • Owner changed from nobody to numerodix
  • Patch needs improvement unset
  • Status changed from new to assigned

comment:2 Changed 17 months ago by numerodix

I can reproduce it.

comment:3 follow-up: Changed 17 months ago by numerodix

Trying with a few special characters:

Foobar.objects.create(name=' bar')

XML pretty printer corrupts this completely:

    <object pk="9" model="app.foobar">
        <field type="CharField" name="name"> bar</field>
    <object pk="10" model="app.foobar">
        <field type="CharField" name="name">bar</field>
    <object pk="11" model="app.foobar">
        <field type="CharField" name="name"bar</field>
    <object pk="12" model="app.foobar">
        <field type="CharField" name="name">
    <object pk="13" model="app.foobar">
        <field type="CharField" name="name">
    <object pk="14" model="app.foobar">
bar</field>eld type="CharField" name="name">
    <object pk="15" model="app.foobar">
        <field type="CharField" name="name">    bar</field>
    <object pk="16" model="app.foobar">
        <field type="CharField" name="name">

In terse mode it's more likely to be correct when loaded again, but clearly this needs fixing:

<object pk="9" model="app.foobar"><field type="CharField" name="name"> bar</field></object><object pk="10" model="app.foobar"><field type="CharField" name="name">bar</field></object><object pk="11" model="app.foobar"><field type="CharField" name="name"bar</field></object><object pk="12" model="app.foobar"><field type="CharField" name="name">
                               bar</field></object><object pk="13" model="app.foobar"><field type="CharField" name="name">
bar</field></object><object pk="15" model="app.foobar"><field type="CharField" name="name">     bar</field></object><object pk="16" model="app.foobar"><field type="CharField" name="name">

So that's dumpdata at fault, not loaddata.

comment:4 Changed 17 months ago by numerodix

Another experiment: hand edit the xml dump file using html escapes (1) to see if loaddata will load it correctly. No, also doesn't work.

Problem 1: The xml serializers (SimplerXMLGenerator/pulldom) do not round trip these characters.
Problem 2: Even if they did a tab character would be stripped due to:

value = field.to_python(getInnerText(field_node).strip())


comment:5 in reply to: ↑ 3 Changed 17 months ago by EvilDMP

  • Triage Stage changed from Unreviewed to Accepted

Replying to numerodix:

    <object pk="11" model="app.foobar">
        <field type="CharField" name="name"bar</field>


Wow. If that's what you're getting in the dumped text file, that's remarkable.

Version 0, edited 17 months ago by EvilDMP (next)

comment:6 Changed 17 months ago by numerodix

What we could do is try to wrap any special characters in a CDATA section. So the xml would look like this:

<object pk="2" model="app.foobar"><field type="CharField" name="name"><![CDATA[\t]]>bar</field></object>

The deserializer then gives us the tab character escaped:


So we'd have to strip the escape.

But this feels very ad-hoc and messy.

comment:7 Changed 17 months ago by smeatonj

Shouldn't all straight up text (CharField/TextField) be wrapped in a CDATA though? Then there's no need to look for special characters at all. Are there any downsides to wrapping all text values in a CDATA? Unsure why the deserialiser would escape the tab though (I haven't looked into it), but that is sort of a separate - related - issue.

comment:8 Changed 17 months ago by numerodix

@smeaton, I think it probably should, yes. The downside would be that you're adding 12 bytes to every value even though most strings would not need it.

One could optimize that by wrapping only strings that need to be wrapped, according to a character range or something like that.

This serializer is also used by the syndication feed framework btw. It would be nice to fix the problem in both places at once.

Note: See TracTickets for help on using tickets.
Back to Top