Make serializers consistently unicode by default.

[hakib]

It is currently not (easily) possible to use the dumpdata managemet command on models with unicode data.
The JSON serializer used by dumpdata is not accepting the ensure_ascii argument used by json.dumps as an argument.

Since ensure_ascii=True is the default, I suggest adding a --dont-ensure-ascii flag to the dumpdata managemet command so it will be easier to use dumpdata with unicode.

./manage.py dumpdata app.model --dont-ensure-ascii

I'm not sure what are the implications on other serializers such as YAML, XML etc.

[Tim Graham]

I believe this would only apply to the JSON serializer, and I'm not sure about adding a dumpdata option that's specific to a particular serializer. I think your best solution is to subclass the JSON serializer, ​register it as a custom format, and then use that format in dumpdata.

[hakib]

The JSON serializer has a ensure_ascii attribute and the YAML serializer has a allow_unicode attribute. I already submitted a ​PR implementing the flag in both serializers.

I haven't looked at the XML serializer yet but I'm sure it will be possible there as well.

As someone who works with unicode as the primary language for most apps (as I'm sure a lot of other developers do) it's a very a useful feature to be able to dump fixtures directly from local db in a readable format.

[Tim Graham]

Okay. My main concern is that an option calls --allow_unicode may suggest to readers that all serializers prohibit unicode by default. That may not be true. Your patch also needs documentation.

[Hasan Ramezani]

​PR

[Mariusz Felisiak]

Current behavior is inconsistent. XML serializer use Unicode by default, on the other hand YAML and JSON serializers force ASCII. I think we should make this behavior consistent instead of adding a new serializer-specific option, i.e. pass allow_unicode=True to yaml.dump() and ensure_ascii=False to json.dump().

[Hasan Ramezani]

​PR

[Mariusz Felisiak <felisiak.mariusz@…>]

In 8970bb4:

Refs #29249 -- Added tests for serializing Unicode data with XML serializer.

[Mariusz Felisiak <felisiak.mariusz@…>]

In 68fc21b:

Fixed #29249 -- Made JSON and YAML serializers use Unicode by default.