Opened 9 years ago

Last modified 2 months ago

#11927 assigned New feature

Allow manage.py dumpdata to dump YAML in block style

Reported by: sampablokuper Owned by: Matthijs Kooijman
Component: Core (Management commands) Version: master
Severity: Normal Keywords:
Cc: sam.kuper@…, tehfink, Matthijs Kooijman Triage Stage: Accepted
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: yes
Easy pickings: no UI/UX: no

Description

Using

python manage.py dumpdata --format=yaml

produces output in flow style. That's OK for automated processing, but it isn't anything like as human-readable as block style YAML. For people who, like me, need to edit the fixtures manually, it would be very useful to be able to do something like:

python manage.py dumpdata --format=yaml --flowstyle=false

Attachments (2)

flow_style.patch (642 bytes) - added by Aviral Dasgupta 8 years ago.
yaml_dump_style.diff (2.8 KB) - added by Aviral Dasgupta 8 years ago.

Download all attachments as: .zip

Change History (24)

comment:1 Changed 9 years ago by sampablokuper

Cc: sam.kuper@… added

Added to CC list.

comment:2 Changed 9 years ago by Russell Keith-Magee

Triage Stage: UnreviewedAccepted

Changed 8 years ago by Aviral Dasgupta

Attachment: flow_style.patch added

comment:3 Changed 8 years ago by Aviral Dasgupta

Has patch: set
Owner: changed from nobody to Aviral Dasgupta

comment:4 Changed 8 years ago by Aviral Dasgupta

Needs documentation: set
Needs tests: set
Patch needs improvement: set

comment:5 Changed 8 years ago by Aviral Dasgupta

milestone: 1.3
Resolution: fixed
Status: newclosed

comment:6 Changed 8 years ago by Karen Tracey

Resolution: fixed
Status: closedreopened

It's not fixed until the code is checked into SVN.

comment:7 Changed 8 years ago by Ramiro Morales

Current patch simply changes situation from hardcoded flow style output to hardcoded not flow styled output and this isn't really an overall enhancement. This should be selectable from the command line (defaulting to the current style?) as proposed by the OP. Otherwise IMHO it has little changes of being committed.

Changed 8 years ago by Aviral Dasgupta

Attachment: yaml_dump_style.diff added

comment:8 Changed 8 years ago by Aviral Dasgupta

Patch needs improvement: unset
Version: 1.1SVN

Okay, I've added a new patch. With this one, you can use --flowstyle for flow style and --blockstyle for block style YAML dumps.

comment:9 Changed 8 years ago by Łukasz Rekucki

milestone: 1.3

It still needs docs and tests. Also, it's a feature, so not going to make it for 1.3;

comment:10 Changed 8 years ago by Russell Keith-Magee

Patch needs improvement: set

I'm also not wild about the idea of having 2 options to control one flag. You can't specify both --flow and --block, so why have two options? I'd rather see just --block (enabling the new block style), or if there are likely to be more styling options, a --style=block argument.

comment:11 Changed 7 years ago by Julien Phalip

Severity: Normal
Type: New feature

comment:12 Changed 7 years ago by Aymeric Augustin

UI/UX: unset

Change UI/UX from NULL to False.

comment:13 Changed 7 years ago by Aymeric Augustin

Easy pickings: unset

Change Easy pickings from NULL to False.

comment:14 Changed 7 years ago by Aviral Dasgupta

Owner: Aviral Dasgupta deleted
Status: reopenednew

comment:15 Changed 4 years ago by blockstyle@…

--blockstyle option not found?!! I'm dumping my fixtures as yaml. The flowstyle is a pita to see.
Why isn't it available? I'm using Django 1.6

comment:16 Changed 4 years ago by Simon Charette

This ticket patch needs improvement based on @russellm comments, tests and documentation.

comment:17 Changed 3 years ago by Xiao Hanyu

Owner: set to Xiao Hanyu
Status: newassigned

comment:18 Changed 8 months ago by tehfink

Cc: tehfink added

comment:19 Changed 4 months ago by Matthijs Kooijman

Cc: Matthijs Kooijman added
Component: Core (Serialization)Core (Management commands)
Needs tests: unset
Owner: changed from Xiao Hanyu to Matthijs Kooijman
Patch needs improvement: unset
Triage Stage: AcceptedUnreviewed

I've taken the time time to redo this patch and address the comments on the previous patch. The patch now also adds tests and documentation.

The PyYAML docs say the following:

By default, PyYAML chooses the style of a collection depending on whether it has nested collections. If a collection has nested collections, it will be assigned the block style. Otherwise it will have the flow style.

If you want collections to be always serialized in the block style, set the parameter default_flow_style of dump() to False. For instance,

In other words, this option configures the style for collections, and there are three options: Always use the flow style, always use the block style, or the default of automatically selecting. To match that, I've added a --yaml-collection-style option that takes auto, flow or block option.

Some uncertainties / notes when reviewing the patch:

  • The option name is a lot more verbose than the --style option Russel proposed. However, since it is only relevant to the yaml formatter, I've made the option name more verbose.
  • The code in django/core/management/commands/dumpdata.py only passes the option on to the serializer when the current formatter is 'yaml'. This might be a bit hardcoded, but I didn't see an easy and elegant way to do this otherwise?
  • I've left the default at "auto" as before, but I suspect that a default of "block" might be more appropriate for loaddata. The current mixed style likely produces more user surprise that the more verbose block style would, as well as being more appropriate for version control. Would it make sense to change the default? If so, should the default of the serializer class be changed as well, or just the dumpdata command?

Given this is a new feature, I've targeted the docs at 2.2.

The patch is available on github: https://github.com/django/django/pull/10117

comment:20 Changed 4 months ago by Tim Graham

Needs documentation: unset
Triage Stage: UnreviewedAccepted

comment:21 Changed 3 months ago by Tim Graham

Patch needs improvement: set

comment:22 Changed 2 months ago by Matthijs Kooijman

Meanwhile, on github Tim Graham has reviewed the patch and suggested to generalize the concept of format-specific commandline options, rather than hardcoding it as I had done. I've implemented this, but ran into some problems with the testing framework. See the PR for comments and further discussion.

Meanwhile, I also tried changing the default style from the current mixed style, to always using the block style (which I think is a better style for dumping model data). However, I noticed that this style also changes plain lists, not just mappings. E.g., a model with a many2manyfield ends up being serialized in block style too, which might not be a good default. For example the categories field in:

 - model: serializers.article
   pk: 1
   fields:
     author: 2
     headline: Poker has no place on ESPN
     pub_date: 2006-06-16 11:00:00
     categories:
     - 3
     - 1
     meta_data: []

Better would be to use the flow style for such simple lists:

    categories: [3, 1]

My current patch has three styles, 'flow' (everything flow style, single line), 'block' (everything block style, like above) and 'mixed' (the PyYAML default with flow style for containers with scalars and block style for containers with other containers.

I can imagine a fourth style, which uses block for all mappings, flow style for sequences with scalars and block style for sequences with containers. This is is essentially the PyYAML default style, except that mappings always use the block style. A quick test shows this is easy to implement by overriding the represent_mapping() method on DjangSafeDumper to force flow_style=False. Not sure how to name this style, though. It could also be described as "mixed", but I do not want to drop the PyYAML default style. The latter could be renamed to "default" or even "pyyaml-default", or perhaps this new style could be more explicitly called "block-mappings" to indicate it uses block style for mappings (without indicating the rest).

Any other thoughts?

Note: See TracTickets for help on using tickets.
Back to Top