django-admin.py dumpdata should be deterministic for VCS and diff friendliness

[Geoffrey Fairchild]

I have several projects in which I like to store fixtures that are shared between project instances (e.g., development and production instances). Since fixtures are just text files, they're easily stored in version control; I use git for all my projects.

The problem I'm experiencing is that the dumpdata management command outputs data differently every time it runs (I'm on Django 1.7.7, so it may be different on other versions of Django). I understand why it does this - a lot of the data are stored in dicts or sets, and dicts and sets don't provide any promises on ordering. The reason this is a problem is because, for large fixtures, this can cause significant changes to be perceived by the VCS. Git is pretty smart, but if a 10mb fixture is completely reordered, it can't actually show me what's changed when I do a diff. Additionally, if I re-dump all the data in my database, even if the data haven't changed, git will detect that the files are different because all the data are in a different order.

The feature I'd like is for the dumpdata command to be deterministic; that is, every time it runs, it should produce the same output for the same input. This could even be an option that's turned off by default. This will reduce VCS thrashing and improve the ability for us to diff fixtures in order to understand what's actually changed.

I imagine this could fairly easily be solved by tossing a few sorted statements in the right places, but I'm not sure.

[Claude Paroz]

Agreed that deterministic ordering is a desirable goal. Now can you tell us what sort of changed ordering are you seeing in your data? It should not be with objects themselves, as they are sorted by their primary key. Is it an issue with serializers.sort_dependencies which gives the model ordering?

[Geoffrey Fairchild]

terms dump 1

[Geoffrey Fairchild]

terms dump 2

[Geoffrey Fairchild]

Sure, so here's an example model I have in my swap app:

class Term(models.Model):
    term = models.CharField(max_length=255)
    definition = models.TextField()
    reference = models.TextField()

I've attached the output from two consecutive dumps (./manage.py dumpdata swap.Term).

You can see that the content of each file is identical. And as you say, the objects are indeed sorted by their primary key. The ordering issue is with the key-value pairs printed for each object. Each object is just a dictionary of keys and values, so there's no ordering maintained.

[Simon Charette <charette.s@…>]

In 5bc31234:

Fixed #24558 -- Made dumpdata mapping ordering deterministic.

Thanks to gfairchild for the report and Claude for the review.