Opened 6 years ago

Last modified 10 months ago

#15578 new Cleanup/optimization

loaddata and processing order of fixtures

Reported by: Luc Saffre Owned by: nobody
Component: Documentation Version: 1.2
Severity: Normal Keywords:
Cc: Triage Stage: Accepted
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: yes
Easy pickings: no UI/UX: no


As the documentation for loaddata correctly says: "the order in which fixture files are processed is undefined."
That's a pity because I use fixtures to provide combinable and interdependent sets of demo data.

Would it be difficult to implement a rule for the order in which fixtures are loaded?
I'd suggest the following rule:
If there are 3 applications "a", "b" and "c" and you invoke loaddata foo bar baz

(or you specify

fixtures = ['foo','bar','baz']

in a TestCase, then loaddata would first load all fixtures 'foo' for all applications, then all fixtures named 'bar', then all fixtures 'baz'.

Attachments (1)

doc_patch.diff (1.0 KB) - added by trez 5 years ago.
doc path for fixture ordering - maybe just a draft

Download all attachments as: .zip

Change History (9)

comment:1 Changed 5 years ago by Luke Plant

Type: New feature

comment:2 Changed 5 years ago by Luke Plant

Severity: Normal

comment:3 Changed 5 years ago by Jacob

Component: UncategorizedCore (Management commands)
Needs documentation: unset
Needs tests: unset
Patch needs improvement: unset
Triage Stage: UnreviewedDesign decision needed
Type: New featureCleanup/optimization

Interesting - that clause in the docs dates from r4659, the original commit of fixtures. But as far as I can tell, it's never been true: the code actually does exactly what you suggest and does load fixtures in the order given. I think we should just strike that paragraph from the docs, but I'd like to see if Russ remembers why he wrote that in the first place.

comment:4 Changed 5 years ago by Russell Keith-Magee

Component: Core (Management commands)Documentation
Triage Stage: Design decision neededAccepted

The statement is still accurate -- just imprecise.

Firstly, fixtures *are* loaded in the "foo", "bar", "baz" order. However, "foo" doesn't refer to a single fixture; any fixture named "foo" *in any app* will be loaded. This is what allows the "initial_data" load to occur, but the same trick works for any fixture name. The admonition that is there is intended to wave people off of the idea that specifying fixture order is enough to guarantee 100% predictable. This isn't true -- you also need to take into account the order in which applications are defined.

The admonition was probably also a partial "commit to nothing" statement, giving us some wiggle room if we needed to change things. Fixtures have been in the water for long enough that this reason no longer holds.

So I agree - fixing the documentation is all that is required here. Just deleting the line is the simple approach; a slightly better approach would expand the statement to include a the app ordering part of the process.

comment:5 Changed 5 years ago by Luc Saffre

Glad to read that this is just a documentation bug. Maybe something like this:

If several fixtures are specified, then the application order is also important: the first fixture is loaded first for all applications (in the order specified by INSTALLED_APPS), then the second fixture, and so on.

Changed 5 years ago by trez

Attachment: doc_patch.diff added

doc path for fixture ordering - maybe just a draft

comment:6 Changed 5 years ago by trez

Easy pickings: unset
Has patch: set
UI/UX: unset

The patch attached contains this explanation:

SQL data files are executed in the same order than their ordering
in the fixtures.

For example:

fixtures = ['foo','bar','baz']
foo will be executed first then bar, then baz.

However, fixtures are not loaded for one application but for any
applications which contains this fixture.

Therefore, you should also take into account the order in which application
are defined.

Hope this help but may surely be crystal clear with some improvements

comment:7 Changed 5 years ago by Aymeric Augustin

Patch needs improvement: set

There are a couple issues with this patch.

It should change the following sentences in ref/django-admin.txt:

  • "Note that the order in which fixture files are processed is undefined." — the section about fixtures in topics/testing.txt links to this paragraph.
  • "Note that the order in which the SQL files are processed is undefined." — not sure about this one, but if the order is deterministic, let's fix it too.

Finally, the patch isn't relative to the trunk directory of the source distribution.

comment:8 Changed 10 months ago by pascal chambon

I'd like to highlight another imprecise point in this documentation paragraph : as described in (duplicate) ticket #15578, even if the order of fixture import gets precised, users may not always rely on the "in transaction" behaviour of loaddata, regarding relationships between records.

Indeed DBs like MySQL do not have defered constraint checking, so referential integrity must be true for *each* SQL operation, not just as a whole. Others like PostgreSQL can have it on demand (, but I have alas no experience with these and the way Django uses them.

Does anybody have experience with loading interdependent fixtures on several different engines ? Can we simply state "If your engine supports defered constraint checking etc. etc." ?

Note: See TracTickets for help on using tickets.
Back to Top