#15578 closed Cleanup/optimization (fixed)
loaddata and processing order of fixtures
| Reported by: | Luc Saffre | Owned by: | Leo Suarez |
|---|---|---|---|
| Component: | Documentation | Version: | 1.2 |
| Severity: | Normal | Keywords: | |
| Cc: | Triage Stage: | Accepted | |
| Has patch: | yes | Needs documentation: | no |
| Needs tests: | no | Patch needs improvement: | yes |
| Easy pickings: | no | UI/UX: | no |
Description
As the documentation for loaddata correctly says: "the order in which fixture files are processed is undefined."
That's a pity because I use fixtures to provide combinable and interdependent sets of demo data.
Would it be difficult to implement a rule for the order in which fixtures are loaded?
I'd suggest the following rule:
If there are 3 applications "a", "b" and "c" and you invoke
manage.py loaddata foo bar baz
(or you specify
fixtures = ['foo','bar','baz']
in a TestCase, then loaddata would first load all fixtures 'foo' for all applications, then all fixtures named 'bar', then all fixtures 'baz'.
Attachments (1)
Change History (18)
comment:1 by , 13 years ago
| Type: | → New feature |
|---|
comment:2 by , 13 years ago
| Severity: | → Normal |
|---|
comment:3 by , 13 years ago
| Component: | Uncategorized → Core (Management commands) |
|---|---|
| Triage Stage: | Unreviewed → Design decision needed |
| Type: | New feature → Cleanup/optimization |
comment:4 by , 13 years ago
| Component: | Core (Management commands) → Documentation |
|---|---|
| Triage Stage: | Design decision needed → Accepted |
The statement is still accurate -- just imprecise.
Firstly, fixtures *are* loaded in the "foo", "bar", "baz" order. However, "foo" doesn't refer to a single fixture; any fixture named "foo" *in any app* will be loaded. This is what allows the "initial_data" load to occur, but the same trick works for any fixture name. The admonition that is there is intended to wave people off of the idea that specifying fixture order is enough to guarantee 100% predictable. This isn't true -- you also need to take into account the order in which applications are defined.
The admonition was probably also a partial "commit to nothing" statement, giving us some wiggle room if we needed to change things. Fixtures have been in the water for long enough that this reason no longer holds.
So I agree - fixing the documentation is all that is required here. Just deleting the line is the simple approach; a slightly better approach would expand the statement to include a the app ordering part of the process.
comment:5 by , 13 years ago
Glad to read that this is just a documentation bug. Maybe something like this:
If several fixtures are specified, then the application order is also important: the first fixture is loaded first for all applications (in the order specified by INSTALLED_APPS), then the second fixture, and so on.
by , 13 years ago
| Attachment: | doc_patch.diff added |
|---|
doc path for fixture ordering - maybe just a draft
comment:6 by , 13 years ago
| Easy pickings: | unset |
|---|---|
| Has patch: | set |
| UI/UX: | unset |
The patch attached contains this explanation:
SQL data files are executed in the same order than their ordering
in the fixtures.
For example:
fixtures = ['foo','bar','baz']
foo will be executed first then bar, then baz.
However, fixtures are not loaded for one application but for any
applications which contains this fixture.
Therefore, you should also take into account the order in which application
are defined.
Hope this help but may surely be crystal clear with some improvements
comment:7 by , 13 years ago
| Patch needs improvement: | set |
|---|
There are a couple issues with this patch.
It should change the following sentences in ref/django-admin.txt:
- "Note that the order in which fixture files are processed is undefined." — the section about fixtures in
topics/testing.txtlinks to this paragraph. - "Note that the order in which the SQL files are processed is undefined." — not sure about this one, but if the order is deterministic, let's fix it too.
Finally, the patch isn't relative to the trunk directory of the source distribution.
comment:8 by , 8 years ago
I'd like to highlight another imprecise point in this documentation paragraph : as described in (duplicate) ticket #15578, even if the order of fixture import gets precised, users may not always rely on the "in transaction" behaviour of loaddata, regarding relationships between records.
Indeed DBs like MySQL do not have defered constraint checking, so referential integrity must be true for *each* SQL operation, not just as a whole. Others like PostgreSQL can have it on demand (http://www.postgresql.org/docs/9.1/static/sql-set-constraints.html), but I have alas no experience with these and the way Django uses them.
Does anybody have experience with loading interdependent fixtures on several different engines ? Can we simply state "If your engine supports defered constraint checking etc. etc." ?
comment:9 by , 3 years ago
Is this patch better framed as the following?
Fixtures are processed in the same order as the one specified in the list.
This means that if Fixtures = ['foo', 'bar', 'baz'] then the order in which they are loaded will be 'foo' followed by 'bar' and then 'baz'. Where, "foo" is any fixture named 'foo' in any application. This implies that in cases where there are several fixtures with the same name, the order in which applications are defined in INSTALLED_APPS is also a taken into account.
This order may not always be predictable. If the database backend supports row-level constraints, these constraints will be checked at the end of the transaction.
comment:10 by , 7 months ago
| Owner: | changed from to |
|---|---|
| Status: | new → assigned |
Opened PR to address this item. https://github.com/django/django/pull/17384
Interesting - that clause in the docs dates from r4659, the original commit of fixtures. But as far as I can tell, it's never been true: the code actually does exactly what you suggest and does load fixtures in the order given. I think we should just strike that paragraph from the docs, but I'd like to see if Russ remembers why he wrote that in the first place.