initial_data fixtures using natural keys fail in tests with multiple databases

[David Cramer]

If a fixture using natural keys is referenced before that database is created (in tests), then the fixture fails to load due to a "table does not exist" error (though may also throw database does not exist, since it may not have been created yet).

The reason for this is that objects are instantiated before they are actually created in the database, but they mapper only does this if they're not using natural keys. In the case where they are, it attempts to look them up from the database, rather than the existing in-memory instances.

[David Cramer]

A resolution which may help with some other bad behavior would be to create all databases and then process fixtures. This would also stop the additional unnecessary parsing of the fixtures files repeatedly (N databases).

[Russell Keith-Magee]

I'm having difficulty in seeing how this situation would arise. Fixtures can't cross databases, and they are saved raw on the database they're being loaded onto. All test databases are created before the first test is created. I can't see how a test fixture -- which is loaded after all the databases have been configured -- could cause "table does not exist" or "database does not exist" errors.

Closing invalid; As with #15063, feel free to reopen if you can provide a comprehensive report that describes (or better yet, demonstrates) exactly how this would occur.

[David Cramer]

Natural keys require the data to exist in the database before they're referenced. With the way fixtures work, they're instantiated before running the "allow_syncdb" check. In our case, a database with the name "analytics" was loaded before "default", which contains very few tables. When it gets around to a fixture with a natural key, it attempts to instantiate it using the natural key code path, which attempts to validate a foreign key against a table that does not exist. I'm pretty certain this is valid, and while it may be an edge case (not sure how common natural key usage is), it's certainly very odd behavior.

I attempted to shift loading fixtures after syncdb was done (on every database), but that still doesnt solve it, as the fixtures load per database (not globally), and in this case, they still have the same issue. I think the proper fix would be to change natural keys so they're mapped in memory, but that requires quite a large change to Django itself I imagine, as last I knew relationships dont support associations if they dont have an identifier, which these wouldn't.

[Russell Keith-Magee]

Following discussion with David on IRC: This appears to be a case of #13252. His fixture had objects that didn't provide PKs, and he was assuming that this would cause "create if doesn't exist" type behavior.