Take advantage of SQLite support for FK constraints

[Ramiro Morales]

SQLite3 3.6.19 from Oct 14 2009 adds support for enforcing these constraints. See ​http://www.sqlite.org/foreignkeys.html

Creating this ticket to track this feature, with an initial implementation patch to get feedback. Will also open a django-dev thread.

[Ramiro Morales]

First implementation

[Ramiro Morales]

See also #5698 and #12210.

[Russell Keith-Magee]

Accepted on principle; however, we need to be clear on the backwards compatibility consequences of this change. In particular, I'm concerned about automatically enabling "PRAGMA foreign_keys" just because the feature is available.

[Ramiro Morales]

This is the django-dev mailing list thread: ​http://groups.google.com/group/django-developers/browse_thread/thread/fcda59f4ae21a6fd

[Aymeric Augustin]

Change UI/UX from NULL to False.

[Aymeric Augustin]

Change Easy pickings from NULL to False.

[Simon Charette]

#19659 was a duplicate.

[Ramiro Morales <cramm0@…>]

In 5782c94f2382303dd28a35a9b9591ad480286220:

Added generation of SQLite FK DDL in initial migrations.

SQLite accepts the relevant standard SQL (although by default it doesn't
enforce the constraint), and the 'traditional' creation backend helper
generate it, so this allows us to:

• Maintain the status quo
• Improve readability of the SQL code generated for that backend.

Also, we will need this for when we fix Refs #14204.

[Tim Graham]

5 years later (by the time Django 1.8 will be released), I think we could probably add this unconditionally and say that SQLite versions that don't support FKs aren't supported. Of course, we will still have to consider existing databases for which support isn't enabled.

[Anssi Kääriäinen]

How about we add a database OPTION key for SQLite to allow disabling foreign keys (mainly usable for backwards compatibility). If "disable foreign keys" flag isn't set, then starting in Django 1.8 automatically enable foreign key support.

[Aymeric Augustin]

Since it's called an ORM, I'm theoretically in favour of focusing on relational software. Realistically, the main problem is MyISAM. It was MySQL's default until 5.5. As long as we can't drop the code that works around databases not enforcing constraints, there's less to gain by requiring FKs on SQLite.

I'm +1 on enabling FK checks by default in Django and +0 on requiring it unconditionnally.

[William Schwartz]

+1 on getting this patch merged in. For what it's worth, the patch's implementation of _check_fk_constraint_support looks correct to me based on my reading of the ​SQLite documentation. My only question about the patch is whether the PRAGMA needs to be issued on every new connection as the patch does, or only once when the database is created.

[Tim Graham]

If you'd like to see it merged, feel free to update it to apply cleanly and add documentation (release notes). Then send a pull request and update the ticket flags so the patch appears in the review queue.

[Aymeric Augustin]

The pragma is per-connection. It's a runtime property of the SQLite engine. It doesn't affect the on-disk file format.

[Andriy Sokolovskiy]

I'm working on finishing of this patch

[Claude Paroz]

There is one major issue with supporting FK constraints on SQLite3: they can only be deactivated outside a transaction (this is a documented behavior).
Take for example the Django TestCase implementation which wraps every test class inside a transaction, that means that foreign keys checking can never be deactivated inside TestCase tests (loading forward references in fixtures is problematic for example, as does fixtures_regress.tests.TestFixtures.test_loaddata_works_when_fixture_has_forward_refs).

WIP work in this commit: ​https://github.com/claudep/django/commit/e7a9252aa6f990c4687656e9ea0f1d1ca70dc043

[Simon Charette]

Is there a reason SQLite foreign key constraints are not created DEFERRABLE INITIALLY DEFERRED like they are on PostgreSQL and Oracle?

If they were the issue mentioned by Claude regarding transaction wrapping could be solved.

[Claude Paroz]

I don't have the time to investigate right now, but if it can help someone, I rebased the commit which is now ​https://github.com/claudep/django/commit/15ceb2cc05262fe5f24f16cc31367fe6b5b698c1

[Tim Graham]

I've worked on updating Claude's patch by incorporating Simon's suggestion of adding DEFERRABLE INITIALLY DEFERRED: ​PR

There are still some issues to figure out.

[Tim Graham]

The PR linked in the previous comment is now ready for review.

[Tim Graham <timograham@…>]

In 169c3b3:

Fixed #14204 -- Enforced SQLite foreign key constraints.

Thanks Tim Graham for contributing to the patch and
Simon Charette for advice and review.

[Carlton Gibson <carlton.gibson@…>]

In 1939dd4:

Fixed #29928 -- Enabled deferred constraint checks on SQLite 3.20+.

Refs #11665, #14204.

Thanks Michel Samia for the report.

[Tim Graham <timograham@…>]

In ce8b65a:

Fixed #30054 -- Implemented cascaded flush on SQLite.

This is required to maintain foreign key integrity when using
TransactionTestCase.available_apps.

Refs #30033, #14204, #20483.

[Tim Graham <timograham@…>]

In 7534e434:

Refs #14204 -- Removed obsolete referential integrity comment for SQLite.