Opened 15 years ago

Closed 6 years ago

#10827 closed Bug (fixed)

django.auth create_permissions must clear the content type cache before creating permissions

Reported by: Sean Legassick Owned by: Tim Graham <timograham@…>
Component: contrib.auth Version: dev
Severity: Normal Keywords:
Cc: jodym@…, chris+django@…, rjalves@…, jorgecarleitao, julenx@…, Christoph Heer, Jonas Trappenberg Triage Stage: Accepted
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

I hit a problem which took some time to track down, where at the DB flush stage in a sequence of tests (using TransactionTestCase) the recreation of permissions was failing with a FK constraint error.

This was caused by inserting a permission referring to a content type that didn't exist in the DB.

This happened because the content type was still in the cache, even thought the django_content_type table had been truncated.

The cache hadn't been cleared because post_syncdb signal dispatch had called create_permissions before calling update_contenttypes (which does clear the cache and recreate the content types correctly).

The real problem here is that create_permissions and update_contenttypes are both connected to the post_syncdb signal, with the former depending on the latter having been run first, but the dispatcher doesn't guarantee order of dispatch (or rather it dispatches in the order the signal handlers are connected, but that order depends on the order in which modules are loading which is not well-defined). Unfortunately that's a hard problem to solve, and I don't have any ideas about that short of substantive changes to the signal dispatcher.

An easier solution to this particular problem is for create_permissions to clear the content types cache before it recreates permissions, that way the necesarry content types will be created as needed (see attached patch).

Attachments (2)

10827-create-permissions-must-clear-content-type-cache.diff (561 bytes ) - added by Sean Legassick 15 years ago.
contenttype_error_on_TransactionTestCase.tar.bz2 (3.5 KB ) - added by Renato Alves 11 years ago.
Test Django project - created with Django 1.4.5

Download all attachments as: .zip

Change History (34)

comment:1 by Sean Legassick, 15 years ago

Has patch: set

comment:2 by Alex Gaynor, 15 years ago

Triage Stage: UnreviewedAccepted

comment:3 by Jason Yan <tailofthesun@…>, 14 years ago

I just got hit by this same issue and worked around it in a similar manner. I instead removed the signal that was connected to update_contenttypes and moved the update_contenttypes call to django.core.management.commands.flush.

For anyone that might be possibly searching for this, here's the exception from the tests without the fix:

IntegrityError: insert or update on table "auth_permission" violates foreign key constraint "content_type_id_refs_id_728de91f"
DETAIL:  Key (content_type_id)=(2) is not present in table "django_content_type".

comment:4 by PeterRussell, 13 years ago

It seems that the reason that this happens is that the syncdb command is firing the post_syncdb signal for each application, as returned from get_apps() [source:/django/trunk/django/core/management/sql.py@14849#L178 here]. Get_apps appears to return apps in app-name order (see [source:/django/trunk/django/db/models/loading.py@14849#L120 here]), and auth comes before contenttypes asciibetically. It seems the right solution is probably to have a separate post_contenttypes_refreshed signal, and have create_permissions listen for that. It would also be great if updating permissions could fire its own signal, so you can update groups etc based on that. (We've been having a somewhat similar issue).

comment:5 by Chris Beaven, 13 years ago

Severity: Normal
Type: Bug

comment:6 by Julien Phalip, 13 years ago

Needs tests: set

comment:7 by orb@…, 13 years ago

Easy pickings: unset
UI/UX: unset

I'm having this problem too. We worked around it by extending TransactionTestCase so that _pre_setup() invokes ContentType.objects.clear_cache(). I'd really love to see a solution to this.

comment:8 by jodym@…, 12 years ago

Cc: jodym@… added

comment:9 by anonymous, 12 years ago

A workaround based on comment 7 is available here: http://djangosnippets.org/snippets/2752/

comment:10 by Chris Wilson, 12 years ago

+1. This just bit us as well.

I think if flush is going to invalidate caches (which it appears to do), then it should be sending a signal that allows holders of caches to respond appropriately. So something like this in flush.py:

for app in set(all_models):
    models.signals.post_flush.send(sender=app, app=app,
        created_models=created_models, verbosity=verbosity,
        interactive=interactive, db=db)
emit_post_sync_signal(set(all_models), verbosity, interactive, db)

Assuming a new signal flush which ContentTypeManager listens for, and then flushes its cache.

comment:11 by Chris Wilson, 12 years ago

Cc: chris+django@… added

by Renato Alves, 11 years ago

Test Django project - created with Django 1.4.5

comment:12 by Renato Alves, 11 years ago

Cc: rjalves@… added

The attached file is a fresh django project created with Django 1.4.5 which reproduces the mentioned error.
NOTE: This error is not reproducible with the sqlite backend (MySQL was not tested but was reported to fail with InnoDB in other tickets #9207).

Ticket #9207 is closed as fixed but due to lack of reproducibility. From the error it looks like a duplicate of the current.
The workaround described on comment 3 from #9207 also solves the problem (putting contrib.contenttypes before contrib.auth in INSTALLED_APPS).

PS: I don't know if there's any reason why INSTALLED_APPS has the current order but having django-admin.py invert the order could workaround this problem.

Last edited 11 years ago by Renato Alves (previous) (diff)

comment:13 by Renato Alves, 11 years ago

Another duplicate: #6259

comment:14 by anonymous, 11 years ago

I think I have run into the same problem while trying to use Selenium to test Django admin capabilities. When I try to update a user's permissions, I get the FK error. You can see [my code and issue on SO](http://stackoverflow.com/questions/18281137/selenium-django-gives-foreign-key-error). So I think this is still a present bug in Django 1.5.1 / MySQL.

comment:15 by jorgecarleitao, 10 years ago

Cc: jorgecarleitao added

comment:16 by julen, 10 years ago

Cc: julenx@… added

comment:17 by Christoph Heer, 10 years ago

Cc: Christoph Heer added

comment:18 by Thomas Güttler, 9 years ago

I think this bug is still alive in Django1.7.

I get this traceback:

File ".../django/core/management/__init__.py", line 385, in execute_from_command_line
  utility.execute()
File ".../django/core/management/__init__.py", line 377, in execute
  self.fetch_command(subcommand).run_from_argv(self.argv)
File ".../django/core/management/base.py", line 288, in run_from_argv
  self.execute(*args, **options.__dict__)
File ".../django/core/management/base.py", line 338, in execute
  output = self.handle(*args, **options)
File ".../django/core/management/commands/migrate.py", line 165, in handle
  emit_post_migrate_signal(created_models, self.verbosity, self.interactive, connection.alias)
File ".../django/core/management/sql.py", line 268, in emit_post_migrate_signal
  using=db)
File ".../django/dispatch/dispatcher.py", line 198, in send
  response = receiver(signal=self, sender=sender, **named)
File ".../django/contrib/auth/management/__init__.py", line 117, in create_permissions
  Permission.objects.using(using).bulk_create(perms)
File ".../django/db/models/query.py", line 410, in bulk_create
  self._batched_insert(objs_without_pk, fields, batch_size)
File ".../django/db/transaction.py", line 339, in __exit__
  connection.commit()
File ".../django/db/backends/__init__.py", line 176, in commit
  self._commit()
File ".../django/db/backends/__init__.py", line 145, in _commit
  return self.connection.commit()
File ".../django/db/utils.py", line 94, in __exit__
  six.reraise(dj_exc_type, dj_exc_value, traceback)
File ".../django/db/backends/__init__.py", line 145, in _commit
  return self.connection.commit()

django.db.utils.IntegrityError: insert or update on table "auth_permission" violates foreign key constraint "auth_content_type_id_508cf46651277a81_fk_django_content_type_id"
 DETAIL:  Key (content_type_id)=(1) is not present in table "django_content_type".

 

comment:19 by Thomas Güttler, 9 years ago

The six years old patch solved the issue for me:

ContentType.objects.clear_cache()

comment:20 by Thomas Güttler, 9 years ago

A work around is this line in your migration:

from django.contrib.contenttypes.models import ContentType

class Migration(migrations.Migration):
    operations = [
        ...,
        migrations.RunPython(lambda apps, schema_editor: ContentType.objects.clear_cache()),
    ]

comment:21 by Thomas Güttler, 9 years ago

I created a pull request: https://github.com/django/django/pull/5184

It includes a test. Thanks to Tim Graham for the support.

https://groups.google.com/d/msg/django-developers/CB4cGwBYo8o/7qcekYNeAgAJ

comment:22 by Peter Bengtsson, 8 years ago

I can confirm that this is still a bug in Django 1.7.
I can't speak of the PR because I haven't studied it but making sure the order of the INSTALLED_APPS is right, as per http://stackoverflow.com/a/18292090/205832 did solve it for me.

comment:23 by Koen Vossen, 8 years ago

Owner: changed from nobody to Koen Vossen
Status: newassigned

comment:24 by Koen Vossen, 8 years ago

Owner: Koen Vossen removed
Status: assignednew

Don't forget to use a database that checks foreign key constraints instead of the default sqlite when trying to reproduce this bug.

Version 0, edited 8 years ago by Koen Vossen (next)

comment:25 by Jonas Trappenberg, 8 years ago

Cc: Jonas Trappenberg added

comment:26 by Sven R. Kunze, 8 years ago

@guettli @timgraham

It just works. Nice. Thanks for that suggestion.

comment:27 by Lynn Cyrin, 7 years ago

issue is still present in django 1.11.2

comment:28 by Michał Pasternak, 6 years ago

I am having this problem with Django 1.11.5 and purest with pytest-django

________________________________________________________________ ERROR at teardown of test_foo ________________________________________________________________

self = <django.db.backends.postgresql.base.DatabaseWrapper object at 0x10ee8e630>

    def _commit(self):
        if self.connection is not None:
            with self.wrap_database_errors:
>               return self.connection.commit()
E               psycopg2.IntegrityError: B����D:  wstawianie lub modyfikacja na tabeli "auth_permission" narusza klucz obcy "auth_permission_content_type_id_2f476e4b_fk_django_co"
E               DETAIL:  Klucz (content_type_id)=(893) nie wyst��puje w tabeli "django_content_type".

../../envs/django-bpp/lib/python3.6/site-packages/django/db/backends/base/base.py:236: IntegrityError

The above exception was the direct cause of the following exception:

self = <django.test.testcases.TransactionTestCase testMethod=__init__>

    def _post_teardown(self):
        """Performs any post-test things. This includes:

            * Flushing the contents of the database, to leave a clean slate. If
              the class has an 'available_apps' attribute, post_migrate isn't fired.
            * Force-closing the connection, so the next test gets a clean cursor.
            """
        try:
>           self._fixture_teardown()

../../envs/django-bpp/lib/python3.6/site-packages/django/test/testcases.py:925:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../../envs/django-bpp/lib/python3.6/site-packages/django/test/testcases.py:960: in _fixture_teardown
    inhibit_post_migrate=inhibit_post_migrate)
../../envs/django-bpp/lib/python3.6/site-packages/django/core/management/__init__.py:131: in call_command
    return command.execute(*args, **defaults)
../../envs/django-bpp/lib/python3.6/site-packages/django/core/management/base.py:330: in execute
    output = self.handle(*args, **options)
../../envs/django-bpp/lib/python3.6/site-packages/django/core/management/commands/flush.py:88: in handle
    emit_post_migrate_signal(verbosity, interactive, database)
../../envs/django-bpp/lib/python3.6/site-packages/django/core/management/sql.py:53: in emit_post_migrate_signal
    **kwargs
../../envs/django-bpp/lib/python3.6/site-packages/django/dispatch/dispatcher.py:193: in send
    for receiver in self._live_receivers(sender)
../../envs/django-bpp/lib/python3.6/site-packages/django/dispatch/dispatcher.py:193: in <listcomp>
    for receiver in self._live_receivers(sender)
../../envs/django-bpp/lib/python3.6/site-packages/django/contrib/auth/management/__init__.py:83: in create_permissions
    Permission.objects.using(using).bulk_create(perms)
../../envs/django-bpp/lib/python3.6/site-packages/django/db/models/query.py:449: in bulk_create
    obj_without_pk._state.db = self.db
../../envs/django-bpp/lib/python3.6/site-packages/django/db/transaction.py:223: in __exit__
    connection.commit()
../../envs/django-bpp/lib/python3.6/site-packages/django/db/backends/base/base.py:262: in commit
    self._commit()
../../envs/django-bpp/lib/python3.6/site-packages/django/db/backends/base/base.py:236: in _commit
    return self.connection.commit()
../../envs/django-bpp/lib/python3.6/site-packages/django/db/utils.py:94: in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
../../envs/django-bpp/lib/python3.6/site-packages/django/utils/six.py:685: in reraise
    raise value.with_traceback(tb)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <django.db.backends.postgresql.base.DatabaseWrapper object at 0x10ee8e630>

    def _commit(self):
        if self.connection is not None:
            with self.wrap_database_errors:
>               return self.connection.commit()
E               django.db.utils.IntegrityError: B����D:  wstawianie lub modyfikacja na tabeli "auth_permission" narusza klucz obcy "auth_permission_content_type_id_2f476e4b_fk_django_co"
E               DETAIL:  Klucz (content_type_id)=(893) nie wyst��puje w tabeli "django_content_type".

../../envs/django-bpp/lib/python3.6/site-packages/django/db/backends/base/base.py:236: IntegrityError

===================================================================== 29 tests deselected =====================================================================
====================================================== 1 passed, 29 deselected, 1 error in 3.18 seconds =======================================================

My package versions

pytest-base-url (1.4.1)
pytest-cov (2.5.1)
pytest-django (3.1.2)
pytest-html (1.16.0)
pytest-instafail (0.3.0)
pytest-localserver (0.3.6)
pytest-metadata (1.5.0)
pytest-splinter (1.8.5)
pytest-variables (1.7.0)

Django (1.11.5)

comment:29 by Michał Pasternak, 6 years ago

Moving contenttypes apps before auth in INSTALLED_APPS fixed it.

comment:30 by Jared Proffitt, 6 years ago

Michał's fix works for me as well. If I move 'django.contrib.contenttypes' before 'django.contrib.auth' in INSTALLED_APPS, it works.

Would be nice to either have a fix that doesn't require a specific order of apps, or some documentation mentioning you need the specific app order.

Last edited 6 years ago by Jared Proffitt (previous) (diff)

comment:31 by Claude Paroz, 6 years ago

Needs tests: unset

comment:32 by Tim Graham <timograham@…>, 6 years ago

Owner: set to Tim Graham <timograham@…>
Resolution: fixed
Status: newclosed

In bec651a:

Fixed #10827 -- Ensured ContentTypes are created before permission creation.

Note: See TracTickets for help on using tickets.
Back to Top