create_permissions uses ContentType cache which contains non-existing pks

[Sven R. Kunze]

While migrating a larger project, we found the traceback attached.

We found that create_permissions from django/contrib/auth/management/__init__.py uses the ContentType.objects._cache instead of querying the database for the real pk of the contenttype.

Real pks from database:

 id  |       app_label       |                   model                    
-----+-----------------------+--------------------------------------------
   1 | auth                  | permission
   2 | auth                  | group
   3 | auth                  | user

Pks generated by django (execution at breakpoint at line 103):

>>> [(id, ct) for id, ct in ContentType.objects._cache['default'].items() if ct.app_label == 'auth']
[
    (2, <ContentType: group>),
    (3, <ContentType: permission>),
    (4, <ContentType: user>),
]

It seems Django confused real pks with imaginary pks. We created a placeholder contenttype with pk=4 before rerunning the migrations, which fixed the exception but created a new permission linked to this "false" contenttype.

If the contenttype with pk=4 were a real one (i.e. an existing model etc.), a wrong permission would have been created then.

=> select * from django_content_type where id=4;
 id | app_label |     model     
----+-----------+---------------
  4 | auth      | XXXXXXXXXXXXX

=> select * from auth_permission where content_type_id=4;
 id  |      name       | content_type_id |  codename   
-----+-----------------+-----------------+-------------
 598 | Can delete user |               4 | delete_user
 597 | Can change user |               4 | change_user
 596 | Can add user    |               4 | add_user

The migration traceback:

Traceback (most recent call last):
  File "lib/python2.7/site-packages/django/core/management/__init__.py", line 354, in execute_from_command_line
    utility.execute()
  File "lib/python2.7/site-packages/django/core/management/__init__.py", line 346, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "lib/python2.7/site-packages/django/core/management/base.py", line 394, in run_from_argv
    self.execute(*args, **cmd_options)
  File "lib/python2.7/site-packages/django/core/management/base.py", line 445, in execute
    output = self.handle(*args, **options)
  File "lib/python2.7/site-packages/django/core/management/commands/migrate.py", line 226, in handle
    emit_post_migrate_signal(created_models, self.verbosity, self.interactive, connection.alias)
  File "lib/python2.7/site-packages/django/core/management/sql.py", line 280, in emit_post_migrate_signal
    using=db)
  File "lib/python2.7/site-packages/django/dispatch/dispatcher.py", line 189, in send
    response = receiver(signal=self, sender=sender, **named)
  File "lib/python2.7/site-packages/django/contrib/auth/management/__init__.py", line 114, in create_permissions 
    Permission.objects.using(using).bulk_create(perms)
  File "lib/python2.7/site-packages/django/db/models/query.py", line 392, in bulk_create
    self._batched_insert(objs_without_pk, fields, batch_size)
  File "lib/python2.7/site-packages/django/db/transaction.py", line 225, in __exit__
    connection.commit() 
  File "lib/python2.7/site-packages/django/db/backends/base/base.py", line 173, in commit
    self._commit() 
  File "lib/python2.7/site-packages/django/db/backends/base/base.py", line 142, in _commit
    return self.connection.commit()
  File "lib/python2.7/site-packages/django/db/utils.py", line 98, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "lib/python2.7/site-packages/django/db/backends/base/base.py", line 142, in _commit
    return self.connection.commit()
django.db.utils.IntegrityError: insert or update on table "auth_permission" violates foreign key constraint "auth_content_type_id_508cf46651277a81_fk_django_content_type_id"
DETAIL:  Key (content_type_id)=(4) is not present in table "django_content_type".

[Sven R. Kunze]

A quick fix we found could be (line 90 to line 100):

all_perms=set(Permission.objects.using(using).filter(
    content_type__in=ctypes,
).values_list(
    "content_type__app_label", "content_type__model", "codename"
))


perms =[
    Permission(codename=codename, name=name, content_type=ContentType.objects.get(app_label=ct.app_label, model=ct.model))
    for ct, (codename, name) in searched_perms
    if (ct.app_label, ct.model, codename) not in all_perms
]

[Tim Graham]

Might be a duplicate of #10827. Could you test the fix proposed on that ticket? We're still waiting for a test for that patch if you're able to provide one.

[Sven R. Kunze]

It seems it is.