Opened 3 years ago

Closed 2 years ago

Last modified 22 months ago

#23727 closed Bug (fixed)

IntegrityError with TransactionTestCase and serialized_rollback=True

Reported by: Tim Court Owned by: Tommy Beadle
Component: Testing framework Version: 1.7
Severity: Normal Keywords:
Cc: khoobks, dpoirier@…, Louis-Dominique Dubeau Triage Stage: Accepted
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

Using a TransactionTestCase with a sqlite database raises an IntegrityError in Django 1.7. This is the error I'm seeing:

Traceback (most recent call last):
  File "/tmp/djangomigrate/venv/local/lib/python2.7/site-packages/django/test/testcases.py", line 182, in __call__
    self._pre_setup()
  File "/tmp/djangomigrate/venv/local/lib/python2.7/site-packages/django/test/testcases.py", line 754, in _pre_setup
    self._fixture_setup()
  File "/tmp/djangomigrate/venv/local/lib/python2.7/site-packages/django/test/testcases.py", line 797, in _fixture_setup
    connections[db_name]._test_serialized_contents
  File "/tmp/djangomigrate/venv/local/lib/python2.7/site-packages/django/db/backends/creation.py", line 428, in deserialize_db_from_string
    obj.save()
  File "/tmp/djangomigrate/venv/local/lib/python2.7/site-packages/django/core/serializers/base.py", line 173, in save
    models.Model.save_base(self.object, using=using, raw=True)
  File "/tmp/djangomigrate/venv/local/lib/python2.7/site-packages/django/db/models/base.py", line 618, in save_base
    updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)
  File "/tmp/djangomigrate/venv/local/lib/python2.7/site-packages/django/db/models/base.py", line 699, in _save_table
    result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)
  File "/tmp/djangomigrate/venv/local/lib/python2.7/site-packages/django/db/models/base.py", line 732, in _do_insert
    using=using, raw=raw)
  File "/tmp/djangomigrate/venv/local/lib/python2.7/site-packages/django/db/models/manager.py", line 92, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/tmp/djangomigrate/venv/local/lib/python2.7/site-packages/django/db/models/query.py", line 921, in _insert
    return query.get_compiler(using=using).execute_sql(return_id)
  File "/tmp/djangomigrate/venv/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 920, in execute_sql
    cursor.execute(sql, params)
  File "/tmp/djangomigrate/venv/local/lib/python2.7/site-packages/django/db/backends/utils.py", line 65, in execute
    return self.cursor.execute(sql, params)
  File "/tmp/djangomigrate/venv/local/lib/python2.7/site-packages/django/db/utils.py", line 94, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/tmp/djangomigrate/venv/local/lib/python2.7/site-packages/django/db/backends/utils.py", line 65, in execute
    return self.cursor.execute(sql, params)
  File "/tmp/djangomigrate/venv/local/lib/python2.7/site-packages/django/db/backends/sqlite3/base.py", line 485, in execute
    return Database.Cursor.execute(self, query, params)
IntegrityError: columns app_label, model are not unique

I can reproduce this behaviour using the default project template, created using startproject and startapp, and nothing more than a single TransactionTestCase with two tests and the serialized_rollback option.

Here is a minimal sample project that demonstrates the problem: https://github.com/tctimmeh/djangomigrate. Running manage.py test mtestapp results in the error above.

It looks like the django.contrib.contenttypes app has a post-migrate script that populates the content type table. However, using the serialized_rollback option to refresh any migrated data also inserts saved duplicates into the content type table, which results in this exception.

Change History (13)

comment:1 Changed 3 years ago by Tim Graham

Triage Stage: UnreviewedAccepted

comment:2 Changed 3 years ago by khoobks

I realise that this is not a good workaround but as a temporary measure I found that the following works in my situation.

I subclassed the TransactionTestCase and added the following code.

def _fixture_setup(self):
    super(MyTransactionTestCase, self)._fixture_setup()
    for app_config in apps.get_app_configs():
        update_contenttypes(app_config)
        create_permissions(app_config)
        create_default_site(app_config)

def _fixture_teardown(self):
    signals.post_migrate.disconnect(create_default_site, sender=apps.get_app_config('sites'))
    signals.post_migrate.disconnect(update_contenttypes)
    signals.post_migrate.disconnect(create_permissions, dispatch_uid="django.contrib.auth.management.create_permissions")
    signals.post_migrate.disconnect(update_group_permissions)

    super(MyTransactionTestCase, self)._fixture_teardown()

    signals.post_migrate.connect(update_contenttypes)
    signals.post_migrate.connect(create_permissions, dispatch_uid="django.contrib.auth.management.create_permissions")
    signals.post_migrate.connect(create_default_site, sender=apps.get_app_config('sites'))

The the overridden _fixture_teardown disables the post_migrate signal handlers from executing and creating the problematic rows in the database. This allows the TransactionTestCase to successfully reload the original database state. The handlers are then manually invoked in the overriden _fixture_setup in order to ensure that for subsequent tests, the ContentTypes, Sites and Permissions are available.

Last edited 3 years ago by khoobks (previous) (diff)

comment:3 Changed 3 years ago by khoobks

Cc: khoobks added

comment:4 Changed 3 years ago by Dan Poirier

Cc: dpoirier@… added

comment:5 Changed 3 years ago by Dan Poirier

FYI, I see this with Postgres as well, so it doesn't appear to be SQLite specific.

comment:6 Changed 3 years ago by Tommy Beadle

Owner: changed from nobody to Tommy Beadle
Status: newassigned

comment:7 Changed 3 years ago by Tim Graham

Has patch: set

I had a look at the PR, but I'm not confident enough to commit it. Also, some documentation might be needed (the loss of the post_migrate signal in this scenario might be backwards incompatible for some users).

comment:8 Changed 3 years ago by Louis-Dominique Dubeau

Cc: Louis-Dominique Dubeau added

comment:9 Changed 3 years ago by Tim Graham

Needs tests: set

Added a comment with an idea about a test on the pull request.

comment:10 Changed 3 years ago by Tim Graham

Summary: IntegrityError with TransactionTestCase and sqliteIntegrityError with TransactionTestCase and serialized_rollback=True

comment:11 Changed 2 years ago by Tim Graham

Needs tests: unset

PR with test added.

comment:12 Changed 2 years ago by Tim Graham <timograham@…>

Resolution: fixed
Status: assignedclosed

In d3fdaf90:

Fixed #23727 -- Inhibited the post_migrate signal when using serialized_rollback.

When using a TransactionTestCase with serialized_rollback=True,
after creating the database and running its migrations (along with
emitting the post_migrate signal), the contents of the database
are serialized to _test_serialized_contents.

After the first test case, _fixture_teardown() would flush the
tables but then the post_migrate signal would be emitted and new
rows (with new PKs) would be created in the django_content_type
table. Then in any subsequent test cases in a suite,
_fixture_setup() attempts to deserialize the content of

_test_serialized_contents, but these rows are identical to the

rows already in the database except for their PKs. This causes an
IntegrityError due to the unique constraint in the
django_content_type table.

This change made it so that in the above scenario the post_migrate
signal is not emitted after flushing the tables, since it will be
repopulated during fixture_setup().

comment:13 Changed 22 months ago by Romain Garrigues

I have seen that this patch has been applied to Django 1.9.x.

For previous Django versions, I guess we can resolve this problem by setting:

TEST_NON_SERIALIZED_APPS = ['django.contrib.contenttypes']

as proposed in StackOverflow issue http://stackoverflow.com/questions/29226869/django-transactiontestcase-with-rollback-emulation/35359897 by @se7entyse7en. Is it right ?

Note: See TracTickets for help on using tickets.
Back to Top