Opened 3 months ago

Last modified 3 months ago

#36429 assigned Bug

IntegrityError on contentype creation when using transactional test with AND without serialized_rollback

Reported by: Julie Rymer Owned by: Clifford Gama
Component: Testing framework Version: 5.2
Severity: Normal Keywords:
Cc: Triage Stage: Accepted
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

Related to #23727

When using TransactionTestCase, if you use rollback emulation with serialized_rollback=True for some test but not all, you'll get a db error because of a duplicate on ContentType.

django.db.utils.IntegrityError: UNIQUE constraint failed: django_content_type.app_label, django_content_type.model

Traceback (most recent call last):
  File "[...]/site-packages/django/db/backends/base/creation.py", line 163, in deserialize_db_from_string
    obj.save()

Problem analysis

When using serialized_rollback=True, on test setup the content of the database that was previously serialised will be applied.
On teardown, the data will be flushed and the post_migrate signal won't be emitted because of inhibit_post_migrate argument.

When using serialized_rollback=False no database content is loaded on setup and on teardown the database is flushed and post_migrate signal will be emitted.

ContentType objects get created from a post_migrate handler. They also get serialised from database content.

So, what happens if a test with serialized_rollback=True get run after a test with serialized_rollback=False? You get an IntegrityError because the ContentType object are loaded from connection._test_serialized_contents after they already were created from the post_migrate handler emitted when flushing.

I think this is what is happening, please correct me if my assumptions are wrong.

How to reproduce

Using the "Writing your first Django app" tutorial, you can reproduce by adding the following tests to the polls app:

from django.test import TransactionTestCase

class QuestionModel1Tests(TransactionTestCase):
    serialized_rollback = False
    def test_was_published_recently_with_future_question(self):
        self.assertTrue(True)


class QuestionModel2Tests(TransactionTestCase):
    serialized_rollback = True
    def test_was_published_recently_with_future_question(self):
        self.assertTrue(True)

This error fully depends on the order of the test execution. If you don't get the error the first time, just add some more tests alternating serialized_rollback True and False until you get the error. you'll get it as soon as a serialised rollback test execute after a non-serialised one.

Change History (4)

comment:1 by Sarah Boyce, 3 months ago

Triage Stage: UnreviewedAccepted

Thank you, replicated. Also see this behavior on 5.1, 5.0, 4.2

comment:2 by Julie Rymer, 3 months ago

Also for those affected, I found a workaround I'm using with pytest.
You can use the pytest_collection_modifyitems hook to change the order of the tests and have all transactional tests with serialized_rollback=False all run at the end. This way the post_migrate won't affect any test with serialized_rollback=True.

def pytest_collection_modifyitems(config, items):
    def transactional_attr_order(item):
        marker = item.get_closest_marker("django_db")
        if (
            marker
            and marker.kwargs.get("transaction", False)
            and not marker.kwargs.get("serialized_rollback", False)
        ):
            return 1
        return 0

    items.sort(key=transactional_attr_order)

comment:3 by Simon Charette, 3 months ago

A similar re-ordering strategy could be employed in this case. Something like

  • django/test/runner.py

    index c8bb16e7b3..85b9c26427 100644
    a b  
    2121import django
    2222from django.core.management import call_command
    2323from django.db import connections
    24 from django.test import SimpleTestCase, TestCase
     24from django.test import SimpleTestCase, TestCase, TransactionTestCase
    2525from django.test.utils import NullTimeKeeper, TimeKeeper, iter_test_cases
    2626from django.test.utils import setup_databases as _setup_databases
    2727from django.test.utils import setup_test_environment
    def shuffle(self, items, key):  
    656656        return [hashes[hashed] for hashed in sorted(hashes)]
    657657
    658658
     659class _SerializedRollbackTransactionTestCaseType(type):
     660    def __instancecheck__(self, instance):
     661        return (
     662            isinstance(instance, TransactionTestCase) and instance.serialized_rollback
     663        )
     664
     665
     666class _SerializedRollbackTransactionTestCase(
     667    metaclass=_SerializedRollbackTransactionTestCaseType
     668):
     669    pass
     670
     671
    659672class DiscoverRunner:
    660673    """A Django test runner that uses unittest2 test discovery."""
    661674
    class DiscoverRunner:  
    663676    parallel_test_suite = ParallelTestSuite
    664677    test_runner = unittest.TextTestRunner
    665678    test_loader = unittest.defaultTestLoader
    666     reorder_by = (TestCase, SimpleTestCase)
     679    reorder_by = (TestCase, _SerializedRollbackTransactionTestCase, SimpleTestCase)
    667680
    668681    def __init__(
    669682        self,
Last edited 3 months ago by Simon Charette (previous) (diff)

comment:4 by Clifford Gama, 3 months ago

Owner: set to Clifford Gama
Status: newassigned
Note: See TracTickets for help on using tickets.
Back to Top