Opened 4 years ago

Closed 4 years ago

#16088 closed Bug (needsinfo)

Multi-database tests fail in ContentType

Reported by: def@… Owned by: nobody
Component: Database layer (models, ORM) Version: 1.2
Severity: Normal Keywords: ContentType
Cc: Triage Stage: Unreviewed
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description (last modified by lrekucki)

I have a multi-database setup:

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.mysql',
        'NAME': 'wall2',
        'HOST': 'localhost',
        'PORT': '',
    },

    'slave': {
        'ENGINE': 'django.db.backends.mysql',
        'NAME': 'wall2',
        'HOST': 'localhost',
        'PORT': '',
        'TEST_MIRROR': 'default',
    },

    'station': {
        'ENGINE': 'django.db.backends.mysql',
        'NAME': 'station',
        'HOST': 'localhost',
        'PORT': '',
        'TEST_DEPENDENCIES': ['default',]
    },

    'catalog': {
        'ENGINE': 'django.db.backends.mysql',
        'NAME': 'catalog',
        'HOST': 'localhost',
        'PORT': '',
        'TEST_DEPENDENCIES': ['default', 'station']
    },
}

I have 2 routers, one handles the master/slave and one handles the 2 "data" database station and catalog. they run fine on production, but when we try to run our unit tests i get the following stack trace:

Traceback (most recent call last):
  File "./27.py", line 11, in <module>
    execute_manager(settings)
  File "/usr/lib/python2.7/site-packages/django/core/management/__init__.py", line 438, in execute_manager
    utility.execute()
  File "/usr/lib/python2.7/site-packages/django/core/management/__init__.py", line 379, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/lib/python2.7/site-packages/django/core/management/base.py", line 191, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "/usr/lib/python2.7/site-packages/django/core/management/base.py", line 220, in execute
    output = self.handle(*args, **options)
  File "/usr/lib/python2.7/site-packages/south/management/commands/test.py", line 8, in handle
    super(Command, self).handle(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/django/core/management/commands/test.py", line 37, in handle
    failures = test_runner.run_tests(test_labels)
  File "/usr/lib/python2.7/site-packages/django/test/simple.py", line 396, in run_tests
    old_config = self.setup_databases()
  File "/usr/lib/python2.7/site-packages/django/test/simple.py", line 334, in setup_databases
    test_db_name = connection.creation.create_test_db(self.verbosity, autoclobber=not self.interactive)
  File "/usr/lib/python2.7/site-packages/django/db/backends/creation.py", line 357, in create_test_db
    load_initial_data=False)
  File "/usr/lib/python2.7/site-packages/django/core/management/__init__.py", line 166, in call_command
    return klass.execute(*args, **defaults)
  File "/usr/lib/python2.7/site-packages/django/core/management/base.py", line 220, in execute
    output = self.handle(*args, **options)
  File "/usr/lib/python2.7/site-packages/django/core/management/base.py", line 351, in handle
    return self.handle_noargs(**options)
  File "/usr/lib/python2.7/site-packages/django/core/management/commands/syncdb.py", line 107, in handle_noargs
    emit_post_sync_signal(created_models, verbosity, interactive, db)
  File "/usr/lib/python2.7/site-packages/django/core/management/sql.py", line 182, in emit_post_sync_signal
    interactive=interactive, db=db)
  File "/usr/lib/python2.7/site-packages/django/dispatch/dispatcher.py", line 172, in send
    response = receiver(signal=self, sender=sender, **named)
  File "/usr/lib/python2.7/site-packages/django/contrib/contenttypes/management.py", line 25, in update_contenttypes
    ct.save()
  File "/usr/lib/python2.7/site-packages/django/db/models/base.py", line 458, in save
    self.save_base(using=using, force_insert=force_insert, force_update=force_update)
  File "/usr/lib/python2.7/site-packages/django/db/models/base.py", line 551, in save_base
    result = manager._insert(values, return_id=update_pk, using=using)
  File "/usr/lib/python2.7/site-packages/django/db/models/manager.py", line 195, in _insert
    return insert_query(self.model, values, **kwargs)
  File "/usr/lib/python2.7/site-packages/django/db/models/query.py", line 1524, in insert_query
    return query.get_compiler(using=using).execute_sql(return_id)
  File "/usr/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 788, in execute_sql
    cursor = super(SQLInsertCompiler, self).execute_sql(None)
  File "/usr/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 732, in execute_sql
    cursor.execute(sql, params)
  File "/usr/lib/python2.7/site-packages/django/db/backends/mysql/base.py", line 86, in execute
    return self.cursor.execute(query, args)
  File "/usr/lib64/python2.7/site-packages/MySQLdb/cursors.py", line 174, in execute
    self.errorhandler(self, exc, value)
  File "/usr/lib64/python2.7/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
    raise errorclass, errorvalue
django.db.utils.IntegrityError: (1062, "Duplicate entry 'tools-station' for key 'app_label'")

after some debugging i've found that the

    def update_contenttypes(app, created_models, verbosity=2, **kwargs):

method looks up the tools-station content type in the station database, doesn't find it, and then tries to write it to the default database, where it already exists.

by simply adding "using('default')" to the two ContentType.objects calls in the above method everything works fine. i'm not sure if all content types are meant to be in the 'default' database, but that seems to be the only way it can work at this point?

i'll attach a simple patch,

Attachments (3)

management.diff (392 bytes) - added by anonymous 4 years ago.
dbutil.py (1.6 KB) - added by def@… 4 years ago.
db router
db.py (699 bytes) - added by def@… 4 years ago.
last router in chain

Download all attachments as: .zip

Change History (12)

Changed 4 years ago by anonymous

comment:1 Changed 4 years ago by aaugustin

  • Needs documentation unset
  • Needs tests unset
  • Patch needs improvement unset
  • Resolution set to needsinfo
  • Status changed from new to closed

Either your database routers allow reading ContentType objects in the 'station' database (which they shouldn't), or they are bypassed somehow.

Could you post the code of your routers?

comment:2 Changed 4 years ago by aaugustin

And, obviously, re-open the ticket - thanks :)

Changed 4 years ago by def@…

db router

comment:3 Changed 4 years ago by anonymous

  • Resolution needsinfo deleted
  • Status changed from closed to reopened

Changed 4 years ago by def@…

last router in chain

comment:4 Changed 4 years ago by def@…

  • UI/UX unset

i have added a test for ContentType and the first router returns 'None' and the final router returns slave on calls to db_for_read for ContentType.

comment:5 Changed 4 years ago by bpeschier

Seems like this is not the only ticket about multi-db contenttype problems, could you test whether ticket #16281 solves your problem?

comment:6 Changed 4 years ago by lrekucki

  • Description modified (diff)

Fixed formatting (please use preview!).

comment:7 Changed 4 years ago by aaugustin

I tried and failed to reproduce this bug. We'll need the original reporter's feedback to determine if it's a duplicate of #16281.

While investigating, I stumbled upon another problem and I created #16353.

comment:8 Changed 4 years ago by anonymous

well, i'm testing with 1.2.5, so i had to edit the patch as follows, but it didnt' work for me:

        #return self.model_class()._default_manager.using(self._state.db).get(**kwargs)
        return self.model_class()._default_manager.using(self.model_class().objects.db).get(**kwargs)

but still got the same error:

django.db.utils.IntegrityError: (1062, "Duplicate entry 'tools-station' for key 'app_label'")

comment:9 Changed 4 years ago by aaugustin

  • Resolution set to needsinfo
  • Status changed from reopened to closed

If someone can provide a way to reproduce this on 1.3, please reopen the ticket.

Only critical fixes are backported to the 1.2 branch, and this problem is certainly not critical.

Note: See TracTickets for help on using tickets.
Back to Top