Wrong related fields in ._meta.get_fields with multiple database on foreignkey

[aRkadeFR]

Hello,

I tried to find a similar ticket and didn't find it.

I setup a simple use case to show the error.
2 applications: appa and appb.

You override (by inheritance) django.contrib.auth.models.Group in <appa>, you have a model ModelB in <appb> that has a foreign key to Group of django.contrib.auth.
You setup a DatabaseRouter to route every <appa> model to the 'default' database, and all <appb> model to the 'other' database.

If you ._meta.get_fields() on GroupA, you see the foreign key of the ModelB which is related to django.contrib.auth in the other database. The related field shouldn't be visible from the get_fields() on GroupA.

I got this error from debugging a "missing attribute" while Django was emulating the DELETE ON_CASCADE in python and following the wrong parent model to get then the related_fields.

Thanks for all the information and keep up the good work :)

# settings
DATABASE_ROUTERS = ['projecta.router.CustomRouter']
DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql_psycopg2',
        'HOST': 'localhost',
        'PORT': '5432',
        'NAME': 'default',
        'USER': 'postgres_user',
        'PASSWORD': 'pouetpouet',
    },
    'other': {
        'ENGINE': 'django.db.backends.postgresql_psycopg2',
        'HOST': 'localhost',
        'PORT': '5432',
        'NAME': 'other',
        'USER': 'postgres_user',
        'PASSWORD': 'pouetpouet',
    }
}


# projecta/router.py
class CustomRouter(object):
    """ CustomRouter """
    def db_for_read(self, model, **hints):
        if model._meta.app_label == 'appb':
            return 'other'
        if model._meta.app_label == 'auth':
            return 'other'
        return None

    def db_for_write(self, model, **hints):
        if model._meta.app_label == 'appb':
            return 'other'
        if model._meta.app_label == 'auth':
            return 'other'
        return None

    def allow_relation(self, obj1, obj2, **hints):
        if obj1._meta.app_label == 'auth' or \
           obj2._meta.app_label == 'auth':
            return True
        return None

    def allow_migrate(self, db, app_label, model=None, **hints):
        """ auth need to go to the other database (for projectb) """
        if app_label == 'appa':
            return db == 'default'
        if app_label == 'appb':
            return db == 'other'
        return None


# appa/models.py
from django.contrib.auth.models import Group
from django.db import models

class GroupA(Group):
    extra_field = models.IntegerField()

# appb/models.py
from django.contrib.auth.models import Group

class ModelB(models.Model):
    group = models.ForeignKey(Group, null=False)


# ipython
In [3]: GroupA._meta.get_fields()
Out[3]: 
(<ManyToManyRel: auth.user>,
 <ManyToOneRel: projectb.modelb>,
<...>

[Tim Graham]

Can you put together a sample project that reproduces the crash you mentioned in the description?

I don't think get_fields() is or should be affected by database routers.

[Simon Charette]

Hi aRkadeFR,

As you might be aware the ORM implements concrete model inheritance using ​multi-table inheritance with a unique foreign key (OneToOneField).

In your case a your GroupA model could also be explicitly defined as:

from django.contrib.auth.models import Group
from django.db import models

class GroupA(Group):
    group = models.OneToOneField(Group, parent_link=True)

Given your router definition it looks you're trying to define a cross-database relation, ​a feature documented as unsupported. I can see you even explicitly tried to work around this limitations by overriding allow_relation.

Without a full traceback it's hard to figure out from where the failure originates but I suppose a simple GroupA.object.all() should fail since it will be resolved to an SQL query JOINing the Group table.

Now, the _meta.get_field() API always returned inherited fields from parents so this doesn't look like a bug to me. If you want to make sure the field is defined on a specific model you should access its model attribute.

If you could provide a full traceback of the exception that was triggered during the emulated cascade deletion you might be able to get help from our support channels but given this is documented as unsupported I'll close the ticket as invalid.

[aRkadeFR]

We are not doing a cross database relation in our projects.

We have actually 2 distinct project, a "core" project, and a "side" project we will call them.
The auth_group table is in both the "core" database, and the "side" database (with extra fields in the "core" database).
The "side" project uses the "core" database as read-only models (through the router).
And to setup the "side" project, we install both project (as app) at the same time to makemigrations / migrate etc.
We have a missing attribute error or other error as provided in the following stacktrace.

I think our real problem is to have 2 different "app" / "project" at the same time with 2 different django.contrib.auth (in different database).
And we don't know how to set this up.

Can we have your thoughts? Thanks again for the answer

In [9]: GroupA.objects.all().delete()
---------------------------------------------------------------------------
ProgrammingError                          Traceback (most recent call last)
<ipython-input-9-1976efc2649b> in <module>()
----> 1 GroupA.objects.all().delete()

/usr/local/lib/python2.7/dist-packages/django/db/models/query.pyc in delete(self)
    598         collector = Collector(using=del_query.db)
    599         collector.collect(del_query)
--> 600         deleted, _rows_count = collector.delete()
    601 
    602         # Clear the result cache, in case this QuerySet gets reused.

/usr/local/lib/python2.7/dist-packages/django/db/models/deletion.pyc in delete(self)
    290             # fast deletes
    291             for qs in self.fast_deletes:
--> 292                 count = qs._raw_delete(using=self.using)
    293                 deleted_counter[qs.model._meta.label] += count
    294 

/usr/local/lib/python2.7/dist-packages/django/db/models/query.pyc in _raw_delete(self, using)
    612         query. No signals are sent, and there is no protection for cascades.
    613         """
--> 614         return sql.DeleteQuery(self.model).delete_qs(self, using)
    615     _raw_delete.alters_data = True
    616 

/usr/local/lib/python2.7/dist-packages/django/db/models/sql/subqueries.pyc in delete_qs(self, query, using)
     79             self.where = self.where_class()
     80             self.add_q(Q(pk__in=values))
---> 81         cursor = self.get_compiler(using).execute_sql(CURSOR)
     82         return cursor.rowcount if cursor else 0
     83 

/usr/local/lib/python2.7/dist-packages/django/db/models/sql/compiler.pyc in execute_sql(self, result_type)
    846         cursor = self.connection.cursor()
    847         try:
--> 848             cursor.execute(sql, params)
    849         except Exception:
    850             cursor.close()

/usr/local/lib/python2.7/dist-packages/django/db/backends/utils.pyc in execute(self, sql, params)
     77         start = time()
     78         try:
---> 79             return super(CursorDebugWrapper, self).execute(sql, params)
     80         finally:
     81             stop = time()

/usr/local/lib/python2.7/dist-packages/django/db/backends/utils.pyc in execute(self, sql, params)
     62                 return self.cursor.execute(sql)
     63             else:
---> 64                 return self.cursor.execute(sql, params)
     65 
     66     def executemany(self, sql, param_list):

/usr/local/lib/python2.7/dist-packages/django/db/utils.pyc in __exit__(self, exc_type, exc_value, traceback)
     93                 if dj_exc_type not in (DataError, IntegrityError):
     94                     self.wrapper.errors_occurred = True
---> 95                 six.reraise(dj_exc_type, dj_exc_value, traceback)
     96 
     97     def __call__(self, func):

/usr/local/lib/python2.7/dist-packages/django/db/backends/utils.pyc in execute(self, sql, params)
     62                 return self.cursor.execute(sql)
     63             else:
---> 64                 return self.cursor.execute(sql, params)
     65 
     66     def executemany(self, sql, param_list):

ProgrammingError: relation "appb_modelb" does not exist
LINE 1: DELETE FROM "appb_modelb" WHERE "appb_modelb"."group_id" IN ...

[Simon Charette]

Hi aRkadeFR,

From what I can see GroupA deletion triggered cascade deletion attempts of ModelB which is normal because of the FK relationships you defined (GroupA < -- o2o --> Group <-- fk -- ModelB).

Because of your router definition the app_modelb table is not created on the default database and thus defines a cross-database relationship (even if you auth_group table is in both databases) with GroupA in the context of the database used for its writes (router.db_for_write(GroupA) is None -> default).

I won't go into more details about you question because this ticket tracker is not a support channel but an application to track bugs. If you want to get help to set things up please submit your question to the mailing list or join the #django IRC channel.