Admin model.ManyToManyField doesn't support legacy db tables and custom model.ForeignKey fields

[mwilson@…]

There appear to be issues using the models.ManyToManyFields with legacy database Xref tables if the referenced PK's are not named FIELD_ID in the Admin and/or the ORM layer. Below find the example model.py file to simulate the problem and comments in the code showing how to setup the test and reproduce the behaviour I'm seeing.

Pertinants:

• Using Postgres 8.3. postgresql_psycopg2 driver.
• Using Django trunk: rev 10787
• App is called "core" so db tables are called core_TABLE

Test Case Setup:

• models.py

  class Blog(models.Model):
      pk_blog = models.IntegerField(primary_key=True, db_column="pk_blog", default=random.randint(1,10000000))
      name = models.CharField(max_length=100)
      tagline = models.TextField()
  
      def __unicode__(self):
          return self.name
  
      class Admin:
          pass
  
  class Author(models.Model):
      pk_author = models.IntegerField(primary_key=True, db_column="pk_author", default=random.randint(1,10000000))
      name = models.CharField(max_length=50)
      email = models.EmailField()
  
      def __unicode__(self):
          return self.name
  
      class Admin:
          pass
  class Entry(models.Model):
      pk_entry = models.IntegerField(primary_key=True, db_column="pk_entry", default=random.randint(1,10000000))
      pk_blog = models.ForeignKey(Blog, db_column="pk_blog", to_field="pk_blog")
      headline = models.CharField(max_length=255)
      
          #### STEP 1:
          ### next line: leave this uncommented when you first generate the db tables
          ### via the manage.py syncdb command.  After that it should be commented out.
          ### Once syncdb is sucessfull we will be attempting to use the table that 
          ### is generated by the class EntryAuthor below. 
      authors = models.ManyToManyField(Author)
          ### NOTE: after doing a syncdb comment out this author definition and uncomment
          ### out the next one to start testing.  You may also want to drop the 
          ### core_entry_authors table that syncdb created as it isn't used.  We only
          ### use this definition just to get testing started and to have Django build
          ### tables easily for this test.   
      
          #### STEP 2: Try with db_table
          ### next line: if enabled once core_entry_author table is in place then
          ### you can't reference any custom fields on the db_table as the
          ### EntryAuthor model (see below) isn't referenced.  Additionally
          ### since the model isn't referenced Django assumes that the FK's in
          ### the XREF table (core_entry_author) are of the format field_id.
      #authors = models.ManyToManyField(Author, db_table='core_entry_author')
          ### SQL generated: SELECT "core_author"."pk_author", "core_author"."name", "core_author"."email" FROM "core_author" INNER JOIN "core_entry_author" ON ("core_author"."pk_author" = "core_entry_author"."author_id") WHERE "core_entry_author"."entry_id" = 6557280
          #### NOTE: the incorrect usage of "core_entry_author"."author_id"
          #### Seems if you use the db_table parameter of M2M default FK field names
          #### are assumed.  Won't work in my case since this is a legacy table with
          #### Field names like pk_author (not author_id).  This won't work in this 
          #### situation.
          #### EXCEPTION: ProgrammingError: column core_entry_author.author_id does not exist LINE 1: ...ore_entry_author" ON ("core_author"."pk_author" = "core_entr...
  
  
          #### STEP 3: Try "through" paremeter.
          ### next line: if enabled once core_entry_author table is in place then
          ### the Authors M2M combo-box isn't visible in the Admin form for Entry
          ### Trying to get Django to understand the custom pk field names.
      #authors = models.ManyToManyField(Author, through='EntryAuthor')
          ### NOTES: The admin form has no Authors field!  Seems to be silently
          ### dropped from the form for some reason.  Admin form will let you save
          ### the record with no errors but no M2M relationship record is written
          ### to the core_entry_author xref table.
          ### This one is the one I think would be nice to get to work.   
  
  
      def __unicode__(self):
          return self.headline
  
      class Admin:
          pass
  
  # NOTE: this class is only used by Django if M2M field def in Entry
  # is specified as (Author, through='EntryAuthor').  This definition creates the test
  # table core_entry_author.  You may want to drop the table created by the author M2M 
  # definition above (called core_entry_authors)
  class EntryAuthor(models.Model):
      pk_entry_author = models.IntegerField(primary_key=True, db_column="pk_entry_author", default=random.randint(1,10000000))
      pk_entry = models.ForeignKey(Entry, db_column="pk_entry")
      pk_author = models.ForeignKey(Author, db_column="pk_author")
      
      class Meta:
          db_table = 'core_entry_author'
          
      def __unicode__(self):
          return u'[%s]' % (self.pk_entry_author)
  
      class Admin:
          pass 
  
  

Discussion:

[Ramiro Morales]

As you suspected, if you test things (with you STEP 3 option) in the Python shell would show this isn't an ORM issue but an admin-related one instead, it's caused by a mis-configuration on you part. Reading the relevant documentation will reveal this section: ​http://docs.djangoproject.com/en/dev/ref/contrib/admin/#working-with-many-to-many-intermediary-models

(please try user support channels like the django-users mailing list or the #django IRC channel before opening a ticket when you aren't sure it is really a bug).

# models.py
from django.db import models

class Author(models.Model):
    pk_author = models.IntegerField(primary_key=True, db_column="pk_author")
    name = models.CharField(max_length=50)

    def __unicode__(self):
        return self.name

class Entry(models.Model):
    pk_entry = models.IntegerField(primary_key=True, db_column="pk_entry")
    headline = models.CharField(max_length=255)
    authors = models.ManyToManyField(Author, through='EntryAuthor')

    class Meta:
        verbose_name_plural = 'Entries'

    def __unicode__(self):
        return self.headline

class EntryAuthor(models.Model):
    pk_entry_author = models.IntegerField(primary_key=True, db_column="pk_entry_author")
    pk_entry = models.ForeignKey(Entry, db_column="pk_entry")
    pk_author = models.ForeignKey(Author, db_column="pk_author")

    class Meta:
        db_table = 'core_entry_author'

    def __unicode__(self):
        return u'[%s]' % (self.pk_entry_author)

# admin.py
from models import Author, Entry, EntryAuthor
from django.contrib import admin

class EntryAuthorInline(admin.TabularInline):
    model = EntryAuthor
    extra = 1

class AuthorAdmin(admin.ModelAdmin):
        inlines = (EntryAuthorInline,)

class EntryAdmin(admin.ModelAdmin):
        inlines = (EntryAuthorInline,)

admin.site.register(Author, AuthorAdmin)
admin.site.register(Entry, EntryAdmin)

[anonymous]

Although the above is not a true bug, it is something that's difficult to work with and non-intuitive. So consider this a vote for a new feature request. The root problem is not configuration of the admin interface, it is that if you use the "through" parameter it assumes you have additional data that needs to be added. This is rarely the case in a legacy database situation.

There are a couple possible solutions:

1. Have the ORM inspect the through model and find the correct foreign key fields to make normal manytomany functionality work (like add() or admin forms).
2. Allow two more optional parameters (or a tuple or something), to allow manually naming the foreign key fields.

This would allow a legacy m2m behave like a native one when there's no additional data (which is most of the time). I work in a corporate setting where getting django to connect to "legacy" data (any database not created by django) is indispensable, and this will only become more common when the multi-db branch lands in the trunk.

[anonymous]

​http://undefined.org.ua/blog/2008/02/29/manytomanywithcustomfield/?lang=en

Here's a great link illustrating one way that the manytomany field should work without having to override it.