KeyError: u"Problem installing fixture '<filename>.json': u'manager'"

[Brian May]

Since upgrading to Django 1.8, my tests are failing to load the test data, and I don't see anything about this in the release notes:

======================================================================
ERROR: setUpClass (karaage.tests.test_people.PersonTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/python/local/lib/python2.7/site-packages/django/test/testcases.py", line 952, in setUpClass
    'database': db_name,
  File "/tmp/python/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 120, in call_command
    return command.execute(*args, **defaults)
  File "/tmp/python/local/lib/python2.7/site-packages/django/core/management/base.py", line 441, in execute
    output = self.handle(*args, **options)
  File "/tmp/python/local/lib/python2.7/site-packages/django/core/management/commands/loaddata.py", line 60, in handle
    self.loaddata(fixture_labels)
  File "/tmp/python/local/lib/python2.7/site-packages/django/core/management/commands/loaddata.py", line 90, in loaddata
    self.load_label(fixture_label)
  File "/tmp/python/local/lib/python2.7/site-packages/django/core/management/commands/loaddata.py", line 147, in load_label
    obj.save(using=self.using)
  File "/tmp/python/local/lib/python2.7/site-packages/django/core/serializers/base.py", line 173, in save
    models.Model.save_base(self.object, using=using, raw=True)
  File "/tmp/python/local/lib/python2.7/site-packages/django/db/models/base.py", line 747, in save_base
    update_fields=update_fields, raw=raw, using=using)
  File "/tmp/python/local/lib/python2.7/site-packages/django/dispatch/dispatcher.py", line 201, in send
    response = receiver(signal=self, sender=sender, **named)
  File "/home/brian/tree/django/karaage/karaage/karaage/people/models.py", line 508, in _project_membership_changed
    projectmembership__project=project,
  File "/home/brian/tree/django/karaage/karaage/karaage/people/models.py", line 644, in sync_members
    *[tmp_person_id for tmp_person_id in add_member_ids])
  File "/tmp/python/local/lib/python2.7/site-packages/django/db/models/fields/related.py", line 973, in add
    self._add_items(self.source_field_name, self.target_field_name, *objs)
  File "/tmp/python/local/lib/python2.7/site-packages/django/db/models/fields/related.py", line 1105, in _add_items
    model=self.model, pk_set=new_ids, using=db)
  File "/tmp/python/local/lib/python2.7/site-packages/django/dispatch/dispatcher.py", line 201, in send
    response = receiver(signal=self, sender=sender, **named)
  File "/home/brian/tree/django/karaage/karaage/karaage/people/models.py", line 708, in _members_changed
    _add_person_to_group(person, group)
  File "/home/brian/tree/django/karaage/karaage/karaage/people/models.py", line 673, in _add_person_to_group
    add_accounts_to_project(a_list, group.project)
  File "/home/brian/tree/django/karaage/karaage/karaage/datastores/__init__.py", line 472, in add_accounts_to_project
    query = query.filter(machine_category__projectquota__project=project)
  File "/tmp/python/local/lib/python2.7/site-packages/django/db/models/query.py", line 679, in filter
    return self._filter_or_exclude(False, *args, **kwargs)
  File "/tmp/python/local/lib/python2.7/site-packages/django/db/models/query.py", line 697, in _filter_or_exclude
    clone.query.add_q(Q(*args, **kwargs))
  File "/tmp/python/local/lib/python2.7/site-packages/django/db/models/sql/query.py", line 1301, in add_q
    clause, require_inner = self._add_q(where_part, self.used_aliases)
  File "/tmp/python/local/lib/python2.7/site-packages/django/db/models/sql/query.py", line 1328, in _add_q
    current_negated=current_negated, connector=connector, allow_joins=allow_joins)
  File "/tmp/python/local/lib/python2.7/site-packages/django/db/models/sql/query.py", line 1150, in build_filter
    value, lookups, used_joins = self.prepare_lookup_value(value, lookups, can_reuse, allow_joins)
  File "/tmp/python/local/lib/python2.7/site-packages/django/db/models/sql/query.py", line 997, in prepare_lookup_value
    value = value()
  File "/tmp/python/local/lib/python2.7/site-packages/django/db/models/fields/related.py", line 691, in __call__
    manager = getattr(self.model, kwargs.pop('manager'))
KeyError: u"Problem installing fixture '/home/brian/tree/django/karaage/karaage/karaage/fixtures/test_karaage.json': u'manager'"

----------------------------------------------------------------------

[Brian May]

I think the message is somewhat confusing; this isn't a problem with test_karaage.json, rather it an exception generated in response to a signal.

Here is the relevant function call that generates the exception:

File "/home/brian/tree/django/karaage/karaage/karaage/datastores/__init__.py", line 472, in add_accounts_to_project
    query = query.filter(machine_category__projectquota__project=project)

[Brian May]

Ok, this is getting a tad confusing. For some reason the following code

group.project

returns a RelatedManager instead of a Project (did I miss something in the release notes about this?). As in, the following code:

print("======", type(group), type(group.project))

displays:

('======', <class 'karaage.people.models.Group'>, <class 'django.db.models.fields.related.RelatedManager'>)

Group.project is a reverse relationship from a Project:

from mptt.models import MPTTModel
class Project(MPTTModel):
    ...
    group = models.OneToOneField(Group)
    ...

So I believe group.project is suppose to return a Project, not a RelatedManager. I then pass the "project" to the Django function:

query = query.filter(machine_category__projectquota__project=project)

Which gets confused because it has the wrong type.

[Tim Graham]

It's a bit difficult to say what the issue is here without a way to reproduce it. Either a minimal project that we can use to reproduce the issue or a better diagnosis of what's going wrong would be helpful.

[Brian May]

Will try to simplify things. I have a suspicion however, unless OneToOneField is broken for everyone (which I doubt), that simplifying the code will not trigger the problem anymore.

Can I confirm that group.project, the reverse relation for a OneToOneField from the project, should return a group and not a RelatedManager?

It would appear that OneToOneField is generating an incorrect reverse mapping in the related object.

[Brian May]

Also, in case I wasn't clear, this code works perfectly with Django 1.7; so this does look like a regression in Django 1.8.

[Tim Graham]

Yes, I'd expect the reverse relation attribute to be a model instance.

[Brian May]

Just did a git bisect, and found that the following commit f233bf47dde1d481108142c8d6b4bb3b3d8c6d08 in Django appears to introduce this problem.

commit f233bf47dde1d481108142c8d6b4bb3b3d8c6d08
Author: Anssi Kääriäinen <akaariai@gmail.com>
Date:   Sat Nov 9 14:25:15 2013 +0200

    Fixed #21414 -- Removed RelatedObject and deprecated Field.related.

Now to investigate why...

[Brian May]

I note that before commit f233bf47dde1d481108142c8d6b4bb3b3d8c6d08, SingleRelatedObjectDescriptor is used, after that commit, ForeignRelatedObjectsDescriptor is used.

At quick glance, this seems wrong for a OneToOneField, however out of time to investigate just now. Will check this out later.

[Brian May]

Ok, so I think the reason for the wrong type is because there is a clash between two reverse accessors. Somehow django-audit-log is involved.

However, there was no clash in Django 1.7, and I don't understand why there should be any clash in Django 1.8 either.

I have test code that will illustrate the problem, see ​https://github.com/brianmay/django-24768

% python ./manage.py makemigrations polls --dry
SystemCheckError: System check identified some issues:

ERRORS:
polls.Project.group: (fields.E304) Reverse accessor for 'Project.group' clashes with reverse accessor for 'Project.group'.
	HINT: Add or change a related_name argument to the definition for 'Project.group' or 'Project.group'.
polls.Project.group: (fields.E305) Reverse query name for 'Project.group' clashes with reverse query name for 'Project.group'.
	HINT: Add or change a related_name argument to the definition for 'Project.group' or 'Project.group'.
polls.ProjectAuditLogEntry.group: (fields.E304) Reverse accessor for 'ProjectAuditLogEntry.group' clashes with reverse accessor for 'Project.group'.
	HINT: Add or change a related_name argument to the definition for 'ProjectAuditLogEntry.group' or 'Project.group'.

I cannot reproduce my initial problem with this test program (see test.py in that repository), however I strongly suspect that this was just a symptom of the above problem.

The first two messages don't seem to make any sense. Project.group clashes with itself???

The last one is probably more significant: Reverse accessor for 'ProjectAuditLogEntry.group' clashes with reverse accessor for 'Project.group'.

Shouldn't the first be called projectauditlogentry_set and the second be called project? Maybe Django 1.8 is getting the related_name wrong in this particular situation?

[Brian May]

Also see ​https://github.com/Atomidata/django-audit-log/issues/21

I don't know if this is a django bug or django-audit-log bug, I don't really understand how this automatic table generation works.

[Brian May]

The following code is in django-audit-log - can somebody tell me if this is valid Django 1.8 code? I suspect maybe not...

if field.rel and field.rel.related_name:
    field.rel.related_name = '_auditlog_%s' % field.rel.related_name

If this code is not valid any more, what should it be?

I see that field.related was depreciated in favor on field.rel, however I can't find anything about the field.rel API having changed.

[Tim Graham]

That code looks okay to me. It would be helpful if you could bisect Django's commit history to identify the commit that caused the breakage.

[Brian May]

The git commit that broke that remains the same as before, f233bf47dde1d481108142c8d6b4bb3b3d8c6d08.

At one commit before that one, 6e08bde8c4525dda7d82bbf55b4b45a6e16213da, it works fine.

Ignore the code excerpt I posted from django-audit-log, looks like this is basically a NOP, even when it is working, as field.rel.related_name is always None. So that was a red herring (although very likely an unrelated bug in django-audit-log).

[Tim Graham]

I think the author of django-audit-log (or an interested person) needs to try to add compatibility with Django 1.8 as it's not clear where the issue lies (although audit-log seems to use some private APIs, so I wouldn't be surprised if a fix needs to be made there). Please reopen if after investigation you can say more definitely that there's a problem in Django.