Opened 5 years ago

Closed 5 years ago

#24768 closed Bug (needsinfo)

KeyError: u"Problem installing fixture '<filename>.json': u'manager'"

Reported by: Brian May Owned by: nobody
Component: Database layer (models, ORM) Version: 1.8
Severity: Normal Keywords:
Cc: Triage Stage: Unreviewed
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

Since upgrading to Django 1.8, my tests are failing to load the test data, and I don't see anything about this in the release notes:

======================================================================
ERROR: setUpClass (karaage.tests.test_people.PersonTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/python/local/lib/python2.7/site-packages/django/test/testcases.py", line 952, in setUpClass
    'database': db_name,
  File "/tmp/python/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 120, in call_command
    return command.execute(*args, **defaults)
  File "/tmp/python/local/lib/python2.7/site-packages/django/core/management/base.py", line 441, in execute
    output = self.handle(*args, **options)
  File "/tmp/python/local/lib/python2.7/site-packages/django/core/management/commands/loaddata.py", line 60, in handle
    self.loaddata(fixture_labels)
  File "/tmp/python/local/lib/python2.7/site-packages/django/core/management/commands/loaddata.py", line 90, in loaddata
    self.load_label(fixture_label)
  File "/tmp/python/local/lib/python2.7/site-packages/django/core/management/commands/loaddata.py", line 147, in load_label
    obj.save(using=self.using)
  File "/tmp/python/local/lib/python2.7/site-packages/django/core/serializers/base.py", line 173, in save
    models.Model.save_base(self.object, using=using, raw=True)
  File "/tmp/python/local/lib/python2.7/site-packages/django/db/models/base.py", line 747, in save_base
    update_fields=update_fields, raw=raw, using=using)
  File "/tmp/python/local/lib/python2.7/site-packages/django/dispatch/dispatcher.py", line 201, in send
    response = receiver(signal=self, sender=sender, **named)
  File "/home/brian/tree/django/karaage/karaage/karaage/people/models.py", line 508, in _project_membership_changed
    projectmembership__project=project,
  File "/home/brian/tree/django/karaage/karaage/karaage/people/models.py", line 644, in sync_members
    *[tmp_person_id for tmp_person_id in add_member_ids])
  File "/tmp/python/local/lib/python2.7/site-packages/django/db/models/fields/related.py", line 973, in add
    self._add_items(self.source_field_name, self.target_field_name, *objs)
  File "/tmp/python/local/lib/python2.7/site-packages/django/db/models/fields/related.py", line 1105, in _add_items
    model=self.model, pk_set=new_ids, using=db)
  File "/tmp/python/local/lib/python2.7/site-packages/django/dispatch/dispatcher.py", line 201, in send
    response = receiver(signal=self, sender=sender, **named)
  File "/home/brian/tree/django/karaage/karaage/karaage/people/models.py", line 708, in _members_changed
    _add_person_to_group(person, group)
  File "/home/brian/tree/django/karaage/karaage/karaage/people/models.py", line 673, in _add_person_to_group
    add_accounts_to_project(a_list, group.project)
  File "/home/brian/tree/django/karaage/karaage/karaage/datastores/__init__.py", line 472, in add_accounts_to_project
    query = query.filter(machine_category__projectquota__project=project)
  File "/tmp/python/local/lib/python2.7/site-packages/django/db/models/query.py", line 679, in filter
    return self._filter_or_exclude(False, *args, **kwargs)
  File "/tmp/python/local/lib/python2.7/site-packages/django/db/models/query.py", line 697, in _filter_or_exclude
    clone.query.add_q(Q(*args, **kwargs))
  File "/tmp/python/local/lib/python2.7/site-packages/django/db/models/sql/query.py", line 1301, in add_q
    clause, require_inner = self._add_q(where_part, self.used_aliases)
  File "/tmp/python/local/lib/python2.7/site-packages/django/db/models/sql/query.py", line 1328, in _add_q
    current_negated=current_negated, connector=connector, allow_joins=allow_joins)
  File "/tmp/python/local/lib/python2.7/site-packages/django/db/models/sql/query.py", line 1150, in build_filter
    value, lookups, used_joins = self.prepare_lookup_value(value, lookups, can_reuse, allow_joins)
  File "/tmp/python/local/lib/python2.7/site-packages/django/db/models/sql/query.py", line 997, in prepare_lookup_value
    value = value()
  File "/tmp/python/local/lib/python2.7/site-packages/django/db/models/fields/related.py", line 691, in __call__
    manager = getattr(self.model, kwargs.pop('manager'))
KeyError: u"Problem installing fixture '/home/brian/tree/django/karaage/karaage/karaage/fixtures/test_karaage.json': u'manager'"

----------------------------------------------------------------------

Change History (14)

comment:1 Changed 5 years ago by Brian May

I think the message is somewhat confusing; this isn't a problem with test_karaage.json, rather it an exception generated in response to a signal.

Here is the relevant function call that generates the exception:

File "/home/brian/tree/django/karaage/karaage/karaage/datastores/__init__.py", line 472, in add_accounts_to_project
    query = query.filter(machine_category__projectquota__project=project)

comment:2 Changed 5 years ago by Brian May

Ok, this is getting a tad confusing. For some reason the following code

group.project

returns a RelatedManager instead of a Project (did I miss something in the release notes about this?). As in, the following code:

print("======", type(group), type(group.project))

displays:

('======', <class 'karaage.people.models.Group'>, <class 'django.db.models.fields.related.RelatedManager'>)

Group.project is a reverse relationship from a Project:

from mptt.models import MPTTModel
class Project(MPTTModel):
    ...
    group = models.OneToOneField(Group)
    ...

So I believe group.project is suppose to return a Project, not a RelatedManager. I then pass the "project" to the Django function:

query = query.filter(machine_category__projectquota__project=project)

Which gets confused because it has the wrong type.

comment:3 Changed 5 years ago by Tim Graham

It's a bit difficult to say what the issue is here without a way to reproduce it. Either a minimal project that we can use to reproduce the issue or a better diagnosis of what's going wrong would be helpful.

comment:4 Changed 5 years ago by Brian May

Will try to simplify things. I have a suspicion however, unless OneToOneField is broken for everyone (which I doubt), that simplifying the code will not trigger the problem anymore.

Can I confirm that group.project, the reverse relation for a OneToOneField from the project, should return a group and not a RelatedManager?

It would appear that OneToOneField is generating an incorrect reverse mapping in the related object.

comment:5 Changed 5 years ago by Brian May

Also, in case I wasn't clear, this code works perfectly with Django 1.7; so this does look like a regression in Django 1.8.

comment:6 Changed 5 years ago by Tim Graham

Yes, I'd expect the reverse relation attribute to be a model instance.

comment:7 Changed 5 years ago by Brian May

Just did a git bisect, and found that the following commit f233bf47dde1d481108142c8d6b4bb3b3d8c6d08 in Django appears to introduce this problem.

commit f233bf47dde1d481108142c8d6b4bb3b3d8c6d08
Author: Anssi Kääriäinen <akaariai@gmail.com>
Date:   Sat Nov 9 14:25:15 2013 +0200

    Fixed #21414 -- Removed RelatedObject and deprecated Field.related.

Now to investigate why...

comment:8 Changed 5 years ago by Brian May

I note that before commit f233bf47dde1d481108142c8d6b4bb3b3d8c6d08, SingleRelatedObjectDescriptor is used, after that commit, ForeignRelatedObjectsDescriptor is used.

At quick glance, this seems wrong for a OneToOneField, however out of time to investigate just now. Will check this out later.

comment:9 Changed 5 years ago by Brian May

Ok, so I think the reason for the wrong type is because there is a clash between two reverse accessors. Somehow django-audit-log is involved.

However, there was no clash in Django 1.7, and I don't understand why there should be any clash in Django 1.8 either.

I have test code that will illustrate the problem, see https://github.com/brianmay/django-24768

% python ./manage.py makemigrations polls --dry
SystemCheckError: System check identified some issues:

ERRORS:
polls.Project.group: (fields.E304) Reverse accessor for 'Project.group' clashes with reverse accessor for 'Project.group'.
	HINT: Add or change a related_name argument to the definition for 'Project.group' or 'Project.group'.
polls.Project.group: (fields.E305) Reverse query name for 'Project.group' clashes with reverse query name for 'Project.group'.
	HINT: Add or change a related_name argument to the definition for 'Project.group' or 'Project.group'.
polls.ProjectAuditLogEntry.group: (fields.E304) Reverse accessor for 'ProjectAuditLogEntry.group' clashes with reverse accessor for 'Project.group'.
	HINT: Add or change a related_name argument to the definition for 'ProjectAuditLogEntry.group' or 'Project.group'.

I cannot reproduce my initial problem with this test program (see test.py in that repository), however I strongly suspect that this was just a symptom of the above problem.

The first two messages don't seem to make any sense. Project.group clashes with itself???

The last one is probably more significant: Reverse accessor for 'ProjectAuditLogEntry.group' clashes with reverse accessor for 'Project.group'.

Shouldn't the first be called projectauditlogentry_set and the second be called project? Maybe Django 1.8 is getting the related_name wrong in this particular situation?

comment:10 Changed 5 years ago by Brian May

Also see https://github.com/Atomidata/django-audit-log/issues/21

I don't know if this is a django bug or django-audit-log bug, I don't really understand how this automatic table generation works.

comment:11 Changed 5 years ago by Brian May

The following code is in django-audit-log - can somebody tell me if this is valid Django 1.8 code? I suspect maybe not...

if field.rel and field.rel.related_name:
    field.rel.related_name = '_auditlog_%s' % field.rel.related_name

If this code is not valid any more, what should it be?

I see that field.related was depreciated in favor on field.rel, however I can't find anything about the field.rel API having changed.

comment:12 Changed 5 years ago by Tim Graham

That code looks okay to me. It would be helpful if you could bisect Django's commit history to identify the commit that caused the breakage.

comment:13 Changed 5 years ago by Brian May

The git commit that broke that remains the same as before, f233bf47dde1d481108142c8d6b4bb3b3d8c6d08.

At one commit before that one, 6e08bde8c4525dda7d82bbf55b4b45a6e16213da, it works fine.

Ignore the code excerpt I posted from django-audit-log, looks like this is basically a NOP, even when it is working, as field.rel.related_name is always None. So that was a red herring (although very likely an unrelated bug in django-audit-log).

comment:14 Changed 5 years ago by Tim Graham

Resolution: needsinfo
Status: newclosed

I think the author of django-audit-log (or an interested person) needs to try to add compatibility with Django 1.8 as it's not clear where the issue lies (although audit-log seems to use some private APIs, so I wouldn't be surprised if a fix needs to be made there). Please reopen if after investigation you can say more definitely that there's a problem in Django.

Note: See TracTickets for help on using tickets.
Back to Top