MemoryError when deleting objects with many generic relations

[bjourne@…]

When I delete an object which has several generic relations pointing to it, I get a memory error. It seems like there is some memory leak or infinite recursion going on because the process just locks up for a few seconds while the Python process grows to 3gb in size and then terminates with the MemoryError.

I don't have an isolated test case for it, but the problem is easy for me to reproduce and I can provide more details if requested. Here is the full traceback:

In [9]: g = Game.objects.get(id = 34519)

In [10]: g.delete()

MemoryError                               Traceback (most recent call last)

/app/tngfooty/<ipython console> in <module>()

/usr/local/lib/python2.7/dist-packages/django/db/models/base.pyc in delete(self, using)
    578 
    579         collector = Collector(using=using)
--> 580         collector.collect([self])
    581         collector.delete()
    582 

/usr/local/lib/python2.7/dist-packages/django/db/models/deletion.pyc in collect(self, objs, source, nullable, collect_related, source_attr, reverse_dependency)
    176             for relation in model._meta.many_to_many:
    177                 if not relation.rel.through:
--> 178                     sub_objs = relation.bulk_related_objects(new_objs, self.using)
    179                     self.collect(sub_objs,
    180                                  source=model,

/usr/local/lib/python2.7/dist-packages/django/contrib/contenttypes/generic.pyc in bulk_related_objects(self, objs, using)
    183 
    184         """
--> 185         return self.rel.to._base_manager.db_manager(using).filter(**{
    186                 "%s__pk" % self.content_type_field_name:
    187                     ContentType.objects.db_manager(using).get_for_model(self.model).pk,

/usr/local/lib/python2.7/dist-packages/django/db/models/manager.pyc in db_manager(self, using)
     90 
     91     def db_manager(self, using):
---> 92         obj = copy.copy(self)
     93         obj._db = using
     94         return obj

/usr/lib/python2.7/copy.pyc in copy(x)
     86         reductor = getattr(x, "__reduce_ex__", None)
     87         if reductor:
---> 88             rv = reductor(2)
     89         else:
     90             reductor = getattr(x, "__reduce__", None)

/usr/local/lib/python2.7/dist-packages/django/db/models/query.pyc in __getstate__(self)
     60         """
     61         # Force the cache to be fully populated.

---> 62         len(self)
     63 
     64         obj_dict = self.__dict__.copy()

/usr/local/lib/python2.7/dist-packages/django/db/models/query.pyc in __len__(self)
     80                 self._result_cache = list(self._iter)
     81             else:
---> 82                 self._result_cache = list(self.iterator())
     83         elif self._iter:
     84             self._result_cache.extend(self._iter)

/usr/local/lib/python2.7/dist-packages/django/db/models/query.pyc in iterator(self)
    284                 else:
    285                     # Omit aggregates in object creation.

--> 286                     obj = model(*row[index_start:aggregate_start])
    287 
    288                 # Store the source database of the object


/usr/local/lib/python2.7/dist-packages/django/db/models/base.pyc in __init__(self, *args, **kwargs)
    274 
    275     def __init__(self, *args, **kwargs):
--> 276         signals.pre_init.send(sender=self.__class__, args=args, kwargs=kwargs)
    277 
    278         # Set up the storage for instance state


/usr/local/lib/python2.7/dist-packages/django/dispatch/dispatcher.pyc in send(self, sender, **named)
    170 
    171         for receiver in self._live_receivers(_make_id(sender)):
--> 172             response = receiver(signal=self, sender=sender, **named)
    173             responses.append((receiver, response))
    174         return responses

MemoryError: 

[Anssi Kääriäinen]

Looking at the stack trace, I think I see the error: The copy.copy() in db_manager() is causing the cache to be filled, in other words, the query is executed, without any filters. That is not good!

I haven't verified anything about this yet. The reason is I am having a hard time to set up generic relations, as the documentation doesn't give me a copy-paste example. Could you provide this, and I will see what I can find out.

[bjourne]

I have found the cause for this. For the child model in the GenericRelation, I'm replacing the default manager with a custom one. That manager has some slight magic, like overriding the getattr method which may be what is triggering the bug. I will investigate some more and see if it's just PEBKAC or if Django does something wrong.

[Anssi Kääriäinen]

While investigating this, I could not reproduce any extra queries using the test suite models.

I am closing this as invalid. If you find out that there is reason to believe there is an bug in Django, then reopen this ticket.