Opened 5 years ago

Closed 5 years ago

#17619 closed Bug (invalid)

MemoryError when deleting objects with many generic relations

Reported by: bjourne@… Owned by: nobody
Component: contrib.contenttypes Version: 1.3
Severity: Normal Keywords:
Cc: bjourne@… Triage Stage: Unreviewed
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no


When I delete an object which has several generic relations pointing to it, I get a memory error. It seems like there is some memory leak or infinite recursion going on because the process just locks up for a few seconds while the Python process grows to 3gb in size and then terminates with the MemoryError.

I don't have an isolated test case for it, but the problem is easy for me to reproduce and I can provide more details if requested. Here is the full traceback:

In [9]: g = Game.objects.get(id = 34519)

In [10]: g.delete()

MemoryError                               Traceback (most recent call last)

/app/tngfooty/<ipython console> in <module>()

/usr/local/lib/python2.7/dist-packages/django/db/models/base.pyc in delete(self, using)
    579         collector = Collector(using=using)
--> 580         collector.collect([self])
    581         collector.delete()

/usr/local/lib/python2.7/dist-packages/django/db/models/deletion.pyc in collect(self, objs, source, nullable, collect_related, source_attr, reverse_dependency)
    176             for relation in model._meta.many_to_many:
    177                 if not relation.rel.through:
--> 178                     sub_objs = relation.bulk_related_objects(new_objs, self.using)
    179                     self.collect(sub_objs,
    180                                  source=model,

/usr/local/lib/python2.7/dist-packages/django/contrib/contenttypes/generic.pyc in bulk_related_objects(self, objs, using)
    184         """
--> 185         return**{
    186                 "%s__pk" % self.content_type_field_name:
    187                     ContentType.objects.db_manager(using).get_for_model(self.model).pk,

/usr/local/lib/python2.7/dist-packages/django/db/models/manager.pyc in db_manager(self, using)
     91     def db_manager(self, using):
---> 92         obj = copy.copy(self)
     93         obj._db = using
     94         return obj

/usr/lib/python2.7/copy.pyc in copy(x)
     86         reductor = getattr(x, "__reduce_ex__", None)
     87         if reductor:
---> 88             rv = reductor(2)
     89         else:
     90             reductor = getattr(x, "__reduce__", None)

/usr/local/lib/python2.7/dist-packages/django/db/models/query.pyc in __getstate__(self)
     60         """
     61         # Force the cache to be fully populated.

---> 62         len(self)
     64         obj_dict = self.__dict__.copy()

/usr/local/lib/python2.7/dist-packages/django/db/models/query.pyc in __len__(self)
     80                 self._result_cache = list(self._iter)
     81             else:
---> 82                 self._result_cache = list(self.iterator())
     83         elif self._iter:
     84             self._result_cache.extend(self._iter)

/usr/local/lib/python2.7/dist-packages/django/db/models/query.pyc in iterator(self)
    284                 else:
    285                     # Omit aggregates in object creation.

--> 286                     obj = model(*row[index_start:aggregate_start])
    288                 # Store the source database of the object

/usr/local/lib/python2.7/dist-packages/django/db/models/base.pyc in __init__(self, *args, **kwargs)
    275     def __init__(self, *args, **kwargs):
--> 276         signals.pre_init.send(sender=self.__class__, args=args, kwargs=kwargs)
    278         # Set up the storage for instance state

/usr/local/lib/python2.7/dist-packages/django/dispatch/dispatcher.pyc in send(self, sender, **named)
    171         for receiver in self._live_receivers(_make_id(sender)):
--> 172             response = receiver(signal=self, sender=sender, **named)
    173             responses.append((receiver, response))
    174         return responses


Change History (3)

comment:1 Changed 5 years ago by Anssi Kääriäinen

Needs documentation: unset
Needs tests: unset
Patch needs improvement: unset

Looking at the stack trace, I think I see the error: The copy.copy() in db_manager() is causing the cache to be filled, in other words, the query is executed, without any filters. That is not good!

I haven't verified anything about this yet. The reason is I am having a hard time to set up generic relations, as the documentation doesn't give me a copy-paste example. Could you provide this, and I will see what I can find out.

comment:2 Changed 5 years ago by bjourne

I have found the cause for this. For the child model in the GenericRelation, I'm replacing the default manager with a custom one. That manager has some slight magic, like overriding the getattr method which may be what is triggering the bug. I will investigate some more and see if it's just PEBKAC or if Django does something wrong.

comment:3 Changed 5 years ago by Anssi Kääriäinen

Resolution: invalid
Status: newclosed

While investigating this, I could not reproduce any extra queries using the test suite models.

I am closing this as invalid. If you find out that there is reason to believe there is an bug in Django, then reopen this ticket.

Note: See TracTickets for help on using tickets.
Back to Top