Querying crashes with RuntimeError (dictionary changed size during iteration)

[Joonas Paalasmaa]

We have defined the following model.

class SignalAnalysisTask(models.Model):
    continuous_signal = models.ForeignKey(ContinuousSignal, related_name='signal_analysis_tasks')
    closed = models.BooleanField(default=False, db_index=True)
    ticket = models.OneToOneField(AnalysisTicket, null=True, blank=True, related_name='analysis_task')
    latest_fragment = models.ForeignKey(SignalFragment, null=True, blank=True, related_name='analysis_task_latest_fragments')

When the below query is performed with the model, Django crashes with "RuntimeError: dictionary changed size during iteration". See the full traceback. Django is configured to run with Apache and mod_wsgi on Linux.

open_tasks = SignalAnalysisTask.objects.filter(closed=False)
open_tasks_with_new_data = open_tasks.exclude(latest_fragment=F('continuous_signal__last_signal_fragment'))
open_tasks_with_new_data = open_tasks_with_new_data.order_by("continuous_signal__start_time")

Traceback:

Traceback (most recent call last):
  File "/usr/lib/finsor/virtualenvs/signalstorage/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 115, in get_response
    response = callback(request, *callback_args, **callback_kwargs)
  File "/usr/lib/finsor/virtualenvs/signalstorage/local/lib/python2.7/site-packages/bedsenseweb/utils/viewutils.py", line 27, in check
    return view_func(request, *args, **kwargs)
  File "/usr/lib/finsor/virtualenvs/signalstorage/local/lib/python2.7/site-packages/bedsenseweb/tickets/views.py", line 21, in acquire
    ticket = models.create_new_ticket()
  File "/usr/lib/finsor/virtualenvs/signalstorage/local/lib/python2.7/site-packages/bedsenseweb/tickets/models.py", line 41, in create_new_ticket
    open_tasks_with_new_data = open_tasks_with_new_data.order_by("continuous_signal__start_time")
  File "/usr/lib/finsor/virtualenvs/signalstorage/local/lib/python2.7/site-packages/django/db/models/query.py", line 791, in order_by
    obj = self._clone()
  File "/usr/lib/finsor/virtualenvs/signalstorage/local/lib/python2.7/site-packages/django/db/models/query.py", line 907, in _clone
    query = self.query.clone()
  File "/usr/lib/finsor/virtualenvs/signalstorage/local/lib/python2.7/site-packages/django/db/models/sql/query.py", line 264, in clone
    obj.where = copy.deepcopy(self.where, memo=memo)
  File "/usr/lib/python2.7/copy.py", line 174, in deepcopy
    y = copier(memo)
  File "/usr/lib/finsor/virtualenvs/signalstorage/local/lib/python2.7/site-packages/django/utils/tree.py", line 61, in __deepcopy__
    obj.children = copy.deepcopy(self.children, memodict)
  File "/usr/lib/python2.7/copy.py", line 163, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python2.7/copy.py", line 230, in _deepcopy_list
    y.append(deepcopy(a, memo))
  File "/usr/lib/python2.7/copy.py", line 174, in deepcopy
    y = copier(memo)
  File "/usr/lib/finsor/virtualenvs/signalstorage/local/lib/python2.7/site-packages/django/utils/tree.py", line 61, in __deepcopy__
    obj.children = copy.deepcopy(self.children, memodict)
  File "/usr/lib/python2.7/copy.py", line 163, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python2.7/copy.py", line 230, in _deepcopy_list
    y.append(deepcopy(a, memo))
  File "/usr/lib/python2.7/copy.py", line 174, in deepcopy
    y = copier(memo)
  File "/usr/lib/finsor/virtualenvs/signalstorage/local/lib/python2.7/site-packages/django/utils/tree.py", line 61, in __deepcopy__
    obj.children = copy.deepcopy(self.children, memodict)
  File "/usr/lib/python2.7/copy.py", line 163, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python2.7/copy.py", line 230, in _deepcopy_list
    y.append(deepcopy(a, memo))
  File "/usr/lib/python2.7/copy.py", line 174, in deepcopy
    y = copier(memo)
  File "/usr/lib/finsor/virtualenvs/signalstorage/local/lib/python2.7/site-packages/django/utils/tree.py", line 61, in __deepcopy__
    obj.children = copy.deepcopy(self.children, memodict)
  File "/usr/lib/python2.7/copy.py", line 163, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python2.7/copy.py", line 230, in _deepcopy_list
    y.append(deepcopy(a, memo))
  File "/usr/lib/python2.7/copy.py", line 163, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python2.7/copy.py", line 237, in _deepcopy_tuple
    y.append(deepcopy(a, memo))
  File "/usr/lib/python2.7/copy.py", line 190, in deepcopy
    y = _reconstruct(x, rv, 1, memo)
  File "/usr/lib/python2.7/copy.py", line 334, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/lib/python2.7/copy.py", line 163, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python2.7/copy.py", line 257, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python2.7/copy.py", line 190, in deepcopy
    y = _reconstruct(x, rv, 1, memo)
  File "/usr/lib/python2.7/copy.py", line 334, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/lib/python2.7/copy.py", line 163, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python2.7/copy.py", line 257, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python2.7/copy.py", line 163, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python2.7/copy.py", line 256, in _deepcopy_dict
    for key, value in x.iteritems():
RuntimeError: dictionary changed size during iteration

[svisser]

Some observations:

• I couldn't reproduce this today on master with the given model and a few sample instances that I could think of.

• The line obj.where = copy.deepcopy(self.where, memo=memo) (line 264 in clone) is present in Django 1.5 but no longer in master as it was rewritten for ticket #16759 (commit: ​https://github.com/django/django/commit/23ca3a01940c63942885df4709712cebf4df79ec).

So I think the problem isn't there anymore but if the bug can be demonstrated using a small test case then further investigation is needed.

[Simon Meers]

Thanks @svisser.

@jpaalasm are you able to confirm whether using the development version of Django fixes your problem?

If so, I wonder whether the fix for #16759 can be backported (or even partially, since there seem to be some backwards compatibility concerns which were debatably "internal"), or if we need to address the issue specifically in 1.5.

[Baptiste Mispelon]

I also tried reproducing the issue without any success.

Your example is missing the definition of the models you use in the ForeignKeys, so I had to improvise. Here's what I used:

class SignalFragment(models.Model):
    pass


class ContinuousSignal(models.Model):
    last_signal_fragment = models.ForeignKey(SignalFragment)


class AnalysisTicket(models.Model):
    pass


class SignalAnalysisTask(models.Model):
    continuous_signal = models.ForeignKey(ContinuousSignal, related_name='signal_analysis_tasks')
    closed = models.BooleanField(default=False, db_index=True)
    ticket = models.OneToOneField(AnalysisTicket, null=True, blank=True, related_name='analysis_task')
    latest_fragment = models.ForeignKey(SignalFragment, null=True, blank=True, related_name='analysis_task_latest_fragments')

I tried running the example you gave with master, 1.5, and even with 1.4 but none of them raised any exception.

In consequence, I'm marking this as needsinfo.
Please re-open with more information on how to reproduce the issue.

Thanks.

[Joonas Paalasmaa]

Here's more information about the issue.

This crash occurs in about 0.3% of all requests, in a view that gets some hundreds of requests per minute.

I think that installing the development version is not currently possible, because we seem to be using some removed functionality (e.g. importing django.conf.urls.defaults fails).

[Anssi Kääriäinen]

I think this is data-dependant and trying to fix this in 1.5 is impossible without getting a reproducible test case. Also, this is very likely fixed in master as __deepcopy__ isn't used any more for cloning the query.

Backpatching the __deepcopy__ removal can't be done, it is just too big of a change and has a real possibility for nasty regressions.

I don't see any other option as to close this as needsinfo. This doesn't mean at all that we don't trust the issue exists, just that there is no way to fix this unless we can reproduce the issue.