#20470 closed Bug (needsinfo)

Querying crashes with RuntimeError (dictionary changed size during iteration)

Reported by: jpaalasm Owned by: nobody
Component: Database layer (models, ORM) Version: 1.5
Severity: Normal Keywords:
Cc: bmispelon@… Triage Stage: Unreviewed
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

We have defined the following model.

class SignalAnalysisTask(models.Model):
    continuous_signal = models.ForeignKey(ContinuousSignal, related_name='signal_analysis_tasks')
    closed = models.BooleanField(default=False, db_index=True)
    ticket = models.OneToOneField(AnalysisTicket, null=True, blank=True, related_name='analysis_task')
    latest_fragment = models.ForeignKey(SignalFragment, null=True, blank=True, related_name='analysis_task_latest_fragments')

When the below query is performed with the model, Django crashes with "RuntimeError: dictionary changed size during iteration". See the full traceback. Django is configured to run with Apache and mod_wsgi on Linux.

open_tasks = SignalAnalysisTask.objects.filter(closed=False)
open_tasks_with_new_data = open_tasks.exclude(latest_fragment=F('continuous_signal__last_signal_fragment'))
open_tasks_with_new_data = open_tasks_with_new_data.order_by("continuous_signal__start_time")

Traceback:

Traceback (most recent call last):
  File "/usr/lib/finsor/virtualenvs/signalstorage/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 115, in get_response
    response = callback(request, *callback_args, **callback_kwargs)
  File "/usr/lib/finsor/virtualenvs/signalstorage/local/lib/python2.7/site-packages/bedsenseweb/utils/viewutils.py", line 27, in check
    return view_func(request, *args, **kwargs)
  File "/usr/lib/finsor/virtualenvs/signalstorage/local/lib/python2.7/site-packages/bedsenseweb/tickets/views.py", line 21, in acquire
    ticket = models.create_new_ticket()
  File "/usr/lib/finsor/virtualenvs/signalstorage/local/lib/python2.7/site-packages/bedsenseweb/tickets/models.py", line 41, in create_new_ticket
    open_tasks_with_new_data = open_tasks_with_new_data.order_by("continuous_signal__start_time")
  File "/usr/lib/finsor/virtualenvs/signalstorage/local/lib/python2.7/site-packages/django/db/models/query.py", line 791, in order_by
    obj = self._clone()
  File "/usr/lib/finsor/virtualenvs/signalstorage/local/lib/python2.7/site-packages/django/db/models/query.py", line 907, in _clone
    query = self.query.clone()
  File "/usr/lib/finsor/virtualenvs/signalstorage/local/lib/python2.7/site-packages/django/db/models/sql/query.py", line 264, in clone
    obj.where = copy.deepcopy(self.where, memo=memo)
  File "/usr/lib/python2.7/copy.py", line 174, in deepcopy
    y = copier(memo)
  File "/usr/lib/finsor/virtualenvs/signalstorage/local/lib/python2.7/site-packages/django/utils/tree.py", line 61, in __deepcopy__
    obj.children = copy.deepcopy(self.children, memodict)
  File "/usr/lib/python2.7/copy.py", line 163, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python2.7/copy.py", line 230, in _deepcopy_list
    y.append(deepcopy(a, memo))
  File "/usr/lib/python2.7/copy.py", line 174, in deepcopy
    y = copier(memo)
  File "/usr/lib/finsor/virtualenvs/signalstorage/local/lib/python2.7/site-packages/django/utils/tree.py", line 61, in __deepcopy__
    obj.children = copy.deepcopy(self.children, memodict)
  File "/usr/lib/python2.7/copy.py", line 163, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python2.7/copy.py", line 230, in _deepcopy_list
    y.append(deepcopy(a, memo))
  File "/usr/lib/python2.7/copy.py", line 174, in deepcopy
    y = copier(memo)
  File "/usr/lib/finsor/virtualenvs/signalstorage/local/lib/python2.7/site-packages/django/utils/tree.py", line 61, in __deepcopy__
    obj.children = copy.deepcopy(self.children, memodict)
  File "/usr/lib/python2.7/copy.py", line 163, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python2.7/copy.py", line 230, in _deepcopy_list
    y.append(deepcopy(a, memo))
  File "/usr/lib/python2.7/copy.py", line 174, in deepcopy
    y = copier(memo)
  File "/usr/lib/finsor/virtualenvs/signalstorage/local/lib/python2.7/site-packages/django/utils/tree.py", line 61, in __deepcopy__
    obj.children = copy.deepcopy(self.children, memodict)
  File "/usr/lib/python2.7/copy.py", line 163, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python2.7/copy.py", line 230, in _deepcopy_list
    y.append(deepcopy(a, memo))
  File "/usr/lib/python2.7/copy.py", line 163, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python2.7/copy.py", line 237, in _deepcopy_tuple
    y.append(deepcopy(a, memo))
  File "/usr/lib/python2.7/copy.py", line 190, in deepcopy
    y = _reconstruct(x, rv, 1, memo)
  File "/usr/lib/python2.7/copy.py", line 334, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/lib/python2.7/copy.py", line 163, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python2.7/copy.py", line 257, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python2.7/copy.py", line 190, in deepcopy
    y = _reconstruct(x, rv, 1, memo)
  File "/usr/lib/python2.7/copy.py", line 334, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/lib/python2.7/copy.py", line 163, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python2.7/copy.py", line 257, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/lib/python2.7/copy.py", line 163, in deepcopy
    y = copier(x, memo)
  File "/usr/lib/python2.7/copy.py", line 256, in _deepcopy_dict
    for key, value in x.iteritems():
RuntimeError: dictionary changed size during iteration

Change History (5)

comment:1 Changed 23 months ago by svisser

  • Needs documentation unset
  • Needs tests unset
  • Patch needs improvement unset

Some observations:

  • I couldn't reproduce this today on master with the given model and a few sample instances that I could think of.

So I think the problem isn't there anymore but if the bug can be demonstrated using a small test case then further investigation is needed.

comment:2 Changed 23 months ago by DrMeers

Thanks @svisser.

@jpaalasm are you able to confirm whether using the development version of Django fixes your problem?

If so, I wonder whether the fix for #16759 can be backported (or even partially, since there seem to be some backwards compatibility concerns which were debatably "internal"), or if we need to address the issue specifically in 1.5.

comment:3 Changed 22 months ago by bmispelon

  • Cc bmispelon@… added
  • Resolution set to needsinfo
  • Status changed from new to closed

I also tried reproducing the issue without any success.

Your example is missing the definition of the models you use in the ForeignKeys, so I had to improvise. Here's what I used:

class SignalFragment(models.Model):
    pass


class ContinuousSignal(models.Model):
    last_signal_fragment = models.ForeignKey(SignalFragment)


class AnalysisTicket(models.Model):
    pass


class SignalAnalysisTask(models.Model):
    continuous_signal = models.ForeignKey(ContinuousSignal, related_name='signal_analysis_tasks')
    closed = models.BooleanField(default=False, db_index=True)
    ticket = models.OneToOneField(AnalysisTicket, null=True, blank=True, related_name='analysis_task')
    latest_fragment = models.ForeignKey(SignalFragment, null=True, blank=True, related_name='analysis_task_latest_fragments')

I tried running the example you gave with master, 1.5, and even with 1.4 but none of them raised any exception.

In consequence, I'm marking this as needsinfo.
Please re-open with more information on how to reproduce the issue.

Thanks.

comment:4 Changed 22 months ago by jpaalasm

  • Resolution needsinfo deleted
  • Status changed from closed to new

Here's more information about the issue.

This crash occurs in about 0.3% of all requests, in a view that gets some hundreds of requests per minute.

I think that installing the development version is not currently possible, because we seem to be using some removed functionality (e.g. importing django.conf.urls.defaults fails).

comment:5 Changed 22 months ago by akaariai

  • Resolution set to needsinfo
  • Status changed from new to closed

I think this is data-dependant and trying to fix this in 1.5 is impossible without getting a reproducible test case. Also, this is very likely fixed in master as __deepcopy__ isn't used any more for cloning the query.

Backpatching the __deepcopy__ removal can't be done, it is just too big of a change and has a real possibility for nasty regressions.

I don't see any other option as to close this as needsinfo. This doesn't mean at all that we don't trust the issue exists, just that there is no way to fix this unless we can reproduce the issue.

Note: See TracTickets for help on using tickets.
Back to Top