Opened 15 years ago

Closed 11 years ago

#11906 closed Bug (fixed)

QuerySet._fill_cache is not thread-safe

Reported by: mrts Owned by: nobody
Component: Database layer (models, ORM) Version: dev
Severity: Normal Keywords: threading thread
Cc: David Reynolds, macek@… Triage Stage: Accepted
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

If several threads iterate over the same queryset, the following error occurs:

File "/some/path/__init__.py", line 624, in prerender
   for obj in objects: # <-- objects is a QuerySet

 File "/some/path/python2.5/site-packages/django/db/models/query.py", line 106, in _result_iter
   self._fill_cache()

 File "/some/path/python2.5/site-packages/django/db/models/query.py", line 692, in _fill_cache
   self._result_cache.append(self._iter.next())

ValueError: generator already executing

That happens as multiple threads concurrently both append to self._result_cahce and consume self._iter (see source:/django/trunk/django/db/models/query.py@11584#L692).

Change History (14)

comment:1 by mrts, 15 years ago

The question is how and whether this should be fixed, as using a shared queryset is not necessarily a good thing (it makes mostly sense to consume it with list() and store the list or use cache instead).

comment:2 by mrts, 15 years ago

Keywords: threading thread added

comment:3 by mrts, 15 years ago

Summarized it as follows in the unofficial threadsafety wiki page: Django core is generally threadsafe as of 1.0.3 / 1.1. However, QuerySets are known not to be thread-safe, see #11906. Usually that does not pose problems as they are (or should be) not shared between threads in Django. The exception to that rule is the use of exotic global/class-level/shared instance variable querysets in your own code (e.g. when using the ORM outside of the Django dispatch system), where you are assumed to know what you are doing and protect them appropriately anyway.

comment:4 by Russell Keith-Magee, 14 years ago

Triage Stage: UnreviewedAccepted

comment:5 by David Reynolds, 13 years ago

Just thought I'd weigh in and mention that I have seen this occurring too.

Does anyone have any idea on how we could go about fixing it?

comment:6 by David Reynolds, 13 years ago

Cc: David Reynolds added

comment:7 by Jacob, 13 years ago

milestone: 1.3

comment:8 by Julien Phalip, 13 years ago

Severity: Normal
Type: Bug

comment:9 by Jacob, 12 years ago

milestone: 1.3

Milestone 1.3 deleted

comment:11 by Aymeric Augustin, 12 years ago

UI/UX: unset

Change UI/UX from NULL to False.

comment:12 by Aymeric Augustin, 12 years ago

Easy pickings: unset

Change Easy pickings from NULL to False.

comment:13 by Vlada Macek, 12 years ago

Just thought to tell here it happened to me today on a not very intensive site...

comment:14 by Vlada Macek, 12 years ago

Cc: macek@… added

comment:15 by Anssi Kääriäinen, 11 years ago

Resolution: fixed
Status: newclosed

QuerySet _fill_cache is gone as part of QuerySet iterator behaviour fixing.

Note: See TracTickets for help on using tickets.
Back to Top