Opened 7 years ago

Closed 3 years ago

#11906 closed Bug (fixed)

QuerySet._fill_cache is not thread-safe

Reported by: mrts Owned by: nobody
Component: Database layer (models, ORM) Version: master
Severity: Normal Keywords: threading thread
Cc: David Reynolds, macek@… Triage Stage: Accepted
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

If several threads iterate over the same queryset, the following error occurs:

File "/some/path/__init__.py", line 624, in prerender
   for obj in objects: # <-- objects is a QuerySet

 File "/some/path/python2.5/site-packages/django/db/models/query.py", line 106, in _result_iter
   self._fill_cache()

 File "/some/path/python2.5/site-packages/django/db/models/query.py", line 692, in _fill_cache
   self._result_cache.append(self._iter.next())

ValueError: generator already executing

That happens as multiple threads concurrently both append to self._result_cahce and consume self._iter (see source:/django/trunk/django/db/models/query.py@11584#L692).

Change History (14)

comment:1 Changed 7 years ago by mrts

Needs documentation: unset
Needs tests: unset
Patch needs improvement: unset

The question is how and whether this should be fixed, as using a shared queryset is not necessarily a good thing (it makes mostly sense to consume it with list() and store the list or use cache instead).

comment:2 Changed 7 years ago by mrts

Keywords: threading thread added

comment:3 Changed 7 years ago by mrts

Summarized it as follows in the unofficial threadsafety wiki page: Django core is generally threadsafe as of 1.0.3 / 1.1. However, QuerySets are known not to be thread-safe, see #11906. Usually that does not pose problems as they are (or should be) not shared between threads in Django. The exception to that rule is the use of exotic global/class-level/shared instance variable querysets in your own code (e.g. when using the ORM outside of the Django dispatch system), where you are assumed to know what you are doing and protect them appropriately anyway.

comment:4 Changed 7 years ago by Russell Keith-Magee

Triage Stage: UnreviewedAccepted

comment:5 Changed 6 years ago by David Reynolds

Just thought I'd weigh in and mention that I have seen this occurring too.

Does anyone have any idea on how we could go about fixing it?

comment:6 Changed 6 years ago by David Reynolds

Cc: David Reynolds added

comment:7 Changed 6 years ago by Jacob

milestone: 1.3

comment:8 Changed 5 years ago by Julien Phalip

Severity: Normal
Type: Bug

comment:9 Changed 5 years ago by Jacob

milestone: 1.3

Milestone 1.3 deleted

comment:11 Changed 5 years ago by Aymeric Augustin

UI/UX: unset

Change UI/UX from NULL to False.

comment:12 Changed 5 years ago by Aymeric Augustin

Easy pickings: unset

Change Easy pickings from NULL to False.

comment:13 Changed 4 years ago by Vlada Macek

Just thought to tell here it happened to me today on a not very intensive site...

comment:14 Changed 4 years ago by Vlada Macek

Cc: macek@… added

comment:15 Changed 3 years ago by Anssi Kääriäinen

Resolution: fixed
Status: newclosed

QuerySet _fill_cache is gone as part of QuerySet iterator behaviour fixing.

Note: See TracTickets for help on using tickets.
Back to Top