Opened 6 years ago

Closed 22 months ago

#11906 closed Bug (fixed)

QuerySet._fill_cache is not thread-safe

Reported by: mrts Owned by: nobody
Component: Database layer (models, ORM) Version: master
Severity: Normal Keywords: threading thread
Cc: DavidReynolds, macek@… Triage Stage: Accepted
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

If several threads iterate over the same queryset, the following error occurs:

File "/some/path/__init__.py", line 624, in prerender
   for obj in objects: # <-- objects is a QuerySet

 File "/some/path/python2.5/site-packages/django/db/models/query.py", line 106, in _result_iter
   self._fill_cache()

 File "/some/path/python2.5/site-packages/django/db/models/query.py", line 692, in _fill_cache
   self._result_cache.append(self._iter.next())

ValueError: generator already executing

That happens as multiple threads concurrently both append to self._result_cahce and consume self._iter (see source:/django/trunk/django/db/models/query.py@11584#L692).

Change History (14)

comment:1 Changed 6 years ago by mrts

  • Needs documentation unset
  • Needs tests unset
  • Patch needs improvement unset

The question is how and whether this should be fixed, as using a shared queryset is not necessarily a good thing (it makes mostly sense to consume it with list() and store the list or use cache instead).

comment:2 Changed 6 years ago by mrts

  • Keywords threading thread added

comment:3 Changed 6 years ago by mrts

Summarized it as follows in the unofficial threadsafety wiki page: Django core is generally threadsafe as of 1.0.3 / 1.1. However, QuerySets are known not to be thread-safe, see #11906. Usually that does not pose problems as they are (or should be) not shared between threads in Django. The exception to that rule is the use of exotic global/class-level/shared instance variable querysets in your own code (e.g. when using the ORM outside of the Django dispatch system), where you are assumed to know what you are doing and protect them appropriately anyway.

comment:4 Changed 5 years ago by russellm

  • Triage Stage changed from Unreviewed to Accepted

comment:5 Changed 5 years ago by DavidReynolds

Just thought I'd weigh in and mention that I have seen this occurring too.

Does anyone have any idea on how we could go about fixing it?

comment:6 Changed 5 years ago by DavidReynolds

  • Cc DavidReynolds added

comment:7 Changed 4 years ago by jacob

  • milestone set to 1.3

comment:8 Changed 4 years ago by julien

  • Severity set to Normal
  • Type set to Bug

comment:9 Changed 4 years ago by jacob

  • milestone 1.3 deleted

Milestone 1.3 deleted

comment:11 Changed 3 years ago by aaugustin

  • UI/UX unset

Change UI/UX from NULL to False.

comment:12 Changed 3 years ago by aaugustin

  • Easy pickings unset

Change Easy pickings from NULL to False.

comment:13 Changed 3 years ago by Tuttle

Just thought to tell here it happened to me today on a not very intensive site...

comment:14 Changed 3 years ago by Tuttle

  • Cc macek@… added

comment:15 Changed 22 months ago by akaariai

  • Resolution set to fixed
  • Status changed from new to closed

QuerySet _fill_cache is gone as part of QuerySet iterator behaviour fixing.

Note: See TracTickets for help on using tickets.
Back to Top