''Part of DjangoSpecifications'' = Threading improvements in Django = According to tickets #5632, #6950 and discussions http://groups.google.com/group/django-users/browse_frm/thread/a7d42475b66530bd, http://groups.google.com/group/django-developers/browse_thread/thread/fbcfa88c997d1bb3, at least the following components of Django are not entirely thread-safe: * django.template.loader * django.db.models * django.contrib.sessions This specification outlines the changes that need to be implemented to solve these issues. A related task is to identify other components not mentioned here that have threading issues. See #1442 and http://groups.google.com/group/django-developers/browse_thread/thread/905f79e350525c95 for threading discussions. == Globals == There are three types of globals: 1. read-only globals that are never assigned to (THREAD-SAFE), 1. globals assigned to in a function by using the `global` keyword (NOT THREAD-SAFE), 1. global mutable data structures (lists and dictionaries, also instances) that are assigned to at module level but whose elements are modified in functions and that are accessed without using the `global` keyword (NOT THREAD-SAFE unless never modified). "Not thread-safe" has two broad subcategories: * inefficiencies due to calls meant to occur only once occurring more than once (the general `if not foo: initialize foo`, including `memoize` decorator), * errors due to incomplete initialization. Incomplete initialization errors can generally be avoided by using full assignment instead of elementwise modification (that's how it is done in the source mostly): '''BAD''' {{{ foo = [] def bar(): ... if not foo: for x in y: foo.append(x) }}} '''BETTER''' {{{ foo = [] def bar(): ... global foo if not foo: tmp = [] for x in y: tmp.append(x) foo = tmp # assignment is "atomic" }}} === Modules' use of globals === See below for raw `grep` results. ||'''Module'''||'''Globals'''||'''Notes'''|| ||settings and global_settings||?||not reviewed|| ||utils/_decimal.py||lots, including code||not reviewed|| ||django/contrib/sites/models.py||`SITE_CACHE`||minor, one db hit intended, more than one possible|| ||django/template/context.py||`_standard_context_processors`|||| ||`django/template/__init__.py`||`invalid_var_format_string, libraries, builtins`|||| ||django/template/loader.py||`template_source_loaders`||#6950, patch attached|| ||django/template/loaders/app_directories.py||`app_template_dirs`|||| ||django/utils/translation/trans_real.py||`_accepted, _active, _default, _translations`|||| ||django/core/urlresolvers.py||`_callable_cache, _resolver_cache`||`memoize` decorator|| ||`django/core/serializers/__init__.py`||`_serializers`|||| ||django/db/models/fields/related.py||`pending_lookups`|||| ||django/db/transaction.py||`dirty, state`|||| ||django/dispatch/dispatcher.py||`connections, senders, sendersBack|||| === Raw `grep` results === ==== Globals accessed with the `global` keyword ==== {{{ $ grep -r '^[[:space:]]*global ' django/ | egrep -v '(\.svn|\.html|\.css|\.pyc|_doctest\.py)' | sort | uniq }}} yields the following results {{{ django/contrib/sites/models.py: global SITE_CACHE django/core/management/__init__.py: global _commands django/template/context.py: global _standard_context_processors django/template/__init__.py: global invalid_var_format_string django/template/loader.py: global template_source_loaders django/utils/translation/trans_real.py: global _accepted django/utils/translation/trans_real.py: global _active django/utils/translation/trans_real.py: global _default, _active django/utils/translation/trans_real.py: global _translations django/utils/translation/trans_real.py: global _translations }}} Out of these, `django.core.management` is not used in multi-threading context. ==== Global dictionaries ==== {{{ $ grep -r '^[[:alnum:]_]\+ *= *{' django | egrep -v '(\.svn|_doctest\.py)' | sort }}} yields the following results {{{ django/conf/global_settings.py:ABSOLUTE_URL_OVERRIDES = {} django/conf/global_settings.py:DATABASE_OPTIONS = {} # Set to empty dictionary for default. django/contrib/admin/utils.py:ROLES = { django/contrib/admin/views/doc.py:DATA_TYPE_MAPPING = { django/contrib/formtools/tests.py:test_data = {'field1': u'foo', django/contrib/localflavor/ca/ca_provinces.py:PROVINCES_NORMALIZED = { django/contrib/localflavor/in_/in_states.py:STATES_NORMALIZED = { django/contrib/localflavor/us/us_states.py:STATES_NORMALIZED = { django/contrib/sites/models.py:SITE_CACHE = {} django/core/cache/__init__.py:BACKENDS = { django/core/cache/__init__.py:DEPRECATED_BACKENDS = { django/core/handlers/wsgi.py:STATUS_CODE_TEXT = { django/core/serializers/__init__.py:BUILTIN_SERIALIZERS = { django/core/serializers/__init__.py:_serializers = {} django/core/servers/basehttp.py:_hop_headers = { django/core/servers/fastcgi.py:FASTCGI_OPTIONS = { django/core/urlresolvers.py:_callable_cache = {} # Maps view and url pattern names to their view functions. django/core/urlresolvers.py:_resolver_cache = {} # Maps urlconf modules to RegexURLResolver instances. django/db/backends/dummy/creation.py:DATA_TYPES = {} django/db/backends/dummy/introspection.py:DATA_TYPES_REVERSE = {} django/db/backends/mysql/creation.py:DATA_TYPES = { django/db/backends/mysql/introspection.py:DATA_TYPES_REVERSE = { django/db/backends/mysql_old/creation.py:DATA_TYPES = { django/db/backends/mysql_old/introspection.py:DATA_TYPES_REVERSE = { django/db/backends/oracle/creation.py:DATA_TYPES = { django/db/backends/oracle/creation.py:REMEMBER = {} django/db/backends/oracle/introspection.py:DATA_TYPES_REVERSE = { django/db/backends/postgresql/creation.py:DATA_TYPES = { django/db/backends/postgresql/introspection.py:DATA_TYPES_REVERSE = { django/db/backends/postgresql_psycopg2/introspection.py:DATA_TYPES_REVERSE = { django/db/backends/sqlite3/creation.py:DATA_TYPES = { django/db/backends/sqlite3/introspection.py:BASE_DATA_TYPES_REVERSE = { django/db/models/fields/related.py:pending_lookups = {} django/db/models/query.py:LEGACY_ORDERING_MAPPING = {'ASC': '_', 'DESC': '-_', 'RANDOM': '?'} django/db/transaction.py:dirty = {} django/db/transaction.py:state = {} django/dispatch/dispatcher.py:connections = {} django/dispatch/dispatcher.py:senders = {} django/dispatch/dispatcher.py:sendersBack = {} django/template/__init__.py:libraries = {} django/utils/dates.py:MONTHS = { django/utils/dates.py:MONTHS_3 = { django/utils/dates.py:MONTHS_3_REV = { django/utils/dates.py:MONTHS_AP = { # month names in Associated Press style django/utils/dates.py:WEEKDAYS = { django/utils/dates.py:WEEKDAYS_ABBR = { django/utils/dates.py:WEEKDAYS_REV = { django/utils/_decimal.py:_condition_map = {ConversionSyntax:InvalidOperation, django/utils/_decimal.py:_infinity_map = { django/utils/simplejson/decoder.py:BACKSLASH = { django/utils/simplejson/decoder.py:_CONSTANTS = { django/utils/simplejson/encoder.py:ESCAPE_DCT = { django/utils/termcolors.py:opt_dict = {'bold': '1', 'underscore': '4', 'blink': '5', 'reverse': '7', 'conceal': '8'} django/utils/translation/trans_null.py:TECHNICAL_ID_MAP = { django/utils/translation/trans_real.py:_accepted = {} django/utils/translation/trans_real.py:_active = {} django/utils/translation/trans_real.py:_translations = {} }}} Out of these, the following are read-only (i.e. not changed anywhere in code) or otherwise irrelevant: `contrib/admin, formtools tests, localflavor mappings`, `core/cache, core/handlers, core/serializers/__init__.py:BUILTIN_SERIALIZERS`, `core/servers, db/backends, db/models/query.py, utils/dates.py`, `utils/_decimal.py, utils/simplejson, utils/termcolors.py`, `utils/translation/trans_null.py`. `SITE_CACHE` and everything in `django.utils.translation.trans_real` has already been listed under `globals` above. `_callable_cache` and `_resolver_cache` in django/core/urlresolvers.py are used within the memoize decorator, `result = func(*args)` may be called more than once in `utils/functional.py`, but this should generally be a non-issue. That leaves the following relevant global dicts not listed before: {{{ django/core/serializers/__init__.py:_serializers = {} django/db/models/fields/related.py:pending_lookups = {} django/db/transaction.py:dirty = {} django/db/transaction.py:state = {} django/dispatch/dispatcher.py:connections = {} django/dispatch/dispatcher.py:senders = {} django/dispatch/dispatcher.py:sendersBack = {} django/template/__init__.py:libraries = {} }}} ==== Global lists ==== {{{ $ grep -r '^[[:alnum:]_]\+ *= *\[' django | egrep -v '(\.svn|_doctest\.py|__all__)' | sort }}} yields the following results {{{ django/db/models/fields/__init__.py:BLANK_CHOICE_DASH = [("", "---------")] django/db/models/fields/__init__.py:BLANK_CHOICE_NONE = [("", "None")] django/template/__init__.py:builtins = [] django/template/loaders/app_directories.py:app_template_dirs = [] django/utils/_decimal.py:rounding_functions = [name for name in Decimal.__dict__.keys() if name.startswith('_round_')] django/utils/_decimal.py:_signals = [Clamped, DivisionByZero, Inexact, Overflow, Rounded, django/utils/html.py:DOTS = ['·', '*', '\xe2\x80\xa2', '•', '•', '•'] django/utils/html.py:LEADING_PUNCTUATION = ['(', '<', '<'] django/utils/html.py:TRAILING_PUNCTUATION = ['.', ',', ')', '>', '\n', '>'] django/utils/simplejson/decoder.py:ANYTHING = [ }}} Leaving out the irrelevant read-only ones, the following remain: {{{ django/template/__init__.py:builtins = [] django/template/loaders/app_directories.py:app_template_dirs = [] }}} As a matter of style, the read-only ones should really be tuples, not lists (the 'say what you mean' idiom -- if it shouldn't be modified, make it a tuple).