ORM objects store connection name, but django does not include it during memcache lookup

[kush@…]

Hi,

We ran into a strange problem with the way ORM objects are being looked up in memcache, by django.
Our setup include two application servers that share the same memcache instance.
We also run a mysql-master and read-replicas.

Appserver1 uses the default mechanism of connecting to the read-replicas (using DATABASES dictionary in settings.py).
Appserver2 uses a DB-router to route writes to the master and reads to the replicas. Please see below for appserver2 settings file:

We noticed that Appserver1 was trying to make a connection to the DB alias called "read", while it only had default settings to connect to the DB (no references to "read" instance are present on appserver-1).

On investigating how this happened we found the following:

1. Appserver-2 got a read request and used the "read" alias to fetch and store the corresponding ORM object in memcache.
2. The ORM object that is stored seems to maintain a reference to the DB connection/alias ("read" in this case).
3. Appserver-1 does a memcache lookup and finds the same ORM object (that has a reference to the DB connection "read").
4. When Appserver-1 tries to use the ORM object, it throws an error saying that it does not know about the "read" connection.

The error message is shown below:

---

...
...
....
File "/usr/local/lib/python2.6/dist-packages/django/db/models/manager.py", line 120, in count

return self.get_query_set().count()

File "/usr/local/lib/python2.6/dist-packages/django/db/models/query.py", line 332, in count

return self.query.get_count(using=self.db)

File "/usr/local/lib/python2.6/dist-packages/django/db/models/sql/query.py", line 407, in get_count

number = obj.get_aggregation(using=using)[None]

File "/usr/local/lib/python2.6/dist-packages/django/db/models/sql/query.py", line 373, in get_aggregation

result = query.get_compiler(using).execute_sql(SINGLE)

File "/usr/local/lib/python2.6/dist-packages/django/db/models/sql/query.py", line 213, in get_compiler

connection = connections[using]

File "/usr/local/lib/python2.6/dist-packages/django/db/utils.py", line 90, in getitem

self.ensure_defaults(alias)

File "/usr/local/lib/python2.6/dist-packages/django/db/utils.py", line 72, in ensure_defaults

raise ConnectionDoesNotExist("The connection %s doesn't exist" % alias)

ConnectionDoesNotExist: The connection read doesn't exist

---

Django Version: 1.2.7

---

Appserver-2 settings:

---

DATABASES = {

'read': {

......
/ Connects to the read-replicas /

},
'default': {

......
/ Connects to the write-master /

},

[PhiR_42]

Could you please explicit what you mean by "ORM objects" ? Do you store querysets in memcache ? or instances of your own django models ?

It seems reasonable that a queryset would store its DB connection.

[anonymous]

We have memcache defined as the CACHE_BACKEND for our application so Django is using that for it's own caching of querysets.

The problem here is that it results in an unexpected situation where Django stores the connection name as well in cache but requests on that connection name rather than picking data from cache. IMO, It should either serve the data from cache or query for a valid connection name again using the DB Router.

[Jacob]

I don't think this is something Django needs to support. If you're sharing cache between two servers with differetn settings you're doing something fairly complex, and it's pretty far outside the realm of what Django's intended to do.

On a side-note: you probably don't want to be storing querysets in cache anyway; it's brittle and doesn't always work (as you've found out!). It's far safer to store list(qs) instead of qs itself.