Assertions in libmemcached when multiple django.core.cache.backends.memcached.PyLibMCCache backends used

[Michał Łowicki]

We're running Django 1.8.3 via uWSGI with threads enabled (uWSGI 2.0.11.5, configuration - ​https://gist.github.com/mlowicki/9a5042b5411537a51bbf).

When two backends using 'django.core.cache.backends.memcached.PyLibMCCache' are added and are used (each box is handling at least hundreds requests per second) then in uWSGI logs we're seeing lots of ​https://gist.github.com/mlowicki/a8b3dbd98f32dc40c4eb and application is misbehaving throwing 500s. When launched in second datacenter with much less threads then problem is much smaller but still exists (uWSGI configuration - ​https://gist.github.com/mlowicki/cbd18c689c3e2f838bf0)

Tried with pylibmc 1.4.1 and 1.5.0. Running on Debian Wheezy.

[Tim Graham]

It's not clear to me that Django is at fault. Can you provide any details about that?

[Michał Łowicki]

I'm not 100% sure it's Django's fault but according to ​https://docs.djangoproject.com/en/1.8/topics/cache/#django.core.cache.caches it should be thread-safe.

[Tim Graham]

Okay, please use TicketClosingReasons/UseSupportChannels and reopen this ticket with more details if does turn out to be a bug in Django. Thanks!

[Michał Łowicki]

FunkyBob told me 1h ago on IRC to file what I've but not it's closed within couple of minutes...

[Simon Charette]

@mlowicki, I'm not familar with uWSGI's threading but Django doesn't reuse pylicmc.Client between threads.

Did you try using python-memcached instead?

[Michał Łowicki]

Yes, for 2nd backend I'm using python-memcached (and it works) but pylimbc has much more options to configure and support for binary protocol. I'll try to debug in pylibmc to see if this will get me somewhere.

[Simon Charette]

Alright, keep us updated.

FWIW Django stores aliased cache instances in a threading.local ​property and the wrapped pylibmc.Client instances are created in PyLibMCCache._cache.

You might want to investigate if pylibmc.ThreadMappedPool does anything different since it looks like it ​solved a similar issue.

#11331 and #5133 might also be related.

[Curtis Maloney]

I believe the specific quirk of this case is using the same memcache driver for two separate cache backends ... so it's likely a problem independent of threading, but possibly that the lib isn't save to use on two connections concurrently in the same thread.

@mlowicki could you please try to replicate the issue without threads?

[Florian Apolloner]

Replying to funkybob:

@mlowicki could you please try to replicate the issue without threads?

I agree on that. I see no reason why this would be a problem in Django, we properly use one Connection per thread per alias. Build your python/memcached/pylibmc with debug infos and throw GDB at it.

[Michał Łowicki]

Just tried without threads and it works just fine.

[Michał Łowicki]

It turned out we were assigning cache backend to variable at the top of the module so all threads were using the same instance.

[Florian Apolloner]

Jupp, that certainly does cause such behavior.