Changeset 8342 makes test suite hang if memcached is not running

[jcrocholl]

Up to changeset [8341] my project works okay. After upgrading to [8342], manage.py test hangs at 100% CPU, with the following Traceback after Ctrl+C:

............^CTraceback (most recent call last):
  ... (beginning of doctest stack trace omitted) ...
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/unittest.py", line 428, in __call__
    return self.run(*args, **kwds)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/unittest.py", line 424, in run
    test(result)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/unittest.py", line 281, in __call__
    return self.run(*args, **kwds)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/unittest.py", line 260, in run
    testMethod()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-packages/django/test/_doctest.py", line 2174, in runTest
    failures, tries = runner.run(
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-packages/django/test/_doctest.py", line 1403, in run
    return self.__run(test, compileflags, out)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-packages/django/test/_doctest.py", line 1267, in __run
    compileflags, 1) in test.globs
  File "<doctest django.contrib.sessions.tests[56]>", line 1, in ?
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-packages/django/contrib/sessions/backends/base.py", line 233, in flush
    self.create()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-packages/django/contrib/sessions/backends/cache.py", line 20, in create
    self.session_key = self._get_new_session_key()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-packages/django/contrib/sessions/backends/base.py", line 137, in _get_new_session_key
    session_key = md5_constructor("%s%s%s%s"
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/random.py", line 188, in randrange
    return int(istart + self._randbelow(width))
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/random.py", line 235, in _randbelow
    r = getrandbits(k)
KeyboardInterrupt

The hanging test seems to be in django.contrib.sessions.tests line 56 or so.

[Alex Gaynor]

I'm not seeing this on r8350, try svn upping(and clearing out all the .pyc files in django) and if it's still an issue reopen.

[Malcolm Tredinnick]

This is a real bug, but, as per ​this thread on django-users, it seems to be very Mac specific in very specific circumstances. In any case, let's call #8314 the tracking ticket for this one and I'll keep that updated with fixes, etc.

(Triagers, close similar tickets as dupes of #8314.)

[jcrocholl]

This problem is different from the crash in the django-users thread, and also different from #8314.

This problem persists after upgrading to [8351]. I have deleted all .pyc files and wiped the installation folder.

This problem is not a Python crash or a stack overflow (but I've experienced both of these too).

Python just keeps running at 100% CPU forever. If I kill it with Ctrl+C, the stack trace always looks like the one I posted originally, except for the exact position of the KeyboardInterrupt. The infinite loop seems to start here:

  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-packages/django/contrib/sessions/backends/base.py", line 236, in flush
    self.create()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-packages/django/contrib/sessions/backends/cache.py", line 22, in create
    self.save(must_create=True)

I'm working on Mac OS X 10.5.4, with the following packages from MacPorts:

memcached @1.2.5_2 (active)
postgresql83 @8.3.1_0 (active)
postgresql83-server @8.3.1_0 (active)
py-memcached @1.40_0 (active)
py-psycopg @1.1.21_1+postgresql83 (active)
py-psycopg2 @2.0.5.1_0 (active)
python24 @2.4.5_0+darwin_9 (active)

[Malcolm Tredinnick]

@jcrocholl: apologies for not appreciating the difference here.

Would you be able to run the tests with --verbosity=2 and work out which test it is hanging on? That should help narrow down where the problem might be.

Also, is there anything special in the settings file you are using, such as setting a particular cache backend? I can't repeat the issue here, but I might not be exercising the right piece of code (plus I'm not on a Mac). So if you can supply that info, I'll try to do it just with eyeballs.

[jcrocholl]

This is the last output of /opt/local/bin/python2.4 manage.py test --verbosity=2 before Python hangs at 100% CPU:

Checking absolute path for fixtures...
Trying absolute path for xml fixture 'initial_data'...
No xml fixture 'initial_data' in absolute path.
Trying absolute path for json fixture 'initial_data'...
No json fixture 'initial_data' in absolute path.
Resetting sequences
Installed 120 object(s) from 4 fixture(s)
test_confirm_complete (django.contrib.auth.tests.views.PasswordResetTest) ... ok
test_confirm_different_passwords (django.contrib.auth.tests.views.PasswordResetTest) ... ok
test_confirm_invalid (django.contrib.auth.tests.views.PasswordResetTest) ... ok
test_confirm_invalid_post (django.contrib.auth.tests.views.PasswordResetTest) ... ok
test_confirm_valid (django.contrib.auth.tests.views.PasswordResetTest) ... ok
Email is sent if a valid email address is provided for password reset ... ok
Error is raised if the provided email address isn't currently registered ... ok
Doctest: django.contrib.auth.tests.__test__.BASIC_TESTS ... ok
Doctest: django.contrib.auth.tests.__test__.FORM_TESTS ... ok
Doctest: django.contrib.auth.tests.__test__.TOKEN_GENERATOR_TESTS ... ok
Doctest: django.contrib.sites.tests ... ok
Doctest: django.contrib.contenttypes.tests ... ok
Doctest: django.contrib.sessions.tests ... 

I already pointed out that this was the hanging test, see the bottom of my first post.

The problem goes away if I don't use memcached (by commenting out the following line in settings.py).

CACHE_BACKEND = 'memcached://127.0.0.1:11211/'

[Alex Gaynor]

Does it occur if you use a different cache backend?

[jcrocholl]

Replying to Alex:

Does it occur if you use a different cache backend?

The problem only occurs with memcached. The test suite works okay with any of the following alternatives:

• No CACHE_BACKEND specified
• CACHE_BACKEND = 'locmem:///'
• CACHE_BACKEND = 'file://cache'

My guess is that memcached fails to delete session keys from the cache.

[jcrocholl]

Okay, here's the simple solution: The memcached daemon should be running when manage.py test is started.

The problem was that memcached was not running on my system. The new session cache code will try to create a new session, but the cache backend will return False when it attempts to save the session to the cache. Then the session code assumes that this session key already exists, and raises CreateError. Then it generates a new session key to try again, and again...

The better solution would be to raise an exception if memcached is selected but unavailable, rather than the current infinite loop. Maybe I should write a patch to limit the number of attempts to 10 or 100? Here's the current implementation:

browser://django/trunk/django/contrib/sessions/backends/cache.py (line 18)

def create(self):
    while True:
        self.session_key = self._get_new_session_key()
        try:
            self.save(must_create=True)
        except CreateError:
            continue
        self.modified = True
        return

[Malcolm Tredinnick]

Need to think about what to do with caching when the cache isn't available. Depending on that, it will decide what to do here. Nice to hear it isn't a fundamental problem with the code, however. Thanks for chasing that down to the end. We'll do something about the error handling, but I'm not sure what the right solution is yet.

[Malcolm Tredinnick]

(In [8381]) Made a few small tweaks to reduce persistent storage accesses in the session
backend. Refs #8311, although doesn't fix the problem there.

[Malcolm Tredinnick]

(In [8620]) Fixed #8311 -- Avoid an infinite loop with session key generation when using
the cache backend and memcached goes away (or is not running).

[Jacob]

Milestone 1.0 deleted