Opened 16 years ago

Closed 16 years ago

Last modified 12 years ago

#8311 closed (fixed)

Changeset 8342 makes test suite hang if memcached is not running

Reported by: jcrocholl Owned by: nobody
Component: contrib.sessions Version: dev
Severity: Keywords:
Cc: johann@…, Trevor Caira Triage Stage: Accepted
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

Up to changeset [8341] my project works okay. After upgrading to [8342], manage.py test hangs at 100% CPU, with the following Traceback after Ctrl+C:

............^CTraceback (most recent call last):
  ... (beginning of doctest stack trace omitted) ...
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/unittest.py", line 428, in __call__
    return self.run(*args, **kwds)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/unittest.py", line 424, in run
    test(result)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/unittest.py", line 281, in __call__
    return self.run(*args, **kwds)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/unittest.py", line 260, in run
    testMethod()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-packages/django/test/_doctest.py", line 2174, in runTest
    failures, tries = runner.run(
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-packages/django/test/_doctest.py", line 1403, in run
    return self.__run(test, compileflags, out)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-packages/django/test/_doctest.py", line 1267, in __run
    compileflags, 1) in test.globs
  File "<doctest django.contrib.sessions.tests[56]>", line 1, in ?
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-packages/django/contrib/sessions/backends/base.py", line 233, in flush
    self.create()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-packages/django/contrib/sessions/backends/cache.py", line 20, in create
    self.session_key = self._get_new_session_key()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-packages/django/contrib/sessions/backends/base.py", line 137, in _get_new_session_key
    session_key = md5_constructor("%s%s%s%s"
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/random.py", line 188, in randrange
    return int(istart + self._randbelow(width))
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/random.py", line 235, in _randbelow
    r = getrandbits(k)
KeyboardInterrupt

The hanging test seems to be in django.contrib.sessions.tests line 56 or so.

Change History (15)

comment:1 by Alex Gaynor, 16 years ago

Resolution: worksforme
Status: newclosed

I'm not seeing this on r8350, try svn upping(and clearing out all the .pyc files in django) and if it's still an issue reopen.

comment:2 by Malcolm Tredinnick, 16 years ago

This is a real bug, but, as per this thread on django-users, it seems to be very Mac specific in very specific circumstances. In any case, let's call #8314 the tracking ticket for this one and I'll keep that updated with fixes, etc.

(Triagers, close similar tickets as dupes of #8314.)

comment:3 by jcrocholl, 16 years ago

Resolution: worksforme
Status: closedreopened

This problem is different from the crash in the django-users thread, and also different from #8314.

This problem persists after upgrading to [8351]. I have deleted all .pyc files and wiped the installation folder.

This problem is not a Python crash or a stack overflow (but I've experienced both of these too).

Python just keeps running at 100% CPU forever. If I kill it with Ctrl+C, the stack trace always looks like the one I posted originally, except for the exact position of the KeyboardInterrupt. The infinite loop seems to start here:

  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-packages/django/contrib/sessions/backends/base.py", line 236, in flush
    self.create()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-packages/django/contrib/sessions/backends/cache.py", line 22, in create
    self.save(must_create=True)

I'm working on Mac OS X 10.5.4, with the following packages from MacPorts:

memcached @1.2.5_2 (active)
postgresql83 @8.3.1_0 (active)
postgresql83-server @8.3.1_0 (active)
py-memcached @1.40_0 (active)
py-psycopg @1.1.21_1+postgresql83 (active)
py-psycopg2 @2.0.5.1_0 (active)
python24 @2.4.5_0+darwin_9 (active)

comment:4 by Malcolm Tredinnick, 16 years ago

milestone: 1.0 beta
Triage Stage: UnreviewedAccepted

@jcrocholl: apologies for not appreciating the difference here.

Would you be able to run the tests with --verbosity=2 and work out which test it is hanging on? That should help narrow down where the problem might be.

Also, is there anything special in the settings file you are using, such as setting a particular cache backend? I can't repeat the issue here, but I might not be exercising the right piece of code (plus I'm not on a Mac). So if you can supply that info, I'll try to do it just with eyeballs.

comment:5 by Alex Gaynor, 16 years ago

milestone: 1.0 beta1.0

in reply to:  4 comment:6 by jcrocholl, 16 years ago

Summary: Changeset 8342 makes test suite hangChangeset 8342 makes test suite hang when using memcached

This is the last output of /opt/local/bin/python2.4 manage.py test --verbosity=2 before Python hangs at 100% CPU:

Checking absolute path for fixtures...
Trying absolute path for xml fixture 'initial_data'...
No xml fixture 'initial_data' in absolute path.
Trying absolute path for json fixture 'initial_data'...
No json fixture 'initial_data' in absolute path.
Resetting sequences
Installed 120 object(s) from 4 fixture(s)
test_confirm_complete (django.contrib.auth.tests.views.PasswordResetTest) ... ok
test_confirm_different_passwords (django.contrib.auth.tests.views.PasswordResetTest) ... ok
test_confirm_invalid (django.contrib.auth.tests.views.PasswordResetTest) ... ok
test_confirm_invalid_post (django.contrib.auth.tests.views.PasswordResetTest) ... ok
test_confirm_valid (django.contrib.auth.tests.views.PasswordResetTest) ... ok
Email is sent if a valid email address is provided for password reset ... ok
Error is raised if the provided email address isn't currently registered ... ok
Doctest: django.contrib.auth.tests.__test__.BASIC_TESTS ... ok
Doctest: django.contrib.auth.tests.__test__.FORM_TESTS ... ok
Doctest: django.contrib.auth.tests.__test__.TOKEN_GENERATOR_TESTS ... ok
Doctest: django.contrib.sites.tests ... ok
Doctest: django.contrib.contenttypes.tests ... ok
Doctest: django.contrib.sessions.tests ... 

I already pointed out that this was the hanging test, see the bottom of my first post.

The problem goes away if I don't use memcached (by commenting out the following line in settings.py).

CACHE_BACKEND = 'memcached://127.0.0.1:11211/'

comment:7 by Alex Gaynor, 16 years ago

Does it occur if you use a different cache backend?

in reply to:  7 comment:8 by jcrocholl, 16 years ago

Replying to Alex:

Does it occur if you use a different cache backend?

The problem only occurs with memcached. The test suite works okay with any of the following alternatives:

  • No CACHE_BACKEND specified
  • CACHE_BACKEND = 'locmem:///'
  • CACHE_BACKEND = 'file://cache'

My guess is that memcached fails to delete session keys from the cache.

comment:9 by jcrocholl, 16 years ago

Cc: johann@… added
Summary: Changeset 8342 makes test suite hang when using memcachedChangeset 8342 makes test suite hang if memcached is not running

Okay, here's the simple solution: The memcached daemon should be running when manage.py test is started.

The problem was that memcached was not running on my system. The new session cache code will try to create a new session, but the cache backend will return False when it attempts to save the session to the cache. Then the session code assumes that this session key already exists, and raises CreateError. Then it generates a new session key to try again, and again...

The better solution would be to raise an exception if memcached is selected but unavailable, rather than the current infinite loop. Maybe I should write a patch to limit the number of attempts to 10 or 100? Here's the current implementation:

browser://django/trunk/django/contrib/sessions/backends/cache.py (line 18)

def create(self):
    while True:
        self.session_key = self._get_new_session_key()
        try:
            self.save(must_create=True)
        except CreateError:
            continue
        self.modified = True
        return

comment:10 by Malcolm Tredinnick, 16 years ago

Triage Stage: AcceptedDesign decision needed

Need to think about what to do with caching when the cache isn't available. Depending on that, it will decide what to do here. Nice to hear it isn't a fundamental problem with the code, however. Thanks for chasing that down to the end. We'll do something about the error handling, but I'm not sure what the right solution is yet.

comment:11 by Malcolm Tredinnick, 16 years ago

(In [8381]) Made a few small tweaks to reduce persistent storage accesses in the session
backend. Refs #8311, although doesn't fix the problem there.

comment:12 by Jacob, 16 years ago

Triage Stage: Design decision neededAccepted

comment:13 by Trevor Caira, 16 years ago

Cc: Trevor Caira added

comment:14 by Malcolm Tredinnick, 16 years ago

Resolution: fixed
Status: reopenedclosed

(In [8620]) Fixed #8311 -- Avoid an infinite loop with session key generation when using
the cache backend and memcached goes away (or is not running).

comment:15 by Jacob, 12 years ago

milestone: 1.0

Milestone 1.0 deleted

Note: See TracTickets for help on using tickets.
Back to Top