Duplicate entry sessions error

[hash.3g@…]

Hi, Sometimes we receive "duplicate entry" error from django sessions. See Traceback.

Traceback (most recent call last):

File "/usr/local/lib/python2.6/dist-packages/django/core/handlers/base.py", line 178, in get_response
  response = middleware_method(request, response)
File "/usr/local/lib/python2.6/dist-packages/django/contrib/sessions/middleware.py", line 36, in process_response
  request.session.save()
File "/usr/local/lib/python2.6/dist-packages/django/contrib/sessions/backends/db.py", line 63, in save
  obj.save(force_insert=must_create, using=using)
File "/usr/local/lib/python2.6/dist-packages/django/db/models/base.py", line 460, in save
  self.save_base(using=using, force_insert=force_insert, force_update=force_update)
File "/usr/local/lib/python2.6/dist-packages/django/db/models/base.py", line 553, in save_base
  result = manager._insert(values, return_id=update_pk, using=using)
File "/usr/local/lib/python2.6/dist-packages/django/db/models/manager.py", line 195, in _insert
  return insert_query(self.model, values, **kwargs)
File "/usr/local/lib/python2.6/dist-packages/django/db/models/query.py", line 1436, in insert_query
  return query.get_compiler(using=using).execute_sql(return_id)
File "/usr/local/lib/python2.6/dist-packages/django/db/models/sql/compiler.py", line 791, in execute_sql
  cursor = super(SQLInsertCompiler, self).execute_sql(None)
File "/usr/local/lib/python2.6/dist-packages/django/db/models/sql/compiler.py", line 735, in execute_sql
  cursor.execute(sql, params)
File "/usr/local/lib/python2.6/dist-packages/django/db/backends/mysql/base.py", line 86, in execute
  return self.cursor.execute(query, args)
File "/usr/lib/pymodules/python2.6/MySQLdb/cursors.py", line 166, in execute
  self.errorhandler(self, exc, value)
File "/usr/lib/pymodules/python2.6/MySQLdb/connections.py", line 35, in defaulterrorhandler
  raise errorclass, errorvalue
IntegrityError: (1062, “Duplicate entry ‘e110c3788ccb21c76865285446102338’ for key 'PRIMARY'”)

Periodically we do cleaning of out-of-date sessions, but it does not help from current error.

[Aymeric Augustin]

From code inspection, I don't see how this can happen: it the call to obj.save(force_insert=must_create, using=using), must_create is False, so the session should just be updated if it already exists. Here, you're hitting a naked raise, which means that the original author thought this should never happen.

I can only imagine one cause for a race condition: if two requests running in two separate processes open a transaction, then commit: under some isolation levels, this may fail.

Could you provide more information about your database setup so we can try to reproduce this?

[hash.3g@…]

Here's my settings for project, and I attached my.cnf file if needed

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.mysql', # Add 'postgresql_psycopg2', 'postgresql', 'mysql', 'sqlite3' or 'oracle'.
        'NAME': 'dou',                      # Or path to database file if using sqlite3.
        'USER': 'hash',                      # Not used with sqlite3.
        'PASSWORD': '*******',                  # Not used with sqlite3.
        'HOST': '',                      # Set to empty string for localhost. Not used with sqlite3.
        'PORT': '',                      # Set to empty string for default. Not used with sqlite3.
        'OPTIONS': {"init_command": "SET storage_engine=INNODB;"}
    }
}

[Aymeric Augustin]

Which version of MySQL are you using?

[anonymous]

I am using 5.1.49

[Aymeric Augustin]

It's really hard to tell what happens here :/

I will accept the ticket because the traceback proves the error.

[sidmitra.del@…]

I think i can duplicate the error. I have the same Mysql version(with INNODB).

File "/home/ubuntu/.virtualenvs/app/lib/python2.6/site-packages/django/core/handlers/base.py", line 178, in get_response
  response = middleware_method(request, response)
 File "/home/ubuntu/.virtualenvs/app/lib/python2.6/site-packages/django/contrib/sessions/middleware.py", line 36, in process_response
  request.session.save()
 File "/home/ubuntu/.virtualenvs/app/lib/python2.6/site-packages/django/contrib/sessions/backends/cached_db.py", line 31, in save
  super(SessionStore, self).save(must_create)
 File "/home/ubuntu/.virtualenvs/app/lib/python2.6/site-packages/django/contrib/sessions/backends/db.py", line 63, in save
  obj.save(force_insert=must_create, using=using)
 File "/home/ubuntu/.virtualenvs/app/lib/python2.6/site-packages/django/db/models/base.py", line 460, in save
  self.save_base(using=using, force_insert=force_insert, force_update=force_update)
 File "/home/ubuntu/.virtualenvs/app/lib/python2.6/site-packages/django/db/models/base.py", line 553, in save_base
  result = manager._insert(values, return_id=update_pk, using=using)
 File "/home/ubuntu/.virtualenvs/app/lib/python2.6/site-packages/django/db/models/manager.py", line 195, in _insert
  return insert_query(self.model, values, **kwargs)
 File "/home/ubuntu/.virtualenvs/app/lib/python2.6/site-packages/django/db/models/query.py", line 1436, in insert_query
  return query.get_compiler(using=using).execute_sql(return_id)
 File "/home/ubuntu/.virtualenvs/app/lib/python2.6/site-packages/johnny/cache.py", line 344, in newfun
  ret = original(cls, *args, **kwargs)
 File "/home/ubuntu/.virtualenvs/app/lib/python2.6/site-packages/django/db/models/sql/compiler.py", line 791, in execute_sql
  cursor = super(SQLInsertCompiler, self).execute_sql(None)
 File "/home/ubuntu/.virtualenvs/app/lib/python2.6/site-packages/johnny/cache.py", line 293, in newfun
  return original(cls, *args, **kwargs)
 File "/home/ubuntu/.virtualenvs/app/lib/python2.6/site-packages/django/db/models/sql/compiler.py", line 735, in execute_sql
  cursor.execute(sql, params)
 File "/home/ubuntu/.virtualenvs/app/lib/python2.6/site-packages/django/db/backends/mysql/base.py", line 86, in execute
  return self.cursor.execute(query, args)
 File "/home/ubuntu/.virtualenvs/app/lib/python2.6/site-packages/MySQLdb/cursors.py", line 174, in execute
  self.errorhandler(self, exc, value)
 File "/home/ubuntu/.virtualenvs/app/lib/python2.6/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
  raise errorclass, errorvalue
 IntegrityError: (1062, “Duplicate entry ‘e4d75e04ee5995bd2e03d25a697a00d1’ for key 'PRIMARY'”)
 

Here're my session settings:

SESSION_ENGINE                      = 'django.contrib.sessions.backends.cached_db'
SESSION_COOKIE_AGE                  = 2 * 60 * 60
SESSION_EXPIRE_AT_BROWSER_CLOSE     = True

Note: I use johnnny-cache alongside

[Aymeric Augustin]

#18344 may describe the same problem.

[Aymeric Augustin]

This is suspiciously similar to #18557.

The conclusion ​on the mailing list was to use the READ COMMITTED isolation level (recommended), or database-level autocommit of every query (not recommended).

MySQL defaults to REPEATABLE READ, which appears to make such situations intractable.

[Hans Andersen]

This is still a problem for Django 1.5.1

  File "/usr/local/lib/python2.7/dist-packages/Django-1.5.1-py2.7.egg/django/core/handlers/base.py", line 187, in get_response
    response = middleware_method(request, response)

  File "/usr/local/lib/python2.7/dist-packages/Django-1.5.1-py2.7.egg/django/contrib/sessions/middleware.py", line 38, in process_response
    request.session.save()

  File "/usr/local/lib/python2.7/dist-packages/Django-1.5.1-py2.7.egg/django/contrib/sessions/backends/db.py", line 57, in save
    obj.save(force_insert=must_create, using=using)

  File "/usr/local/lib/python2.7/dist-packages/Django-1.5.1-py2.7.egg/django/db/models/base.py", line 546, in save
    force_update=force_update, update_fields=update_fields)

  File "/usr/local/lib/python2.7/dist-packages/Django-1.5.1-py2.7.egg/django/db/models/base.py", line 650, in save_base
    result = manager._insert([self], fields=fields, return_id=update_pk, using=using, raw=raw)

  File "/usr/local/lib/python2.7/dist-packages/Django-1.5.1-py2.7.egg/django/db/models/manager.py", line 215, in _insert
    return insert_query(self.model, objs, fields, **kwargs)

  File "/usr/local/lib/python2.7/dist-packages/Django-1.5.1-py2.7.egg/django/db/models/query.py", line 1661, in insert_query
    return query.get_compiler(using=using).execute_sql(return_id)

  File "/usr/local/lib/python2.7/dist-packages/Django-1.5.1-py2.7.egg/django/db/models/sql/compiler.py", line 937, in execute_sql
    cursor.execute(sql, params)

  File "/usr/local/lib/python2.7/dist-packages/Django-1.5.1-py2.7.egg/django/db/backends/mysql/base.py", line 122, in execute
    six.reraise(utils.IntegrityError, utils.IntegrityError(*tuple(e.args)), sys.exc_info()[2])

  File "/usr/local/lib/python2.7/dist-packages/Django-1.5.1-py2.7.egg/django/db/backends/mysql/base.py", line 120, in execute
    return self.cursor.execute(query, args)

  File "/usr/lib/python2.7/dist-packages/MySQLdb/cursors.py", line 174, in execute
    self.errorhandler(self, exc, value)

  File "/usr/lib/python2.7/dist-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
    raise errorclass, errorvalue

IntegrityError: (1062, "Duplicate entry '2v4qsr4c75qbdkn27u3dypfig46fca2z' for key 'PRIMARY'")

[anonymous]

Replying to jokerejoker:

This is still a problem for Django 1.5.1

There was no claim above that this would be "fixed" by Django 1.5.1, since it's suspected the underlying problem is MySQL's default "reapeatable read" transaction isolation level. There is no way (aside from including a commit within get_and_create itself, which is completely unacceptable) to "fix" this inside of Django. The fix is to change your transaction isolation level to "read committed".

[Dylan Young]

I'm not sure where to put this, as this bug has been marked duplicate of another bug, marked duplicate of another bug...(most of which seem to have been closed despite not being fixed), so I'll place this here.

The common conclusion seems to be that this behaviour comes from MySQL, but this surfaces using PostgreSQL as well:

File "/home/www/env/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 223, in get_response
  response = middleware_method(request, response)
File "/home/www/env/local/lib/python2.7/site-packages/django/contrib/sessions/middleware.py", line 50, in process_response
  request.session.save()
File "/home/www/env/local/lib/python2.7/site-packages/django/contrib/sessions/backends/cached_db.py", line 64, in save
  super(SessionStore, self).save(must_create)
File "/home/www/env/local/lib/python2.7/site-packages/django/contrib/sessions/backends/db.py", line 65, in save
  obj.save(force_insert=must_create, using=using)
File "/home/www/env/local/lib/python2.7/site-packages/django/db/models/base.py", line 734, in save
  force_update=force_update, update_fields=update_fields)
File "/home/www/env/local/lib/python2.7/site-packages/django/db/models/base.py", line 762, in save_base
  updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)
File "/home/www/env/local/lib/python2.7/site-packages/django/db/models/base.py", line 846, in _save_table
  result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)
File "/home/www/env/local/lib/python2.7/site-packages/django/db/models/base.py", line 885, in _do_insert
  using=using, raw=raw)
File "/home/www/env/local/lib/python2.7/site-packages/django/db/models/manager.py", line 127, in manager_method
  return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/home/www/env/local/lib/python2.7/site-packages/django/db/models/query.py", line 920, in _insert
  return query.get_compiler(using=using).execute_sql(return_id)
File "/home/www/env/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 974, in execute_sql
  cursor.execute(sql, params)
File "/home/www/env/local/lib/python2.7/site-packages/django/db/backends/utils.py", line 64, in execute
  return self.cursor.execute(sql, params)
File "/home/www/env/local/lib/python2.7/site-packages/django/db/utils.py", line 98, in __exit__
  six.reraise(dj_exc_type, dj_exc_value, traceback)
File "/home/www/env/local/lib/python2.7/site-packages/django/db/backends/utils.py", line 64, in execute
  return self.cursor.execute(sql, params)
IntegrityError: duplicate key value violates unique constraint “django_session_pkey” DETAIL: Key (session_key)=(34j86lxxh4alwelayjq5i77e8lppwdhp) already exists.

This is with the cached_db session storage and no special psycopg options.

Thoughts?

[Tim Graham]

Providing only a traceback does not help much (also none of the previous discussion references PostgreSQL, so reopening doesn't seem appropriate). Please open a new ticket with complete details to reproduce the issue and make sure you can reproduce it with the latest version of Django rather than 1.8.

[Dylan Young]

What information do you need? The stack trace is entirely within django, so I don't know what else I can provide you. A quick check of Stack Overflow says that this is a common, if sporadic, issue.

I mentioned all relevant configurations that came to mind; please request more info if you need it: postgresql version? psycopg version? Not sure what you need. Please elaborate before I open another issue to have it closed immediately,

In any case, I'll see if I can pinpoint the problem myself if I get time. If you can get me the information I need to get you the information you need, I'll open a new issue.

It is a race condition, so I cannot spend the time reproducing it on 1.9 or later (that would be crazy).

If you can point me to a commit that you think may have solved the issue in 1.9, or later, please do.