Opened 7 years ago

Closed 3 months ago

Last modified 3 months ago

#16484 closed Bug (duplicate)

Duplicate entry sessions error

Reported by: hash.3g@… Owned by: nobody
Component: contrib.sessions Version: 1.8
Severity: Normal Keywords:
Cc: Triage Stage: Accepted
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

Hi, Sometimes we receive "duplicate entry" error from django sessions. See Traceback.

Traceback (most recent call last):

File "/usr/local/lib/python2.6/dist-packages/django/core/handlers/base.py", line 178, in get_response
  response = middleware_method(request, response)
File "/usr/local/lib/python2.6/dist-packages/django/contrib/sessions/middleware.py", line 36, in process_response
  request.session.save()
File "/usr/local/lib/python2.6/dist-packages/django/contrib/sessions/backends/db.py", line 63, in save
  obj.save(force_insert=must_create, using=using)
File "/usr/local/lib/python2.6/dist-packages/django/db/models/base.py", line 460, in save
  self.save_base(using=using, force_insert=force_insert, force_update=force_update)
File "/usr/local/lib/python2.6/dist-packages/django/db/models/base.py", line 553, in save_base
  result = manager._insert(values, return_id=update_pk, using=using)
File "/usr/local/lib/python2.6/dist-packages/django/db/models/manager.py", line 195, in _insert
  return insert_query(self.model, values, **kwargs)
File "/usr/local/lib/python2.6/dist-packages/django/db/models/query.py", line 1436, in insert_query
  return query.get_compiler(using=using).execute_sql(return_id)
File "/usr/local/lib/python2.6/dist-packages/django/db/models/sql/compiler.py", line 791, in execute_sql
  cursor = super(SQLInsertCompiler, self).execute_sql(None)
File "/usr/local/lib/python2.6/dist-packages/django/db/models/sql/compiler.py", line 735, in execute_sql
  cursor.execute(sql, params)
File "/usr/local/lib/python2.6/dist-packages/django/db/backends/mysql/base.py", line 86, in execute
  return self.cursor.execute(query, args)
File "/usr/lib/pymodules/python2.6/MySQLdb/cursors.py", line 166, in execute
  self.errorhandler(self, exc, value)
File "/usr/lib/pymodules/python2.6/MySQLdb/connections.py", line 35, in defaulterrorhandler
  raise errorclass, errorvalue
IntegrityError: (1062, “Duplicate entry ‘e110c3788ccb21c76865285446102338’ for key 'PRIMARY'”)

Periodically we do cleaning of out-of-date sessions, but it does not help from current error.

Attachments (1)

my.cnf (4.3 KB) - added by anonymous 7 years ago.

Download all attachments as: .zip

Change History (14)

comment:1 Changed 7 years ago by Aymeric Augustin

Resolution: needsinfo
Status: newclosed

From code inspection, I don't see how this can happen: in the call to obj.save(force_insert=must_create, using=using), must_create is False, so the session should just be updated if it already exists. Here, you're hitting a naked raise, which means that the original author thought this should never happen.

I can only imagine one cause for a race condition: if two requests running in two separate processes open a transaction, then commit: under some isolation levels, this may fail.

Could you provide more information about your database setup so we can try to reproduce this?

Last edited 7 years ago by Aymeric Augustin (previous) (diff)

Changed 7 years ago by anonymous

Attachment: my.cnf added

comment:2 Changed 7 years ago by hash.3g@…

Resolution: needsinfo
Status: closedreopened

Here's my settings for project, and I attached my.cnf file if needed

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.mysql', # Add 'postgresql_psycopg2', 'postgresql', 'mysql', 'sqlite3' or 'oracle'.
        'NAME': 'dou',                      # Or path to database file if using sqlite3.
        'USER': 'hash',                      # Not used with sqlite3.
        'PASSWORD': '*******',                  # Not used with sqlite3.
        'HOST': '',                      # Set to empty string for localhost. Not used with sqlite3.
        'PORT': '',                      # Set to empty string for default. Not used with sqlite3.
        'OPTIONS': {"init_command": "SET storage_engine=INNODB;"}
    }
}

comment:3 Changed 7 years ago by Aymeric Augustin

Which version of MySQL are you using?

comment:4 Changed 7 years ago by anonymous

I am using 5.1.49

comment:5 Changed 7 years ago by Aymeric Augustin

Triage Stage: UnreviewedAccepted

It's really hard to tell what happens here :/

I will accept the ticket because the traceback proves the error.

comment:6 Changed 6 years ago by sidmitra.del@…

I think i can duplicate the error. I have the same Mysql version(with INNODB).

File "/home/ubuntu/.virtualenvs/app/lib/python2.6/site-packages/django/core/handlers/base.py", line 178, in get_response
  response = middleware_method(request, response)
 File "/home/ubuntu/.virtualenvs/app/lib/python2.6/site-packages/django/contrib/sessions/middleware.py", line 36, in process_response
  request.session.save()
 File "/home/ubuntu/.virtualenvs/app/lib/python2.6/site-packages/django/contrib/sessions/backends/cached_db.py", line 31, in save
  super(SessionStore, self).save(must_create)
 File "/home/ubuntu/.virtualenvs/app/lib/python2.6/site-packages/django/contrib/sessions/backends/db.py", line 63, in save
  obj.save(force_insert=must_create, using=using)
 File "/home/ubuntu/.virtualenvs/app/lib/python2.6/site-packages/django/db/models/base.py", line 460, in save
  self.save_base(using=using, force_insert=force_insert, force_update=force_update)
 File "/home/ubuntu/.virtualenvs/app/lib/python2.6/site-packages/django/db/models/base.py", line 553, in save_base
  result = manager._insert(values, return_id=update_pk, using=using)
 File "/home/ubuntu/.virtualenvs/app/lib/python2.6/site-packages/django/db/models/manager.py", line 195, in _insert
  return insert_query(self.model, values, **kwargs)
 File "/home/ubuntu/.virtualenvs/app/lib/python2.6/site-packages/django/db/models/query.py", line 1436, in insert_query
  return query.get_compiler(using=using).execute_sql(return_id)
 File "/home/ubuntu/.virtualenvs/app/lib/python2.6/site-packages/johnny/cache.py", line 344, in newfun
  ret = original(cls, *args, **kwargs)
 File "/home/ubuntu/.virtualenvs/app/lib/python2.6/site-packages/django/db/models/sql/compiler.py", line 791, in execute_sql
  cursor = super(SQLInsertCompiler, self).execute_sql(None)
 File "/home/ubuntu/.virtualenvs/app/lib/python2.6/site-packages/johnny/cache.py", line 293, in newfun
  return original(cls, *args, **kwargs)
 File "/home/ubuntu/.virtualenvs/app/lib/python2.6/site-packages/django/db/models/sql/compiler.py", line 735, in execute_sql
  cursor.execute(sql, params)
 File "/home/ubuntu/.virtualenvs/app/lib/python2.6/site-packages/django/db/backends/mysql/base.py", line 86, in execute
  return self.cursor.execute(query, args)
 File "/home/ubuntu/.virtualenvs/app/lib/python2.6/site-packages/MySQLdb/cursors.py", line 174, in execute
  self.errorhandler(self, exc, value)
 File "/home/ubuntu/.virtualenvs/app/lib/python2.6/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
  raise errorclass, errorvalue
 IntegrityError: (1062, “Duplicate entry ‘e4d75e04ee5995bd2e03d25a697a00d1’ for key 'PRIMARY'”)
 

Here're my session settings:

SESSION_ENGINE                      = 'django.contrib.sessions.backends.cached_db'
SESSION_COOKIE_AGE                  = 2 * 60 * 60
SESSION_EXPIRE_AT_BROWSER_CLOSE     = True

Note: I use johnnny-cache alongside

comment:7 Changed 6 years ago by Aymeric Augustin

#18344 may describe the same problem.

comment:8 Changed 5 years ago by Aymeric Augustin

Resolution: duplicate
Status: reopenedclosed

This is suspiciously similar to #18557.

The conclusion on the mailing list was to use the READ COMMITTED isolation level (recommended), or database-level autocommit of every query (not recommended).

MySQL defaults to REPEATABLE READ, which appears to make such situations intractable.

comment:9 Changed 5 years ago by Hans Andersen

This is still a problem for Django 1.5.1

  File "/usr/local/lib/python2.7/dist-packages/Django-1.5.1-py2.7.egg/django/core/handlers/base.py", line 187, in get_response
    response = middleware_method(request, response)

  File "/usr/local/lib/python2.7/dist-packages/Django-1.5.1-py2.7.egg/django/contrib/sessions/middleware.py", line 38, in process_response
    request.session.save()

  File "/usr/local/lib/python2.7/dist-packages/Django-1.5.1-py2.7.egg/django/contrib/sessions/backends/db.py", line 57, in save
    obj.save(force_insert=must_create, using=using)

  File "/usr/local/lib/python2.7/dist-packages/Django-1.5.1-py2.7.egg/django/db/models/base.py", line 546, in save
    force_update=force_update, update_fields=update_fields)

  File "/usr/local/lib/python2.7/dist-packages/Django-1.5.1-py2.7.egg/django/db/models/base.py", line 650, in save_base
    result = manager._insert([self], fields=fields, return_id=update_pk, using=using, raw=raw)

  File "/usr/local/lib/python2.7/dist-packages/Django-1.5.1-py2.7.egg/django/db/models/manager.py", line 215, in _insert
    return insert_query(self.model, objs, fields, **kwargs)

  File "/usr/local/lib/python2.7/dist-packages/Django-1.5.1-py2.7.egg/django/db/models/query.py", line 1661, in insert_query
    return query.get_compiler(using=using).execute_sql(return_id)

  File "/usr/local/lib/python2.7/dist-packages/Django-1.5.1-py2.7.egg/django/db/models/sql/compiler.py", line 937, in execute_sql
    cursor.execute(sql, params)

  File "/usr/local/lib/python2.7/dist-packages/Django-1.5.1-py2.7.egg/django/db/backends/mysql/base.py", line 122, in execute
    six.reraise(utils.IntegrityError, utils.IntegrityError(*tuple(e.args)), sys.exc_info()[2])

  File "/usr/local/lib/python2.7/dist-packages/Django-1.5.1-py2.7.egg/django/db/backends/mysql/base.py", line 120, in execute
    return self.cursor.execute(query, args)

  File "/usr/lib/python2.7/dist-packages/MySQLdb/cursors.py", line 174, in execute
    self.errorhandler(self, exc, value)

  File "/usr/lib/python2.7/dist-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
    raise errorclass, errorvalue

IntegrityError: (1062, "Duplicate entry '2v4qsr4c75qbdkn27u3dypfig46fca2z' for key 'PRIMARY'")

comment:10 in reply to:  9 Changed 5 years ago by anonymous

Replying to jokerejoker:

This is still a problem for Django 1.5.1

There was no claim above that this would be "fixed" by Django 1.5.1, since it's suspected the underlying problem is MySQL's default "reapeatable read" transaction isolation level. There is no way (aside from including a commit within get_and_create itself, which is completely unacceptable) to "fix" this inside of Django. The fix is to change your transaction isolation level to "read committed".

comment:11 Changed 3 months ago by Dylan Young

Resolution: duplicate
Status: closednew
Version: 1.31.8

I'm not sure where to put this, as this bug has been marked duplicate of another bug, marked duplicate of another bug...(most of which seem to have been closed despite not being fixed), so I'll place this here.

The common conclusion seems to be that this behaviour comes from MySQL, but this surfaces using PostgreSQL as well:

File "/home/www/env/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 223, in get_response
  response = middleware_method(request, response)
File "/home/www/env/local/lib/python2.7/site-packages/django/contrib/sessions/middleware.py", line 50, in process_response
  request.session.save()
File "/home/www/env/local/lib/python2.7/site-packages/django/contrib/sessions/backends/cached_db.py", line 64, in save
  super(SessionStore, self).save(must_create)
File "/home/www/env/local/lib/python2.7/site-packages/django/contrib/sessions/backends/db.py", line 65, in save
  obj.save(force_insert=must_create, using=using)
File "/home/www/env/local/lib/python2.7/site-packages/django/db/models/base.py", line 734, in save
  force_update=force_update, update_fields=update_fields)
File "/home/www/env/local/lib/python2.7/site-packages/django/db/models/base.py", line 762, in save_base
  updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)
File "/home/www/env/local/lib/python2.7/site-packages/django/db/models/base.py", line 846, in _save_table
  result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)
File "/home/www/env/local/lib/python2.7/site-packages/django/db/models/base.py", line 885, in _do_insert
  using=using, raw=raw)
File "/home/www/env/local/lib/python2.7/site-packages/django/db/models/manager.py", line 127, in manager_method
  return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/home/www/env/local/lib/python2.7/site-packages/django/db/models/query.py", line 920, in _insert
  return query.get_compiler(using=using).execute_sql(return_id)
File "/home/www/env/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 974, in execute_sql
  cursor.execute(sql, params)
File "/home/www/env/local/lib/python2.7/site-packages/django/db/backends/utils.py", line 64, in execute
  return self.cursor.execute(sql, params)
File "/home/www/env/local/lib/python2.7/site-packages/django/db/utils.py", line 98, in __exit__
  six.reraise(dj_exc_type, dj_exc_value, traceback)
File "/home/www/env/local/lib/python2.7/site-packages/django/db/backends/utils.py", line 64, in execute
  return self.cursor.execute(sql, params)
IntegrityError: duplicate key value violates unique constraint “django_session_pkey” DETAIL: Key (session_key)=(34j86lxxh4alwelayjq5i77e8lppwdhp) already exists.

This is with the cached_db session storage and no special psycopg options.

Thoughts?

comment:12 Changed 3 months ago by Tim Graham

Resolution: duplicate
Status: newclosed

Providing only a traceback does not help much (also none of the previous discussion references PostgreSQL, so reopening doesn't seem appropriate). Please open a new ticket with complete details to reproduce the issue and make sure you can reproduce it with the latest version of Django rather than 1.8.

comment:13 Changed 3 months ago by Dylan Young

What information do you need? The stack trace is entirely within django, so I don't know what else I can provide you. A quick check of Stack Overflow says that this is a common, if sporadic, issue.

I mentioned all relevant configurations that came to mind; please request more info if you need it: postgresql version? psycopg version? Not sure what you need. Please elaborate before I open another issue to have it closed immediately,

In any case, I'll see if I can pinpoint the problem myself if I get time. If you can get me the information I need to get you the information you need, I'll open a new issue.

It is a race condition, so I cannot spend the time reproducing it on 1.9 or later (that would be crazy).

If you can point me to a commit that you think may have solved the issue in 1.9, or later, please do.

Note: See TracTickets for help on using tickets.
Back to Top