Opened 6 years ago

Closed 4 months ago

#12118 closed New feature (fixed)

in-memory test database does not work with threads

Reported by: bluebird75 Owned by: coldmind
Component: Database layer (models, ORM) Version: master
Severity: Normal Keywords: threads
Cc: Triage Stage: Accepted
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: yes
Easy pickings: no UI/UX: no

Description

When using the test configuration of the DB with XXX and accessing the DB from another thread, it fails miserably.

Using this example script on the Poll tutorial:

import os                                                                       
os.environ[ 'DJANGO_SETTINGS_MODULE' ] = 'settings'                             
import settings                                                                 
import datetime, threading                                                      
                                                                                
#django  stuff                                                                  
from polls.models import *                                                      
from django.core.mail import mail_admins                                        
from django.test.utils import *                                                 
from django.db import connection                                                
                                                                                
                                                                                
def create_object():                                                            
    print 'Creating Poll'                                                       
    p = Poll()                                                                  
    p.question = "What's up doc ?"                                              
    p.pub_date = datetime.date.today()                                          
    p.save()                                                                    
    print 'Poll object saved. Id: %d' % p.id                                    
                                                                                
                                                                                
WITH_THREAD = False                                                             
                                                                                
if __name__ == '__main__':                                                      
    setup_test_environment()                                                    
    old_db_name = settings.DATABASE_NAME                                        
    new_db_name = connection.creation.create_test_db(verbosity=1)               
    print 'New DATABASE:', new_db_name                                          
                                                                                
    if WITH_THREAD:                                                             
        t = threading.Thread( target=create_object )                            
        t.start()                                                               
        t.join()                                                                
    else:                                                                       
        create_object()                                                         
                                                                                
    teardown_test_environment()                                                 
    connection.creation.destroy_test_db( old_db_name )             

If I run it with WITH_THREADS set to False:

Philippe@pc-philippe /cygdrive/d/work/django/django-tuto/mysite $ python run_wi 
th_threads.py                                                                   
Creating test database...                                                       
Creating table auth_permission                                                  
Creating table auth_group                                                       
Creating table auth_user                                                        
Creating table auth_message                                                     
Creating table django_admin_log                                                 
Creating table django_content_type                                              
Creating table django_session                                                   
Creating table django_site                                                      
Creating table polls_poll                                                       
Creating table polls_choice                                                     
Installing index for auth.Permission model                                      
Installing index for auth.Message model                                         
Installing index for admin.LogEntry model                                       
Installing index for polls.Choice model                                         
New DATABASE: :memory:                                                          
Creating Poll                                                                   
Poll object saved. Id: 1                                                        
Destroying test database...   

If I run it with WITH_THREADS set to True:

Philippe@pc-philippe /cygdrive/d/work/django/django-tuto/mysite $ python run_wi 
th_threads.py                                                                   
Creating test database...                                                       
Creating table auth_permission                                                  
Creating table auth_group                                                       
Creating table auth_user                                                        
Creating table auth_message                                                     
Creating table django_admin_log                                                 
Creating table django_content_type                                              
Creating table django_session                                                   
Creating table django_site                                                      
Creating table polls_poll                                                       
Creating table polls_choice                                                     
Installing index for auth.Permission model                                      
Installing index for auth.Message model                                         
Installing index for admin.LogEntry model                                       
Installing index for polls.Choice model                                         
New DATABASE: :memory:                                                          
Creating Poll                                                                   
Exception in thread Thread-1:                                                   
Traceback (most recent call last):                                              
  File "c:\Python26\lib\threading.py", line 522, in __bootstrap_inner           
    self.run()                                                                  
  File "c:\Python26\lib\threading.py", line 477, in run                         
    self.__target(*self.__args, **self.__kwargs)                                
  File "run_with_threads.py", line 19, in create_object                         
    p.save()                                                                    
  File "c:\Python26\lib\site-packages\django\db\models\base.py", line 410, in save                                                                              
    self.save_base(force_insert=force_insert, force_update=force_update)        
  File "c:\Python26\lib\site-packages\django\db\models\base.py", line 495, in save_base                                                                         
    result = manager._insert(values, return_id=update_pk)                       
  File "c:\Python26\lib\site-packages\django\db\models\manager.py", line 177, in _insert                                                                        
    return insert_query(self.model, values, **kwargs)                           
  File "c:\Python26\lib\site-packages\django\db\models\query.py", line 1087, in 
insert_query                                                                    
    return query.execute_sql(return_id)                                         
  File "c:\Python26\lib\site-packages\django\db\models\sql\subqueries.py", line 
320, in execute_sql                                                             
    cursor = super(InsertQuery, self).execute_sql(None)                         
  File "c:\Python26\lib\site-packages\django\db\models\sql\query.py", line 2369, in execute_sql                                                                 
    cursor.execute(sql, params)                                                 
  File "c:\Python26\lib\site-packages\django\db\backends\util.py", line 19, in execute                                                                          
    return self.cursor.execute(sql, params)                                     
  File "c:\Python26\lib\site-packages\django\db\backends\sqlite3\base.py", line 
193, in execute                                                                 
    return Database.Cursor.execute(self, query, params)                         
OperationalError: no such table: polls_poll                                     
                                                                                
Destroying test database...  

Change History (8)

comment:1 Changed 6 years ago by kmtracey

  • Needs documentation unset
  • Needs tests unset
  • Patch needs improvement unset
  • Resolution set to invalid
  • Status changed from new to closed

Near as I can tell this is an sqlite or pysqlite restriction, though I haven't been able to find a definitive statement in sqlite or pysqlite doc. SQLAlchemy, however, appears to have come to the conclusion that you simply cannot share :memory databases between threads, see: http://www.sqlalchemy.org/docs/06/reference/dialects/sqlite.html#threading-behavior. Thus I think there is no bug in Django here, the answer is don't do that and if you really want to do it you'll need to take it up with the maintainers of the lower level code.

comment:2 Changed 15 months ago by anonymous

  • Easy pickings unset
  • Resolution invalid deleted
  • Severity set to Normal
  • Status changed from closed to new
  • Type set to Uncategorized
  • UI/UX unset

I vote to reopen this.

sqlite, as of version 3.7.13 (released 2012-06-11) has the ability to share an in-memory database between multiple connections and threads.

See: http://www.sqlite.org/releaselog/3_7_13.html

Making this work with the Django testing framework should be pretty easy:

In the sqlite database backend, instead of using the database name :memory:, we should use a name such as file:testdb?mode=memory&cache=shared (where "testdb" can be anything and ideally should be unique so that multiple tests can run concurrently, each with its own individual database).

As a bonus, doing this should allow removing the hacky sqlite-specific code from "LiveServerTestCase" (It contains a messy workaround for exactly the issue of this bug report)

The only thing that might be a little tricky is making this update in the Django code to still support older vesions of sqlite by falling back to the current behavior (:memory:), but I imagine that this shouldn't be too difficult.

comment:3 Changed 14 months ago by Alex

  • Triage Stage changed from Unreviewed to Accepted

comment:4 Changed 14 months ago by timo

  • Type changed from Uncategorized to New feature
  • Version changed from 1.1 to master

comment:5 Changed 5 months ago by coldmind

  • Owner changed from nobody to coldmind
  • Status changed from new to assigned

comment:6 Changed 5 months ago by coldmind

  • Has patch set

comment:7 Changed 4 months ago by timgraham

  • Patch needs improvement set

comment:8 Changed 4 months ago by Tim Graham <timograham@…>

  • Resolution set to fixed
  • Status changed from assigned to closed

In 8c99b7920e8187f98cf4d7dbd9918bd6c6da1238:

Fixed #12118 -- Added shared cache support to SQLite in-memory testing.

Note: See TracTickets for help on using tickets.
Back to Top