Opened 13 years ago
Closed 13 years ago
#18117 closed Bug (duplicate)
Django 1.4 Trouble Creating Large Numbers of Tables with SQLite
| Reported by: | Owned by: | nobody | |
|---|---|---|---|
| Component: | Uncategorized | Version: | 1.4 |
| Severity: | Normal | Keywords: | SQLite, max columns, syncdb, permissions |
| Cc: | Triage Stage: | Unreviewed | |
| Has patch: | no | Needs documentation: | no |
| Needs tests: | no | Patch needs improvement: | no |
| Easy pickings: | no | UI/UX: | no |
Description
After upgrading from Django 1.3 to Django 1.4 and running syncdb I get the following error message (with traceback):
Traceback (most recent call last):
File "./manage.py", line 10, in <module>
execute_from_command_line(sys.argv)
File "/usr/local/lib/python2.7/dist-packages/Django-1.4-py2.7.egg/django/core/management/init.py", line 443, in execute_from_command_line
utility.execute()
File "/usr/local/lib/python2.7/dist-packages/Django-1.4-py2.7.egg/django/core/management/init.py", line 382, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/usr/local/lib/python2.7/dist-packages/Django-1.4-py2.7.egg/django/core/management/base.py", line 196, in run_from_argv
self.execute(*args, options.dict)
File "/usr/local/lib/python2.7/dist-packages/Django-1.4-py2.7.egg/django/core/management/base.py", line 232, in execute
output = self.handle(*args, options)
File "/usr/local/lib/python2.7/dist-packages/Django-1.4-py2.7.egg/django/core/management/base.py", line 371, in handle
return self.handle_noargs(options)
File "/usr/local/lib/python2.7/dist-packages/Django-1.4-py2.7.egg/django/core/management/commands/syncdb.py", line 110, in handle_noargs
emit_post_sync_signal(created_models, verbosity, interactive, db)
File "/usr/local/lib/python2.7/dist-packages/Django-1.4-py2.7.egg/django/core/management/sql.py", line 189, in emit_post_sync_signal
interactive=interactive, db=db)
File "/usr/local/lib/python2.7/dist-packages/Django-1.4-py2.7.egg/django/dispatch/dispatcher.py", line 172, in send
response = receiver(signal=self, sender=sender, named)
File "/usr/local/lib/python2.7/dist-packages/Django-1.4-py2.7.egg/django/contrib/auth/management/init.py", line 54, in create_permissions
auth_app.Permission.objects.bulk_create(objs)
File "/usr/local/lib/python2.7/dist-packages/Django-1.4-py2.7.egg/django/db/models/manager.py", line 140, in bulk_create
return self.get_query_set().bulk_create(*args, kwargs)
File "/usr/local/lib/python2.7/dist-packages/Django-1.4-py2.7.egg/django/db/models/query.py", line 410, in bulk_create
self.model._base_manager._insert(objs, fields=fields, using=self.db)
File "/usr/local/lib/python2.7/dist-packages/Django-1.4-py2.7.egg/django/db/models/manager.py", line 203, in _insert
return insert_query(self.model, objs, fields, kwargs)
File "/usr/local/lib/python2.7/dist-packages/Django-1.4-py2.7.egg/django/db/models/query.py", line 1576, in insert_query
return query.get_compiler(using=using).execute_sql(return_id)
File "/usr/local/lib/python2.7/dist-packages/Django-1.4-py2.7.egg/django/db/models/sql/compiler.py", line 910, in execute_sql
cursor.execute(sql, params)
File "/usr/local/lib/python2.7/dist-packages/Django-1.4-py2.7.egg/django/db/backends/util.py", line 40, in execute
return self.cursor.execute(sql, params)
File "/usr/local/lib/python2.7/dist-packages/Django-1.4-py2.7.egg/django/db/backends/sqlite3/base.py", line 337, in execute
return Database.Cursor.execute(self, query, params)
django.db.utils.DatabaseError: too many SQL variables
After doing some research I discovered that this error message is frequently related to trying to create more columns than the sqlite configuration file is set to allow (2000 by default, see http://www.sqlite.org/limits.html). I don't have any one table with more than 2000 columns but all the tables together have well over that amount. It seems this may be a bug and I haven't seen anyone else mention this as a problem.
When I reduce the number of tables in my models.py to well below 2000 the models are all created just fine.
Change History (6)
comment:1 by , 13 years ago
| Resolution: | → duplicate |
|---|---|
| Status: | new → closed |
comment:2 by , 13 years ago
I'm not sure this is a duplicate of #18117. #18117 is about bulk_insert, and we don't use bulk insert to create tables.
However, it isn't up to Django to work around limitations in SQLite. I suggest either recompiling SQLite with larger limits or using a database without such limits. I'd have closed that ticket as "wontfix".
comment:3 by , 13 years ago
This is after all about bulk_create. If you check the stack trace, this is actually about creating permissions, not create table commands.
comment:5 by , 13 years ago
| Keywords: | syncdb permissions added |
|---|---|
| Resolution: | duplicate |
| Status: | closed → reopened |
I agree that the error message is the same as #17788 but I believe these issues are a little different. The other bug is more concerned about bulk_create(). This issue is more concerned about being able to run syncdb on django 1.4. The core issue is before syncdb worked under django 1.3 and now it does not for django 1.4. when running syncdb I cannot control how bulk insert works. The permission system is an integral part of our project so I cannot get rid of that. At the very least why did this work with django 1.3? Has SQLite changed? Is there at least a way around this problem?
comment:6 by , 13 years ago
| Resolution: | → duplicate |
|---|---|
| Status: | reopened → closed |
The implementation of syncdb (actually, the creation of the permissions which is triggered by post_syncdb) changed between 1.3 and 1.4. Now it uses bulk_create. So your problem really is a symptom of the regression tracked by #17788.
If you have to use SQLite, your best alternative is to re-compile it with higher limits. However, the recommended solution is to use a database that isn't limited like SQLite, when you hit its limits.
Duplicate of #17788.