Django 1.4 Trouble Creating Large Numbers of Tables with SQLite

[rball@…]

After upgrading from Django 1.3 to Django 1.4 and running syncdb I get the following error message (with traceback):

Traceback (most recent call last):

File "./manage.py", line 10, in <module>

execute_from_command_line(sys.argv)

File "/usr/local/lib/python2.7/dist-packages/Django-1.4-py2.7.egg/django/core/management/init.py", line 443, in execute_from_command_line

utility.execute()

File "/usr/local/lib/python2.7/dist-packages/Django-1.4-py2.7.egg/django/core/management/init.py", line 382, in execute

self.fetch_command(subcommand).run_from_argv(self.argv)

File "/usr/local/lib/python2.7/dist-packages/Django-1.4-py2.7.egg/django/core/management/base.py", line 196, in run_from_argv

self.execute(*args, options.dict)

File "/usr/local/lib/python2.7/dist-packages/Django-1.4-py2.7.egg/django/core/management/base.py", line 232, in execute

output = self.handle(*args, options)

File "/usr/local/lib/python2.7/dist-packages/Django-1.4-py2.7.egg/django/core/management/base.py", line 371, in handle

return self.handle_noargs(options)

File "/usr/local/lib/python2.7/dist-packages/Django-1.4-py2.7.egg/django/core/management/commands/syncdb.py", line 110, in handle_noargs

emit_post_sync_signal(created_models, verbosity, interactive, db)

File "/usr/local/lib/python2.7/dist-packages/Django-1.4-py2.7.egg/django/core/management/sql.py", line 189, in emit_post_sync_signal

interactive=interactive, db=db)

File "/usr/local/lib/python2.7/dist-packages/Django-1.4-py2.7.egg/django/dispatch/dispatcher.py", line 172, in send

response = receiver(signal=self, sender=sender, named)

File "/usr/local/lib/python2.7/dist-packages/Django-1.4-py2.7.egg/django/contrib/auth/management/init.py", line 54, in create_permissions

auth_app.Permission.objects.bulk_create(objs)

File "/usr/local/lib/python2.7/dist-packages/Django-1.4-py2.7.egg/django/db/models/manager.py", line 140, in bulk_create

return self.get_query_set().bulk_create(*args, kwargs)

File "/usr/local/lib/python2.7/dist-packages/Django-1.4-py2.7.egg/django/db/models/query.py", line 410, in bulk_create

self.model._base_manager._insert(objs, fields=fields, using=self.db)

File "/usr/local/lib/python2.7/dist-packages/Django-1.4-py2.7.egg/django/db/models/manager.py", line 203, in _insert

return insert_query(self.model, objs, fields, kwargs)

File "/usr/local/lib/python2.7/dist-packages/Django-1.4-py2.7.egg/django/db/models/query.py", line 1576, in insert_query

return query.get_compiler(using=using).execute_sql(return_id)

File "/usr/local/lib/python2.7/dist-packages/Django-1.4-py2.7.egg/django/db/models/sql/compiler.py", line 910, in execute_sql

cursor.execute(sql, params)

File "/usr/local/lib/python2.7/dist-packages/Django-1.4-py2.7.egg/django/db/backends/util.py", line 40, in execute

return self.cursor.execute(sql, params)

File "/usr/local/lib/python2.7/dist-packages/Django-1.4-py2.7.egg/django/db/backends/sqlite3/base.py", line 337, in execute

return Database.Cursor.execute(self, query, params)

django.db.utils.DatabaseError: too many SQL variables

After doing some research I discovered that this error message is frequently related to trying to create more columns than the sqlite configuration file is set to allow (2000 by default, see ​http://www.sqlite.org/limits.html). I don't have any one table with more than 2000 columns but all the tables together have well over that amount. It seems this may be a bug and I haven't seen anyone else mention this as a problem.

When I reduce the number of tables in my models.py to well below 2000 the models are all created just fine.

[Claude Paroz]

Duplicate of #17788.

[Aymeric Augustin]

I'm not sure this is a duplicate of #18117. #18117 is about bulk_insert, and we don't use bulk insert to create tables.

However, it isn't up to Django to work around limitations in SQLite. I suggest either recompiling SQLite with larger limits or using a database without such limits. I'd have closed that ticket as "wontfix".

[Anssi Kääriäinen]

This is after all about bulk_create. If you check the stack trace, this is actually about creating permissions, not create table commands.

[Aymeric Augustin]

Ah, indeed. Sorry. I should have reformatted and read the stack trace.

[rballverisys]

I agree that the error message is the same as #17788 but I believe these issues are a little different. The other bug is more concerned about bulk_create(). This issue is more concerned about being able to run syncdb on django 1.4. The core issue is before syncdb worked under django 1.3 and now it does not for django 1.4. when running syncdb I cannot control how bulk insert works. The permission system is an integral part of our project so I cannot get rid of that. At the very least why did this work with django 1.3? Has SQLite changed? Is there at least a way around this problem?

[Aymeric Augustin]

The implementation of syncdb (actually, the creation of the permissions which is triggered by post_syncdb) changed between 1.3 and 1.4. Now it uses bulk_create. So your problem really is a symptom of the regression tracked by #17788.

If you have to use SQLite, your best alternative is to re-compile it with higher limits. However, the recommended solution is to use a database that isn't limited like SQLite, when you hit its limits.