Code

Opened 2 years ago

Closed 2 years ago

#18117 closed Bug (duplicate)

Django 1.4 Trouble Creating Large Numbers of Tables with SQLite

Reported by: rball@… Owned by: nobody
Component: Uncategorized Version: 1.4
Severity: Normal Keywords: SQLite, max columns, syncdb, permissions
Cc: Triage Stage: Unreviewed
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

After upgrading from Django 1.3 to Django 1.4 and running syncdb I get the following error message (with traceback):

Traceback (most recent call last):

File "./manage.py", line 10, in <module>

execute_from_command_line(sys.argv)

File "/usr/local/lib/python2.7/dist-packages/Django-1.4-py2.7.egg/django/core/management/init.py", line 443, in execute_from_command_line

utility.execute()

File "/usr/local/lib/python2.7/dist-packages/Django-1.4-py2.7.egg/django/core/management/init.py", line 382, in execute

self.fetch_command(subcommand).run_from_argv(self.argv)

File "/usr/local/lib/python2.7/dist-packages/Django-1.4-py2.7.egg/django/core/management/base.py", line 196, in run_from_argv

self.execute(*args, options.dict)

File "/usr/local/lib/python2.7/dist-packages/Django-1.4-py2.7.egg/django/core/management/base.py", line 232, in execute

output = self.handle(*args, options)

File "/usr/local/lib/python2.7/dist-packages/Django-1.4-py2.7.egg/django/core/management/base.py", line 371, in handle

return self.handle_noargs(options)

File "/usr/local/lib/python2.7/dist-packages/Django-1.4-py2.7.egg/django/core/management/commands/syncdb.py", line 110, in handle_noargs

emit_post_sync_signal(created_models, verbosity, interactive, db)

File "/usr/local/lib/python2.7/dist-packages/Django-1.4-py2.7.egg/django/core/management/sql.py", line 189, in emit_post_sync_signal

interactive=interactive, db=db)

File "/usr/local/lib/python2.7/dist-packages/Django-1.4-py2.7.egg/django/dispatch/dispatcher.py", line 172, in send

response = receiver(signal=self, sender=sender, named)

File "/usr/local/lib/python2.7/dist-packages/Django-1.4-py2.7.egg/django/contrib/auth/management/init.py", line 54, in create_permissions

auth_app.Permission.objects.bulk_create(objs)

File "/usr/local/lib/python2.7/dist-packages/Django-1.4-py2.7.egg/django/db/models/manager.py", line 140, in bulk_create

return self.get_query_set().bulk_create(*args, kwargs)

File "/usr/local/lib/python2.7/dist-packages/Django-1.4-py2.7.egg/django/db/models/query.py", line 410, in bulk_create

self.model._base_manager._insert(objs, fields=fields, using=self.db)

File "/usr/local/lib/python2.7/dist-packages/Django-1.4-py2.7.egg/django/db/models/manager.py", line 203, in _insert

return insert_query(self.model, objs, fields, kwargs)

File "/usr/local/lib/python2.7/dist-packages/Django-1.4-py2.7.egg/django/db/models/query.py", line 1576, in insert_query

return query.get_compiler(using=using).execute_sql(return_id)

File "/usr/local/lib/python2.7/dist-packages/Django-1.4-py2.7.egg/django/db/models/sql/compiler.py", line 910, in execute_sql

cursor.execute(sql, params)

File "/usr/local/lib/python2.7/dist-packages/Django-1.4-py2.7.egg/django/db/backends/util.py", line 40, in execute

return self.cursor.execute(sql, params)

File "/usr/local/lib/python2.7/dist-packages/Django-1.4-py2.7.egg/django/db/backends/sqlite3/base.py", line 337, in execute

return Database.Cursor.execute(self, query, params)

django.db.utils.DatabaseError: too many SQL variables

After doing some research I discovered that this error message is frequently related to trying to create more columns than the sqlite configuration file is set to allow (2000 by default, see http://www.sqlite.org/limits.html). I don't have any one table with more than 2000 columns but all the tables together have well over that amount. It seems this may be a bug and I haven't seen anyone else mention this as a problem.

When I reduce the number of tables in my models.py to well below 2000 the models are all created just fine.

Attachments (0)

Change History (6)

comment:1 Changed 2 years ago by claudep

  • Needs documentation unset
  • Needs tests unset
  • Patch needs improvement unset
  • Resolution set to duplicate
  • Status changed from new to closed

Duplicate of #17788.

comment:2 Changed 2 years ago by aaugustin

I'm not sure this is a duplicate of #18117. #18117 is about bulk_insert, and we don't use bulk insert to create tables.

However, it isn't up to Django to work around limitations in SQLite. I suggest either recompiling SQLite with larger limits or using a database without such limits. I'd have closed that ticket as "wontfix".

comment:3 Changed 2 years ago by akaariai

This is after all about bulk_create. If you check the stack trace, this is actually about creating permissions, not create table commands.

comment:4 Changed 2 years ago by aaugustin

Ah, indeed. Sorry. I should have reformatted and read the stack trace.

comment:5 Changed 2 years ago by rballverisys

  • Keywords columns, syncdb, permissions added; columns removed
  • Resolution duplicate deleted
  • Status changed from closed to reopened

I agree that the error message is the same as #17788 but I believe these issues are a little different. The other bug is more concerned about bulk_create(). This issue is more concerned about being able to run syncdb on django 1.4. The core issue is before syncdb worked under django 1.3 and now it does not for django 1.4. when running syncdb I cannot control how bulk insert works. The permission system is an integral part of our project so I cannot get rid of that. At the very least why did this work with django 1.3? Has SQLite changed? Is there at least a way around this problem?

comment:6 Changed 2 years ago by aaugustin

  • Resolution set to duplicate
  • Status changed from reopened to closed

The implementation of syncdb (actually, the creation of the permissions which is triggered by post_syncdb) changed between 1.3 and 1.4. Now it uses bulk_create. So your problem really is a symptom of the regression tracked by #17788.

If you have to use SQLite, your best alternative is to re-compile it with higher limits. However, the recommended solution is to use a database that isn't limited like SQLite, when you hit its limits.

Add Comment

Modify Ticket

Change Properties
<Author field>
Action
as closed
as The resolution will be set. Next status will be 'closed'
The resolution will be deleted. Next status will be 'new'
Author


E-mail address and user name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.