id	summary	reporter	owner	description	type	status	component	version	severity	resolution	keywords	cc	stage	has_patch	needs_docs	needs_tests	needs_better_patch	easy	ui_ux
14220	Backwards-compat code in db.fields.subclassing is a bottleneck	Ole Laursen	nobody	"Have a project where we need to import some data from mobile devices from time to time, up to about 20.000 rows. With the regular .create()/save() methods, this takes more than 10 minutes, even after fiddling with transactions. So I did a quick hack to combine the model fields with raw SQL and executemany() (code below).

This was about 20 times faster, so I was now down to about a minute in total. I ran the result through Hotspot (Python profiler) and to my surprise, a large fraction of the time was spent inside inner() in django.db.models.fields.subclassing. The culprit appears to be call_with_connection and call_with_connection_and_prepared. Inserting a ""return func"" in the top of both speeds up the insertion by at least 25% (much of the remaining time is spent in the DB).

I realize backwards compatibility code is necessary, but I guess it's possible to fix this somehow still?

Here's the code that's exercising it (more specifically the line with get_db_prep-save, the function takes a list of not-yet-inserted-in-db Django objects and inserts them):

{{{
def insert_many(objects, using=""default""):
    if not objects:
        return

    import django.db.models
    from django.db import connections
    con = connections[using]
    
    model = objects[0].__class__
    fields = [f for f in model._meta.fields if not isinstance(f, django.db.models.AutoField)]
    parameters = []
    for o in objects:
        parameters.append(tuple(f.get_db_prep_save(f.pre_save(o, True), connection=con) for f in fields))

    table = model._meta.db_table
    column_names = "","".join(f.column for f in fields)
    placeholders = "","".join((""%s"",) * len(fields))
    con.cursor().executemany(
        ""insert into %s (%s) values (%s)"" % (table, column_names, placeholders),
        parameters)
}}}"	Cleanup/optimization	closed	Database layer (models, ORM)	1.2	Normal	wontfix		graham@…	Accepted	0	0	0	0	0	0