id	summary	reporter	owner	description	type	status	component	version	severity	resolution	keywords	cc	stage	has_patch	needs_docs	needs_tests	needs_better_patch	easy	ui_ux
20485	Find faster ways for test fixture loading	Anssi Kääriäinen	Aymeric Augustin	"Currently Django's test suite uses about 70s for fixture loading when running full test suite on in-memory SQLite. Considering that after #20483 Django's test suite takes around 210s in total, this means 1/3 of time is used for fixture loading. Most of the time in fixture loading is used for finding the fixture file. Django needs to do effectively this for each file, for each test:
{{{
for each app in INSTALLED_APPS:
    for each suffix in combo(('json', 'yaml', 'xml') X ('bz2', 'gz', 'zip')):
        try to open file() - if succeeds add to fixture files, else skip
}}}
Unsurprisingly this is expensive. In addition fixture loading after the file has been found isn't that fast, it is actually somewhat faster to create the models by SomeModel.objects.create() than by fixtures.

I have done some testing and it shows that of the 70s, around 50-60s should be avoidable.

Some options:
  - Somehow use absolute paths for the fixtures. Maybe by using `path(__file__) + '/fixtures/' + fixture_name`. A subset of this is to always use, say gz compressed files which removes the need to check for combos.
  - Limit fixture searching to current app (if requested): maybe by something like './somefixture' -> search only from current app. 'somefixture' -> search everywhere.
  - Cache the existing fixtures. That is, go only once through the directories, check what files exists and cache that information, then check against the cached information instead of trying to open each possible combo in each possible directory.
  - Do not use fixtures for Django's test data.

Of the above I like the last option as I find reading models created in Python code much easier than definitions in fixtures (especially, what relations does given instance have). Unfortunately converting the fixtures takes time. In addition, I am sure some developers favour fixtures instead of models created in Python code.

Marking as accepted as in ""lets see what can be done here""."	Cleanup/optimization	closed	Testing framework	dev	Normal	fixed			Ready for checkin	1	0	0	0	0	0