test runner repeatedly parsers fixtures

[Trevor Caira]

The django test runner calls loaddata for every test case which has fixtures. In projects which reuse the same fixtures for multiple test cases, running the test suite results in the wasteful repeated parsing of the same test fixtures, which is very slow. For my application, caching the result of fixture parsing results in an at least 7% speedup in the test suite.

[Eric Holscher]

I the approach of actually caching the loaded fixture data, and we seeing a 25% speedup on the admin_views tests. Running against the full suite only showed a small improvement, and had a couple of test failures. I think this would be a nice thing to address and would provide a non-trivial speedup.

My approach is here, but mostly just a proof of concept: ​http://github.com/ericholscher/django/commit/2d2056d9256bc3599442b93487427a1735c5c8c7

Alex Gaynor suggested using a ref-counting scheme, where it would cache fixtures that it knew would be called again later in the test suite.

[12:59]  <Alex_Gaynor> ericholscher: a) Definitely sholdn't be a classwide cache
[12:59]  <Alex_Gaynor> ericholscher: I think you should make the class take an optional fixture_cache parameter.
[13:00]  <Alex_Gaynor> ericholscher: that way the test runner can be responsible for the refcounting and clearing of the cache

Output from testing:

    eric@Loki:~/checkouts/django$ /usr/bin/time -v python tests/runtests.py admin_views --settings=tests.settings -v0
    ----------------------------------------------------------------------
    Ran 94 tests in 16.200s
    
    OK
            Command being timed: "python tests/runtests.py admin_views --settings=tests.settings -v0"
            User time (seconds): 16.36
            System time (seconds): 0.51
            Percent of CPU this job got: 98%
            Elapsed (wall clock) time (h:mm:ss or m:ss): 0:17.21
            Average shared text size (kbytes): 0
            Average unshared data size (kbytes): 0
            Average stack size (kbytes): 0
            Average total size (kbytes): 0
            Maximum resident set size (kbytes): 0
            Average resident set size (kbytes): 0
            Major (requiring I/O) page faults: 0
            Minor (reclaiming a frame) page faults: 10539
            Voluntary context switches: 1
            Involuntary context switches: 58
            Swaps: 0
            File system inputs: 0
            File system outputs: 4104
            Socket messages sent: 0
            Socket messages received: 0
            Signals delivered: 0
            Page size (bytes): 4096
            Exit status: 0
    eric@Loki:~/checkouts/django$ git branch
      1.1.X
      9449-cache-fixtures
    * master
    eric@Loki:~/checkouts/django$ git co 9449-cache-fixtures
    Switched to branch '9449-cache-fixtures'
    eric@Loki:~/checkouts/django$ /usr/bin/time -v python tests/runtests.py admin_views --settings=tests.settings -v0
    ----------------------------------------------------------------------
    Ran 94 tests in 11.990s
    
    OK
            Command being timed: "python tests/runtests.py admin_views --settings=tests.settings -v0"
            User time (seconds): 12.48
            System time (seconds): 0.41
            Percent of CPU this job got: 98%
            Elapsed (wall clock) time (h:mm:ss or m:ss): 0:13.13
            Average shared text size (kbytes): 0
            Average unshared data size (kbytes): 0
            Average stack size (kbytes): 0
            Average total size (kbytes): 0
            Maximum resident set size (kbytes): 0
            Average resident set size (kbytes): 0
            Major (requiring I/O) page faults: 0
            Minor (reclaiming a frame) page faults: 10090
            Voluntary context switches: 1
            Involuntary context switches: 72
            Swaps: 0
            File system inputs: 0
            File system outputs: 4192
            Socket messages sent: 0
            Socket messages received: 0
            Signals delivered: 0
            Page size (bytes): 4096
            Exit status: 0
    
    
    eric@Loki:~/checkouts/django$ git co master
    Switched to branch 'master'
    eric@Loki:~/checkouts/django$ /usr/bin/time -v python tests/runtests.py --settings=tests.settings -v0
    /home/eric/checkouts/django/django/contrib/admin/sites.py:455: DeprecationWarning: AdminSite.root() is deprecated; use include(admin.site.urls) instead.
      DeprecationWarning
    ----------------------------------------------------------------------
    Ran 1044 tests in 417.810s
    
    OK
            Command being timed: "python tests/runtests.py --settings=tests.settings -v0"
            User time (seconds): 386.75
            System time (seconds): 13.30
            Percent of CPU this job got: 92%
            Elapsed (wall clock) time (h:mm:ss or m:ss): 7:12.97
            Average shared text size (kbytes): 0
            Average unshared data size (kbytes): 0
            Average stack size (kbytes): 0
            Average total size (kbytes): 0
            Maximum resident set size (kbytes): 0
            Average resident set size (kbytes): 0
            Major (requiring I/O) page faults: 179
            Minor (reclaiming a frame) page faults: 489021
            Voluntary context switches: 3094
            Involuntary context switches: 2272
            Swaps: 0
            File system inputs: 38576
            File system outputs: 150872
            Socket messages sent: 0
            Socket messages received: 0
            Signals delivered: 0
            Page size (bytes): 4096
            Exit status: 0
    eric@Loki:~/checkouts/django$ git co 9449-cache-fixtures
    Switched to branch '9449-cache-fixtures'
    eric@Loki:~/checkouts/django$ /usr/bin/time -v python tests/runtests.py --settings=tests.settings -v0
    /home/eric/checkouts/django/django/contrib/admin/sites.py:455: DeprecationWarning: AdminSite.root() is deprecated; use include(admin.site.urls) instead.
    ... Snipped 4 failures ...
    Ran 1043 tests in 387.500s
    
    FAILED (failures=4)
    Command exited with non-zero status 4
            Command being timed: "python tests/runtests.py --settings=tests.settings -v0"
            User time (seconds): 363.75
            System time (seconds): 9.75
            Percent of CPU this job got: 92%
            Elapsed (wall clock) time (h:mm:ss or m:ss): 6:43.70
            Average shared text size (kbytes): 0
            Average unshared data size (kbytes): 0
            Average stack size (kbytes): 0
            Average total size (kbytes): 0
            Maximum resident set size (kbytes): 0
            Average resident set size (kbytes): 0
            Major (requiring I/O) page faults: 115
            Minor (reclaiming a frame) page faults: 489664
            Voluntary context switches: 3240
            Involuntary context switches: 1215
            Swaps: 0
            File system inputs: 40424
            File system outputs: 150544
            Socket messages sent: 0
            Socket messages received: 0
            Signals delivered: 0
            Page size (bytes): 4096
            Exit status: 4

[Eric Holscher]

The failures I was seeing appear to be related to the fact that I am caching the fixture_label, and there are multiple fixtures named testdata.json and initial_data.json.

[Eric Holscher]

Added another commit that fixes the aggregation tests depending on initial_data loading, which is already tested in the fixtures tests.

This also makes these tests pass with this scheme:

​http://github.com/ericholscher/django/commit/08b1f3e5f98529f4b9c46c911ee00cb07ff0bdbc

[Eric Holscher]

Going to work on this for 1.3.

Thinking it should be disablable, because for small test runs it will cause an increase in memory and time. It might be insignificant enough to not worry about though.

[Ramiro Morales]

See also this django-dev thread: ​http://groups.google.com/group/django-developers/browse_thread/thread/65a36a8a5433144c?hl=en

[Jacob]

Milestone 1.3 deleted

[Aymeric Augustin]

Change UI/UX from NULL to False.

[Tim Graham]

In Django's test suite, fixtures have mostly been replaced with TestCase.setUpTestData() which is more readable and maintainable. Given that and the lack of activity here, is there any objection to closing this ticket as wontfix?

[Simon Charette]

I would add that fixtures are now loaded as part or setUpTestData(), preventing per-test parsing, which seems to match ​what was initially proposed on the ML.