Opened 7 years ago

Last modified 4 years ago

#9449 assigned Cleanup/optimization

test runner repeatedly parsers fixtures

Reported by: trevor Owned by: ericholscher
Component: Testing framework Version: 1.0
Severity: Normal Keywords:
Cc: Triage Stage: Accepted
Has patch: yes Needs documentation: yes
Needs tests: yes Patch needs improvement: yes
Easy pickings: no UI/UX: no

Description

The django test runner calls loaddata for every test case which has fixtures. In projects which reuse the same fixtures for multiple test cases, running the test suite results in the wasteful repeated parsing of the same test fixtures, which is very slow. For my application, caching the result of fixture parsing results in an at least 7% speedup in the test suite.

Change History (9)

comment:1 Changed 7 years ago by jacob

  • Needs documentation unset
  • Needs tests unset
  • Patch needs improvement unset
  • Triage Stage changed from Unreviewed to Accepted

comment:2 Changed 6 years ago by ericholscher

I the approach of actually caching the loaded fixture data, and we seeing a 25% speedup on the admin_views tests. Running against the full suite only showed a small improvement, and had a couple of test failures. I think this would be a nice thing to address and would provide a non-trivial speedup.

My approach is here, but mostly just a proof of concept: http://github.com/ericholscher/django/commit/2d2056d9256bc3599442b93487427a1735c5c8c7

Alex Gaynor suggested using a ref-counting scheme, where it would cache fixtures that it knew would be called again later in the test suite.

[12:59]  <Alex_Gaynor> ericholscher: a) Definitely sholdn't be a classwide cache
[12:59]  <Alex_Gaynor> ericholscher: I think you should make the class take an optional fixture_cache parameter.
[13:00]  <Alex_Gaynor> ericholscher: that way the test runner can be responsible for the refcounting and clearing of the cache

Output from testing:

    eric@Loki:~/checkouts/django$ /usr/bin/time -v python tests/runtests.py admin_views --settings=tests.settings -v0
    ----------------------------------------------------------------------
    Ran 94 tests in 16.200s
    
    OK
            Command being timed: "python tests/runtests.py admin_views --settings=tests.settings -v0"
            User time (seconds): 16.36
            System time (seconds): 0.51
            Percent of CPU this job got: 98%
            Elapsed (wall clock) time (h:mm:ss or m:ss): 0:17.21
            Average shared text size (kbytes): 0
            Average unshared data size (kbytes): 0
            Average stack size (kbytes): 0
            Average total size (kbytes): 0
            Maximum resident set size (kbytes): 0
            Average resident set size (kbytes): 0
            Major (requiring I/O) page faults: 0
            Minor (reclaiming a frame) page faults: 10539
            Voluntary context switches: 1
            Involuntary context switches: 58
            Swaps: 0
            File system inputs: 0
            File system outputs: 4104
            Socket messages sent: 0
            Socket messages received: 0
            Signals delivered: 0
            Page size (bytes): 4096
            Exit status: 0
    eric@Loki:~/checkouts/django$ git branch
      1.1.X
      9449-cache-fixtures
    * master
    eric@Loki:~/checkouts/django$ git co 9449-cache-fixtures
    Switched to branch '9449-cache-fixtures'
    eric@Loki:~/checkouts/django$ /usr/bin/time -v python tests/runtests.py admin_views --settings=tests.settings -v0
    ----------------------------------------------------------------------
    Ran 94 tests in 11.990s
    
    OK
            Command being timed: "python tests/runtests.py admin_views --settings=tests.settings -v0"
            User time (seconds): 12.48
            System time (seconds): 0.41
            Percent of CPU this job got: 98%
            Elapsed (wall clock) time (h:mm:ss or m:ss): 0:13.13
            Average shared text size (kbytes): 0
            Average unshared data size (kbytes): 0
            Average stack size (kbytes): 0
            Average total size (kbytes): 0
            Maximum resident set size (kbytes): 0
            Average resident set size (kbytes): 0
            Major (requiring I/O) page faults: 0
            Minor (reclaiming a frame) page faults: 10090
            Voluntary context switches: 1
            Involuntary context switches: 72
            Swaps: 0
            File system inputs: 0
            File system outputs: 4192
            Socket messages sent: 0
            Socket messages received: 0
            Signals delivered: 0
            Page size (bytes): 4096
            Exit status: 0
    
    
    eric@Loki:~/checkouts/django$ git co master
    Switched to branch 'master'
    eric@Loki:~/checkouts/django$ /usr/bin/time -v python tests/runtests.py --settings=tests.settings -v0
    /home/eric/checkouts/django/django/contrib/admin/sites.py:455: DeprecationWarning: AdminSite.root() is deprecated; use include(admin.site.urls) instead.
      DeprecationWarning
    ----------------------------------------------------------------------
    Ran 1044 tests in 417.810s
    
    OK
            Command being timed: "python tests/runtests.py --settings=tests.settings -v0"
            User time (seconds): 386.75
            System time (seconds): 13.30
            Percent of CPU this job got: 92%
            Elapsed (wall clock) time (h:mm:ss or m:ss): 7:12.97
            Average shared text size (kbytes): 0
            Average unshared data size (kbytes): 0
            Average stack size (kbytes): 0
            Average total size (kbytes): 0
            Maximum resident set size (kbytes): 0
            Average resident set size (kbytes): 0
            Major (requiring I/O) page faults: 179
            Minor (reclaiming a frame) page faults: 489021
            Voluntary context switches: 3094
            Involuntary context switches: 2272
            Swaps: 0
            File system inputs: 38576
            File system outputs: 150872
            Socket messages sent: 0
            Socket messages received: 0
            Signals delivered: 0
            Page size (bytes): 4096
            Exit status: 0
    eric@Loki:~/checkouts/django$ git co 9449-cache-fixtures
    Switched to branch '9449-cache-fixtures'
    eric@Loki:~/checkouts/django$ /usr/bin/time -v python tests/runtests.py --settings=tests.settings -v0
    /home/eric/checkouts/django/django/contrib/admin/sites.py:455: DeprecationWarning: AdminSite.root() is deprecated; use include(admin.site.urls) instead.
    ... Snipped 4 failures ...
    Ran 1043 tests in 387.500s
    
    FAILED (failures=4)
    Command exited with non-zero status 4
            Command being timed: "python tests/runtests.py --settings=tests.settings -v0"
            User time (seconds): 363.75
            System time (seconds): 9.75
            Percent of CPU this job got: 92%
            Elapsed (wall clock) time (h:mm:ss or m:ss): 6:43.70
            Average shared text size (kbytes): 0
            Average unshared data size (kbytes): 0
            Average stack size (kbytes): 0
            Average total size (kbytes): 0
            Maximum resident set size (kbytes): 0
            Average resident set size (kbytes): 0
            Major (requiring I/O) page faults: 115
            Minor (reclaiming a frame) page faults: 489664
            Voluntary context switches: 3240
            Involuntary context switches: 1215
            Swaps: 0
            File system inputs: 40424
            File system outputs: 150544
            Socket messages sent: 0
            Socket messages received: 0
            Signals delivered: 0
            Page size (bytes): 4096
            Exit status: 4

comment:3 Changed 6 years ago by ericholscher

The failures I was seeing appear to be related to the fact that I am caching the fixture_label, and there are multiple fixtures named testdata.json and initial_data.json.

comment:4 Changed 6 years ago by ericholscher

Added another commit that fixes the aggregation tests depending on initial_data loading, which is already tested in the fixtures tests.

This also makes these tests pass with this scheme:

http://github.com/ericholscher/django/commit/08b1f3e5f98529f4b9c46c911ee00cb07ff0bdbc

comment:5 Changed 5 years ago by ericholscher

  • Has patch set
  • milestone set to 1.3
  • Needs documentation set
  • Needs tests set
  • Owner changed from nobody to ericholscher
  • Status changed from new to assigned

Going to work on this for 1.3.

Thinking it should be disablable, because for small test runs it will cause an increase in memory and time. It might be insignificant enough to not worry about though.

comment:6 Changed 4 years ago by lukeplant

  • Severity set to Normal
  • Type set to Cleanup/optimization

comment:7 Changed 4 years ago by ramiro

  • Easy pickings unset
  • Patch needs improvement set

comment:8 Changed 4 years ago by jacob

  • milestone 1.3 deleted

Milestone 1.3 deleted

comment:11 Changed 4 years ago by aaugustin

  • UI/UX unset

Change UI/UX from NULL to False.

Note: See TracTickets for help on using tickets.
Back to Top