Performance degradation of template rendering

[yml]

I noticed a performance regression after an upgrade from django 1.6.11 to 1.8.3 ; the project does an extensive usage of include and extends. I have simplified the problem into a mini django project on github called: ​django-perfo-regression. This project contains 2 branches (different settings) and 4 tests used to time various cases of template rendering:

• Standalone
• include
• extends
• include & extends

Depending on the settings there has been 2 major changes in the performance characteristic over the releases since django 1.4.

• 1.7 doubled the rendering time with DEBUG=True for include
• 1.8.3 degraded the simplified production ready (DEBUG=False & cached.Loader)

This ​spreadsheet contains the result I am observing on my laptop.

[Tim Graham]

Preston mentioned in IRC that a391b17ad24bc5f255a3928c23c158c79004c656 should improve performance on 1.9 -- did you check that? Identifying the commits where performance decreased would be helpful.

[yml]

I did another run with all the releases from 1.8 series and django-tip. I confirm that django-tip improves the situation but it is still slower than the releases before 1.8.3 :

	standalone	extends	include	include & extends
1.8.0	0.520450115204	0.841553926468	4.12727284431	7.55168485641
1.8.1	0.549235105515	0.868353843689	4.21892404556	7.14691209793
1.8.2	0.591598987579	0.840734004974	4.22193288803	6.90138411522
1.8.3	1.48196005821	1.79811000824	5.08307695389	8.11770391464
tip 2015-08-10	0.620449066162	0.92530298233	4.18256998062	7.9185230732

This was done using the following command in this ​repo:

tox -e py27-django-1.8.0,py27-django-1.8.1,py27-django-1.8.2,py27-django-1.8.3,py27-django-tip

[Tim Graham]

If you could bisect to find the commit where the performance change significantly, that will help assess what (if anything) can be done.

[yml]

git bisect between 1.8.2 and 1.8.3 point its finger on this commit:

b16f84f15b1344d2a3df8149565cfc8de803eb77 is the first bad commit
commit b16f84f15b1344d2a3df8149565cfc8de803eb77
Author: Tim Graham <timograham@gmail.com>
Date:   Wed May 27 09:19:19 2015 -0400

    [1.8.x] Refs #24836 -- Reverted "Simplified the lazy CSRF token implementation in csrf context processor."
    
    This reverts commit 8099d33b6553c9ee7de779ae9d191a1bf22adbda as it caused
    a regression that cannot be solved without changing force_text() which has
    a small risk of introducing regressions. This change will remain in master
    along with an update to force_text().

:040000 040000 79f27245b250a4ac0c65a4ccef0b8db107432312 6c6a96934b66615d663375cc13199cf50c6fd1ec M      django
:040000 040000 c977299c605629e42fd2ec7bb53264d5ac25db78 9eef450849404da0e8f20c292b8aa8b544176681 M      docs
:040000 040000 96fa0ceb7be7bd9ca14eded3ca98478a22ce8f39 e3c95cc6052e620319c7ef2d64a1716825839414 M      tests

Here it is a link to the commit that introduce the performance regression ​diff on github

[Tim Graham]

That optimization (added in 1.8) has been kept on master with an alternate fix (70be31bba7f8658f17235e33862319780c3dfad1) but was deemed too risky to backport to 1.8.

I am not sure what can be done to resolve this ticket absent a proposed patch. It's possible the correct fix cannot be as fast as the buggy fix. Do you have any interest in looking at it yourself? If not, I guess we can accept the ticket and see if anyone comes along to take a closer look.