Optimize template compilation with intelligent caching for repeated renders

[TheAshutoshMishra]

Currently, Django's template system performs expensive compilation operations (tokenization and filter expression parsing) on every template render, even when rendering the same template repeatedly with different contexts. This causes unnecessary CPU overhead in production.

Problem

The template compilation process involves:

1. Tokenization: Regex splitting of template strings via tag_re.split()
2. Filter expression compilation: Parsing filter syntax like {{ var|filter:"arg" }}

Both operations are repeated identically for every render, even though the results are deterministic based on the template source.

Proposed Solution

Add bounded, memory-safe caching for:

1. Tokenized template strings (max 1000 entries, hash-based keys)
2. Compiled FilterExpression objects (max 500 entries)

Implementation Details

• Hash-based cache keys for O(1) lookups
• Smart eviction: removes 20-25% of oldest entries when limit reached
• Skips caching for stateful verbatim templates
• Graceful fallbacks for edge cases
• New clear_template_caches() utility for production cache management

Performance Impact (Measured)

• 13-47% faster template rendering on repeated renders
• 1.15M tokenizations/second throughput with caching
• Zero configuration required
• Fully backward compatible

PR

Implementation available in PR #20540: ​https://github.com/django/django/pull/20540

Looking for feedback on this approach and guidance on adding comprehensive test coverage.

[Natalia Bidart]

Hello TheAshutoshMishra,

New features proposals must first go through Django's new feature proposal process and be discussed with the community. To do that, please raise this on the ​new feature tracker.

Also note that when using an LLM to assist with a contribution, you are responsible for ensuring it fully reads and follows Django's documentation, including docs/internals/contributing/ in the repo, or ​the online docs. Submissions that do not meet those requirements cannot be reviewed in Trac nor in GitHub.