Optimize template compilation with intelligent caching for repeated renders

[TheAshutoshMishra]

Currently, Django's template system performs expensive compilation operations (tokenization and filter expression parsing) on every template render, even when rendering the same template repeatedly with different contexts. This causes unnecessary CPU overhead in production.

Problem

The template compilation process involves:

1. Tokenization: Regex splitting of template strings via tag_re.split()
2. Filter expression compilation: Parsing filter syntax like {{ var|filter:"arg" }}

Both operations are repeated identically for every render, even though the results are deterministic based on the template source.

Proposed Solution

Add bounded, memory-safe caching for:

1. Tokenized template strings (max 1000 entries, hash-based keys)
2. Compiled FilterExpression objects (max 500 entries)

Implementation Details

• Hash-based cache keys for O(1) lookups
• Smart eviction: removes 20-25% of oldest entries when limit reached
• Skips caching for stateful verbatim templates
• Graceful fallbacks for edge cases
• New clear_template_caches() utility for production cache management

Performance Impact (Measured)

• 13-47% faster template rendering on repeated renders
• 1.15M tokenizations/second throughput with caching
• Zero configuration required
• Fully backward compatible

PR

Implementation available in PR #20540: ​https://github.com/django/django/pull/20540

Looking for feedback on this approach and guidance on adding comprehensive test coverage.

[Natalia Bidart]

Identical duplicate of #36866. Please avoid use of LLM to fully write tickets and PRs. If an LLM is used as an assistance, please carefully review its outcome and also replace the overly verbose output with concise, correct, and precise information.