Thread safe template cache

[German M. Bravo]

I've come across a problem that pops up some times, when a thread clears the cache by resetting the cached templates loader while other is loading it.
I made a patch for this.

[Florian Apolloner]

#19511 seems to be a duplicate but it contains more info so I'll close this one.

[Florian Apolloner]

Doesn't seem to be a duplicate, only the patch seems to be the same which wouldn't make sense. Can you get into more details about the error?

[Aymeric Augustin]

Under normal use of Django (ie. outside tests), when does the cached template loader's cache get reset?

[German M. Bravo]

I added a cleaner patch (the old one had bits of other patch in it). I'm not, however, aware of anywhere where the cache might get reset, I looked around and couldn't find any place in my own code either, but the KeyError exception was happening, and after these days the patch completely fixed it.

I'm adding the correct patch to the other issue (#19511) as well.

[Anssi Kääriäinen]

To me it seems the patch is changing the semantics of load_template(). The origin is cached and returned always while that isn't the case before. There is no explanation if or why the change is needed.

However, there is a clear race in there - we check if the key exists in the dict, then assume it still exists a couple of lines later. So, clear() in between will cause KeyError.

Marking as accepted based on the race. Testing this seems a little complex, as we will need to time the reset() very accurately to hit this problem - so I don't think a test is absolutely necessary.

[regebro]

Simpler way to avoid race.

[regebro]

As mentioned it's near impossible to test race conditions, so I don't have a test. But I have attached a simpler fix for the race condition, by avoiding the separation of test and return. It would be good if Kronuz could test this.

Try-Except should be slower though, but unless you are rendering thousands of small templates in one view that should not make a measurable difference, but only performance tests can determine that.

I made a branch as well: ​https://github.com/regebro/django/tree/issue-19510

[Aymeric Augustin <aymeric.augustin@…>]

In 68905695b897e62b0c18d9edd87171a0eae4e67e:

Fixed #19510 -- Race condition in template loading.

Thanks Kronuz and regebro.