Rate Limit Error Emails
|Reported by:||Andrew Wilkinson||Owned by:||nobody|
|Cc:||Simon Litchfield||Triage Stage:||Design decision needed|
|Has patch:||yes||Needs documentation:||no|
|Needs tests:||yes||Patch needs improvement:||no|
We like the emails we receive when an exception occurs on the live site. However, if an error repeatedly occurs (e.g. the database goes down) we get spammed with a huge number of emails.
I'm attaching a patch which will only send an email once every settings.ERROR_EMAIL_RATE_LIMIT minutes for each error. It determines whether it has sent an email about a particular error by taking an MD5 of the traceback and storing it in whatever cache is configured. If the same MD5 already exists in the cache, then it doesn't send an email.
The big problem with this approach is what if your cache starts to produce errors? Certainly memcached is designed so that if the server goes away all keys appear not to be set, rather than an error. I'm not sure about how the database cache backend would work in this situation.
Any suggestions on how to improve the patch are appreciated. If it's ok then I'll produce a patch for the documentation too.