BrokenLinkEmailsMiddleware shouldn't report 404s when Referer = URL

[Aymeric Augustin]

Many dubious bots send a Referer header that is equal to the current URL, presumably to work around checks for empty Referer headers. Some of these bots are also poorly implemented and trigger a stupid amount of 404s.

BrokenLinkEmailsMiddleware is smart enough not to report 404s without a Referer. I suggest to make it not report 404s when the Referer it equal to the current URL either.

[Aymeric Augustin]

Just to be clear: a 404 with a Referer equal to the current URL must be a false positive, because there's no way you can be coming from a page that doesn't exist.

If the page disappeared between the previous and the current request, then the problem is solved, since the page containing the broken link no longer exists.

[Maxime Lorant]

Should I update the documentation? It is already written that empty referer are ignored, I think we could add something about this new behaviour too...

See ​https://docs.djangoproject.com/en/1.8/howto/error-reporting/#errors

[Aymeric Augustin]

Yes, you should update the documentation as well.

[Tim Graham <timograham@…>]

In 4ce433e:

Fixed #25302 -- Prevented BrokenLinkEmailsMiddleware from reporting 404s when Referer = URL.

[Aymeric Augustin]

I ran 1.9 RC 1 in production for a few days and sadly, the fix doesn't yield the results I hoped for.

Broken bots are hardcoded to use http://<domain><url> as referer. However Django's check is sensitive to the scheme. Since I run on https://... the condition added to fix this ticket here never triggers.

Would it make sense to also ignore the scheme in the check?

[Aymeric Augustin]

I submitted a pull request: ​https://github.com/django/django/pull/5730.

I would like to backport it to 1.9 before it's released.

[Maxime Lorant]

I don't mind ignoring the scheme indeed, +100 for the PR :) Did not think of it but it is clearly something that should be ignored.

[Aymeric Augustin]

One hour after posting this PR, I received 160 emails like the following in 8 minutes:

Referrer: http://REDACTED.com/libraries/joomla/html/language/en-GB/en-GB.jhtmldate.ini
Requested URL: /libraries/joomla/html/language/en-GB/en-GB.jhtmldate.ini
User agent: Go 1.1 package http
IP address:  162.158.56.47

I'm wondering if anyone besides me actually uses this in production...

[Aymeric Augustin <aymeric.augustin@…>]

In 11f10b7:

Fixed #25302 (again) -- Ignored scheme when checking for bad referers.

The check introduced in 4ce433e was too strict in real life. The poorly
implemented bots this patch attempted to ignore are sloppy when it comes
to http vs. https.

[Aymeric Augustin <aymeric.augustin@…>]

In 8dc11dc5:

[1.9.x] Fixed #25302 (again) -- Ignored scheme when checking for bad referers.

The check introduced in 4ce433e was too strict in real life. The poorly
implemented bots this patch attempted to ignore are sloppy when it comes
to http vs. https.

Backport of 11f10b7 from master