id	summary	reporter	owner	description	type	status	component	version	severity	resolution	keywords	cc	stage	has_patch	needs_docs	needs_tests	needs_better_patch	easy	ui_ux
36339	BrokenLinkEmailsMiddleware fires when Referer is invalid and URL redirects	Xeekoo4u		"Hi all,

I discovered some behavior in the BrokenLinkEmailsMiddleware that I don't fully understand: Our Website is scraped by bots on a regular basis and one of the bots likes to use ''https://my.web.site/wp-admin'' as the referrer. The bot requests several URLs that do not exist (e.g. ''/wp-content'') which are part of the ''IGNORABLE_404_URLS'' and are ignored properly, but it also requests the ''/login'' URL that returns a 301 redirect to ''/login/''. For some reason, this request results in an email being sent if the referrer is an invalid URL (even though the actually requested URL does not return a 404).

It confuses me that I cannot find the part of the code that validates whether the referrer is valid or not. I discovered issues from almost a decade ago (https://code.djangoproject.com/ticket/26059, https://code.djangoproject.com/ticket/25971) that face a similar issue, but in their case the referrer was (almost) equal to the URL.

Is there any advice how I could handle this issue? Why is the referrer even validated and why doesn't the validation respect the ''IGNORABLE_404_URLS''? I'd be glad to remove some email spam :)"	Bug	closed	HTTP handling	5.2	Normal	invalid		Claude Paroz Jake Howard Florian Apolloner Aymeric Augustin HARI KRISHNA KANCHI	Unreviewed	0	0	0	0	0	0