URLField validation should use HEAD requet instead of GET

[Claude Paroz]

When an URLField has verify_exists to True, it checks that the page exists with urrlib2.urlopen. Doing a GET request (urllib2 default) is not necessary whereas a simple HEAD request should be sufficient to know if the page exists.

[Claude Paroz]

Use HEAD HTTP request instead of GET

[Claude Paroz]

Note that tests are already done in forms/tests/fields.py (e.g. test_urlfield_3).

[rasca]

I've applied it cleanly, tested it in a demo app and run the tests OK.
Marking it as RFC.

[Graeme]

What about websites that don't allow HEAD request? There are some sites that only allow GET and POST.
How about HEAD first fall back to GET?

[Claude Paroz]

Yeah, you are probably right. Even if it adds a supplementary request for each not found resource, a fallback to a GET request is probably more robust.

[Luke Plant]

​RFC 2616 says that all general purpose servers MUST support both GET and HEAD. However, I don't think it hurts to fall back to GET, but only if the server returns a ​501 code or a ​405 code.

[Claude Paroz]

Thanks Luke for the useful insight. The patch now falls back to GET when response code is 405 or 501.

[Claude Paroz]

Add fallback to GET

[Alex Gaynor]

Class should not be created inside the method; classes are some of the most expensive (memorywise) objects in CPython, and can be slow to gc.

[Claude Paroz]

Put HeadRequest class at root level

[Claude Paroz]

Thanks Alex for your input. I've modified the patch to get HeadRequest class out of the method.

[Jannis Leidel]

In [15500]:

Fixed #14955 -- Made the URLValidator use a HEAD request when verifying a URL. Thanks, Claude Paroz.

[jamstooks <ben.stookey@…>]

After thinking about this, I wonder if it's worth falling back to GET for any failed request? I've found a variety of IIS servers running ASP.NET that are responding with 403 errors or simply timing out on HEAD requests. Any thoughts? Should I open another ticket about this?

[Jacob]

Milestone 1.3 deleted