HEAD Responses Drop Headers

[Paul Bailey]

When using runserver headers are dropped for head requests, in particular content-length. Because my HEAD request is serving a large body content, I do not include it in the body since it is just dropped.

Headers from runserver:

Headers({'date': 'Tue, 19 Dec 2023 12:52:23 GMT', 'server': 'WSGIServer/0.2 CPython/3.11.2', 'accept-ranges': 'bytes', 'content-type': 
'text/html; charset=utf-8', 'x-frame-options': 'DENY', 'x-content-type-options': 'nosniff', 'referrer-policy': 'same-origin', 'cross-or
igin-opener-policy': 'same-origin'})

Headers from uvicorn asgi server:

Headers({'date': 'Tue, 19 Dec 2023 12:54:49 GMT', 'server': 'uvicorn', 'accept-ranges': 'bytes', 'content-length': '121283919', 'content-type': 'text/html; charset=utf-8', 'x-frame-options': 'DENY', 'x-content-type-options': 'nosniff', 'referrer-policy': 'same-origin', 'cross-origin-opener-policy': 'same-origin'})

Notice the uvicorn properly includes the content-length header that was set in the view.

View source snippet:

if request.method == 'HEAD':
    print('HEAD Request')
    return HttpResponse(headers={
        'Accept-Ranges': 'bytes',
        'Content-Length': str(file_size)
      })

[Natalia Bidart]

I did some research on the topic and from ​the corresponding RFC it seems that this report is valid and should be accepted:

A server MAY send a Content-Length header field in a response to a HEAD request (Section 9.3.2); a server MUST NOT send Content-Length in such a response unless its field value equals the decimal number of octets that would have been sent in the content of a response if the same request had used the GET method.

The removal of the Content-Length seems to be located in this code:

django/core/servers/basehttp.py

@@ -131 +131 @@
 
     def cleanup_headers(self):
         super().cleanup_headers()
-        if (
-            self.environ["REQUEST_METHOD"] == "HEAD"
-            and "Content-Length" in self.headers
-        ):
-            del self.headers["Content-Length"]
         # HTTP/1.1 requires support for persistent connections. Send 'close' if
         # the content length is unknown to prevent clients from reusing the
         # connection.

This code was added while fixing ticket #28054, and while it's correct not to return the body of the response, it seems that the Content-Length should be kept.

Regression in 8acc433e415cd771f69dfe84e57878a83641e78b

[Mariusz Felisiak]

I don't agree, we intentionally drop the Content-Length. Please check the entire discussion in PR, e.g. this ​comment. As far as I'm aware, the current implementation is RFC compliant. I'd mark this ticket as invalid.

[Paul Bailey]

The use case for this is that your GET request has a large body and so you do not want to have to produce two large bodies, one for the HEAD, one for GET. Instead you want to match the exact headers a GET request would have without producing the body content for the HEAD request. This is essential for producing HTTP Range Requests in Django without a lot of overhead.

​https://developer.mozilla.org/en-US/docs/Web/HTTP/Range_requests

As per the docs above, a HTTP HEAD should return Content-Length

[Paul Bailey]

The discussion linked is over Content-Length being returned for the body of the HEAD response, so this is always "Content-Length: 0" which doesn't match the GET request.

The easiest thing to do at the time was to just remove Content-Length since it is not required. However, for HTTP Range Requests Content-Length is required.

So I think the proper thing to do is to allow it if Content-Length is not Zero. If the header is not 0 then that means it was set by the user and should be allowed through.

[Paul Bailey]

something like:

    def cleanup_headers(self):
        super().cleanup_headers()
        if (
            self.environ["REQUEST_METHOD"] == "HEAD"
            and "Content-Length" in self.headers
            and str(self.headers["Content-Length"]) ==  "0"
        ):
            del self.headers["Content-Length"]

[Mariusz Felisiak]

As for me this is a new feature request for supporting HTTP ranges.

[Paul Bailey]

Replying to Mariusz Felisiak:

As for me this is a new feature request for supporting HTTP ranges.

I would imagine Content-Length in HEAD requests is also needed for other streaming mechanisms.

[Natalia Bidart]

Replying to Mariusz Felisiak:

I don't agree, we intentionally drop the Content-Length. Please check the entire discussion in PR, e.g. this ​comment. As far as I'm aware, the current implementation is RFC compliant. I'd mark this ticket as invalid.

Thank you Mariusz for the pointer to the specific message from Nick, it provides a very complete and clear reasoning for the change.

With that in mind, I agree that this is not a valid bug. Additionally, I agree that the optional return of Content-Length for HEAD requests, enabling specific use cases, should be approached as a new feature, which should be presented and discussed in the ​Django Forum (following ​the documented guidelines for requesting features). Paul, would you be willing to start a new topic explaining the current situation and outlining potential use cases for the feature?

[Paul Bailey]

sounds good

[Paul Bailey]

Reopening based on discussion at: ​https://forum.djangoproject.com/t/optionally-do-not-drop-content-length-for-head-requests/26305

[Paul Bailey]

PR ready for review: ​https://github.com/django/django/pull/17643#issuecomment-1869135570

[Mariusz Felisiak]

You cannot accept your own tickets.

[HAMA Barhamou]

Hello, after reading all the comments on the tiket, I think we can accept it and start revising the proposed path. Merry Christmas to all Christians

[GitHub <noreply@…>]

In 9d52e072:

Fixed #35051 -- Prevented runserver from removing non-zero Content-Length for HEAD requests.