Opened 2 years ago

Last modified 22 months ago

#29343 new Bug

static file serving fails with "Connection reset by peer" on HEAD requests

Reported by: Yehor Smoliakov Owned by: nobody
Component: HTTP handling Version: 2.0
Severity: Normal Keywords:
Cc: Jannik Schürg Triage Stage: Accepted
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

In developer mode I ran this command in the console:

curl --head http://127.0.0.1:8000/media/blog/author/UqdQXbTuSjW_UbGCNMhFpA.jpg

And got this exception:

[20/Apr/2018 10:36:43] "HEAD /media/blog/author/UqdQXbTuSjW_UbGCNMhFpA.jpg HTTP/1.1" 500 59
----------------------------------------
Exception happened during processing of request from ('127.0.0.1', 49750)
Traceback (most recent call last):
  File "/usr/lib/python3.6/wsgiref/handlers.py", line 138, in run
    self.finish_response()
  File "/usr/lib/python3.6/wsgiref/handlers.py", line 180, in finish_response
    self.write(data)
  File "/usr/lib/python3.6/wsgiref/handlers.py", line 279, in write
    self._write(data)
  File "/usr/lib/python3.6/wsgiref/handlers.py", line 453, in _write
    result = self.stdout.write(data)
  File "/usr/lib/python3.6/socketserver.py", line 775, in write
    self._sock.sendall(b)
ConnectionResetError: [Errno 104] Connection reset by peer

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.6/wsgiref/handlers.py", line 141, in run
    self.handle_error()
  File "/home/yehor/.local/share/virtualenvs/webapp-Dlno2ZDu/lib/python3.6/site-packages/django/core/servers/basehttp.py", line 86, in handle_error
    super().handle_error()
  File "/usr/lib/python3.6/wsgiref/handlers.py", line 368, in handle_error
    self.finish_response()
  File "/usr/lib/python3.6/wsgiref/handlers.py", line 180, in finish_response
    self.write(data)
  File "/usr/lib/python3.6/wsgiref/handlers.py", line 274, in write
    self.send_headers()
  File "/usr/lib/python3.6/wsgiref/handlers.py", line 331, in send_headers
    if not self.origin_server or self.client_is_modern():
  File "/usr/lib/python3.6/wsgiref/handlers.py", line 344, in client_is_modern
    return self.environ['SERVER_PROTOCOL'].upper() != 'HTTP/0.9'
TypeError: 'NoneType' object is not subscriptable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.6/socketserver.py", line 639, in process_request_thread
    self.finish_request(request, client_address)
  File "/usr/lib/python3.6/socketserver.py", line 361, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/usr/lib/python3.6/socketserver.py", line 696, in __init__
    self.handle()
  File "/home/yehor/.local/share/virtualenvs/webapp-Dlno2ZDu/lib/python3.6/site-packages/django/core/servers/basehttp.py", line 154, in handle
    handler.run(self.server.get_app())
  File "/usr/lib/python3.6/wsgiref/handlers.py", line 144, in run
    self.close()
  File "/usr/lib/python3.6/wsgiref/simple_server.py", line 35, in close
    self.status.split(' ',1)[0], self.bytes_sent
AttributeError: 'NoneType' object has no attribute 'split'
----------------------------------------

Change History (7)

comment:1 Changed 2 years ago by Tim Graham

Resolution: needsinfo
Status: newclosed
Summary: An exception AttributeError during HEAD request on media objectstatic file serving fails with "Connection reset by peer" on HEAD requests

I don't think this is a bug in Django -- the traceback comes from Python's wsgiref. Feel free to reopen if you can explain why Django is at fault.

comment:2 Changed 2 years ago by mandm

Resolution: needsinfo
Status: closednew

Similar problem with HEAD requests after upgrading to Django 1.10.8 (1.8.x & 1.9.x work)

curl -I http://localhost:8000/ 

gives this error

Traceback (most recent call last):
  File "/usr/lib/python2.7/wsgiref/handlers.py", line 86, in run
    self.finish_response()
  File "/usr/lib/python2.7/wsgiref/handlers.py", line 128, in finish_response
    self.write(data)
  File "/usr/lib/python2.7/wsgiref/handlers.py", line 217, in write
    self._write(data)
  File "/usr/lib/python2.7/socket.py", line 328, in write
    self.flush()
  File "/usr/lib/python2.7/socket.py", line 307, in flush
    self._sock.sendall(view[write_offset:write_offset+buffer_size])
error: [Errno 104] Connection reset by peer
ERROR: 2018-06-16 11:53:28,639 - "HEAD / HTTP/1.1" 500 59

comment:3 Changed 2 years ago by Claude Paroz

This happens when streaming a FileResponse through a wsgiref.FileWrapper instance, when the file is bigger than the streaming block size. curl is closing the connection when it receives the first chunk, and the error then happens when FileWrapper is writing the second chunk to the socket.
I cannot yet state about a possible culprit in that scenario.

comment:4 Changed 2 years ago by Tim Graham

Triage Stage: UnreviewedAccepted

Tentatively accept for further investigation.

comment:5 Changed 22 months ago by Jannik Schürg

I can not reproduce this. I tried Python 3.6.4 and 2.7.6 with Django 1.10.8 and 2.0 on Mac OS. The setup was a new project with media serving and a 50MB jpg file.

Could you provide an example project or more information in order to reproduce the error?

I got

[2018-11-07 05:07:39,626] - Broken pipe from ('127.0.0.1', 63603)

but no exception.

comment:6 Changed 22 months ago by Jannik Schürg

Cc: Jannik Schürg added

comment:7 Changed 22 months ago by Jannik Schürg

I could reproduce it now a few times but not consistently, it is a timing issue maybe. Or my version of curl is more hesitant with TCP RST (leading to ConnectionResetError).

Here is what happens, I think:

  1. Curl closes the connection by resetting it instead of closing it (RST vs. FIN).
  2. An exception is thrown in wsgiref/socket.
  3. wsgiref.handlers.BaseHandler.close() is called, which reset the state of the handler (in particular self.environ is set to None).
  4. The error handler handle_error() is called. Here Django has overwritten the method and checks if is_broken_pipe_error() in which case nothing would happen.
  5. But curl did reset instead of close, so we have not "broken pipe," but "connection reset". The default handler is called, which tries to use self.environ, exception.

A proper fix probably would be to modify core.servers.basehttp.ServerHandler by adding the method

def finish_response(self):
    try:
        if self.environ['REQUEST_METHOD'] == 'HEAD':
            self.finish_content()
        else:
            if not self.result_is_file() or not self.sendfile():
                for data in self.result:
                    self.write(data)
                self.finish_content()
    finally:
        self.close()

With this the server does no longer try to send the body for a HEAD request (as it should be).
I am sure there are cases where this does produce an incorrect HEAD response according to the HTTP standard, but at least no body is sent anymore.

Alternatively/additionally one might want to change the check with is_broken_pipe(). For example one could also check for connection reset, or if self.environ is not None, or if the connection was closed (overwrite close() method).

Note: See TracTickets for help on using tickets.
Back to Top