Opened 12 months ago

Closed 11 months ago

Last modified 11 months ago

#29849 closed Bug (fixed)

runserver crashes with "Remote end closed connection without response"

Reported by: rvernica Owned by: nobody
Component: HTTP handling Version: 2.1
Severity: Normal Keywords:
Cc: Jaap Roes Triage Stage: Ready for checkin
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

I created an app like in the tutorial and started Django. When I issue successive requests, the session gets disconnected. Here is an example:

>>> s = requests.Session()
>>> s.get('http://localhost:8000/'); s.get('http://localhost:8000')
<Response [404]>
Traceback (most recent call last):
  File "...local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "...local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 384, in _make_request
    six.raise_from(e, None)
  File "<string>", line 2, in raise_from
  File "...local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 380, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/lib64/python3.6/http/client.py", line 1331, in getresponse
    response.begin()
  File "/usr/lib64/python3.6/http/client.py", line 297, in begin
    version, status, reason = self._read_status()
  File "/usr/lib64/python3.6/http/client.py", line 266, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "...local/lib/python3.6/site-packages/requests/adapters.py", line 445, in send
    timeout=timeout
  File "...local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 638, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "...local/lib/python3.6/site-packages/urllib3/util/retry.py", line 367, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "...local/lib/python3.6/site-packages/urllib3/packages/six.py", line 685, in reraise
    raise value.with_traceback(tb)
  File "...local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "...local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 384, in _make_request
    six.raise_from(e, None)
  File "<string>", line 2, in raise_from
  File "...local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 380, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/lib64/python3.6/http/client.py", line 1331, in getresponse
    response.begin()
  File "/usr/lib64/python3.6/http/client.py", line 297, in begin
    version, status, reason = self._read_status()
  File "/usr/lib64/python3.6/http/client.py", line 266, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "...local/lib/python3.6/site-packages/requests/sessions.py", line 525, in get
    return self.request('GET', url, **kwargs)
  File "...local/lib/python3.6/site-packages/requests/sessions.py", line 512, in request
    resp = self.send(prep, **send_kwargs)
  File "...local/lib/python3.6/site-packages/requests/sessions.py", line 622, in send
    r = adapter.send(request, **kwargs)
  File "...local/lib/python3.6/site-packages/requests/adapters.py", line 495, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',))

It I have a small timeout between the requests it works fine:

>>> s = requests.Session()
>>> s.get('http://localhost:8000/'); time.sleep(1); s.get('http://localhost:8000')
<Response [404]>
<Response [404]>

It also works fine if I don't use sessions:

>>> requests.get('http://localhost:8000/'); requests.get('http://localhost:8000')
<Response [404]>
<Response [404]>

I initially reported this as a bug in the requests library but it does not appear to be the case. See discussion here https://github.com/requests/requests/issues/4784

Change History (14)

comment:1 Changed 12 months ago by Herbert Fortes

Hi,

I do not have this error here.

I have 15 unapplied migration(s), the teste/urls.py has the include, the core/views.py app
has only the HttpResponse.

Only 'python manage.py runserver'. I got two 404.

Maybe there is a step that I am missing?

comment:2 Changed 12 months ago by Herbert Fortes

certifi==2018.10.15
chardet==3.0.4
Django==2.1.2
idna==2.7
pytz==2018.5
requests==2.19.1
urllib3==1.23

python 3.6.6

comment:3 Changed 12 months ago by Tim Graham

Component: UncategorizedHTTP handling
Resolution: needsinfo
Status: newclosed
Type: UncategorizedBug

I also can't reproduce. I searched past bugs for "Remote end closed connection without response" and found #28440 but this was fixed in Django 2.0. I also failed to reproduce this with Django 1.11.

comment:4 Changed 11 months ago by Jaap Roes

Cc: Jaap Roes added
Resolution: needsinfo
Status: closednew

I was also hit by this and found a reliable way to reproduce this:

First create a Dockerfile with the following content:

FROM python:3

RUN mkdir /test
WORKDIR /test
RUN pip install requests django pyinotify
RUN django-admin startproject tmp .
RUN echo '\
import requests\n\
with requests.Session() as s:\n\
  for _ in range(5):\n\
    print(s.get("http://localhost:8000/").status_code)\n'\
 > test.py
CMD /bin/bash -c './manage.py runserver 0:8000 & sleep 5 && python test.py'

Now run:

docker build -t connection-error .
docker run -it --rm connection-error

This reliably fails with:

requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

When I remove pyinotify the test script doesn't fail.

So there seems to be some weird interaction with pyinotify, runserver and a requests.Session.

Last edited 11 months ago by Jaap Roes (previous) (diff)

comment:5 in reply to:  4 Changed 11 months ago by Jaap Roes

After some further testing I can also reproduce the issue (less reliably) without pyinotify by increasing the amount of requests (e.g. for _ in range(50)). So I guess it's a more general issue caused by the reloader?

EDIT: Running the devserver with the --noreload flag (and without installing pyinotify) still causes the error to show. Only when I disable threading (--nothreading) the issue goes away, unless pyinotify is installed then both --nothreading and --noreload need to be used.

Last edited 11 months ago by Jaap Roes (previous) (diff)

comment:6 Changed 11 months ago by Tim Graham

I guess the question remains whether Django is at fault or if there's some bug in Python or elsewhere.

comment:8 in reply to:  6 Changed 11 months ago by Jaap Roes

I did some further testing, and the test script I created works with Django 1.11 *and* the current master. So I ran a git bisect and saw that this commit has fixed the issue. Is it possible to backport this fix?

comment:9 Changed 11 months ago by Tim Graham

comment:10 in reply to:  9 Changed 11 months ago by Jaap Roes

I can confirm the test script works fine on e6065c7b8363202c5eb13ba10c97a8c24d014b45 and fails on ac756f16c5bbbe544ad82a8f3ab2eac6cccdb62e, so the latter commit is the root cause of this issue.

EDIT: pasted the correct commit hashes

Last edited 11 months ago by Jaap Roes (previous) (diff)

comment:11 Changed 11 months ago by Tim Graham

Has patch: set
Summary: Remote end closed connection without responserunserver crashes with "Remote end closed connection without response"
Triage Stage: UnreviewedReady for checkin

comment:12 Changed 11 months ago by Tim Graham <timograham@…>

Resolution: fixed
Status: newclosed

In e1721ece:

[2.1.x] Fixed #29849 -- Fixed keep-alive support in runserver.

Ticket #25619 changed the default protocol to HTTP/1.1 but did not
properly implement keep-alive. As a "fix" keep-alive was disabled in
ticket #28440 to prevent clients from hanging (they expect the server to
send more data if the connection is not closed and there is no content
length set).

The combination of those two fixes resulted in yet another problem:
HTTP/1.1 by default allows a client to assume that keep-alive is
supported unless the server disables it via 'Connection: close' -- see
RFC2616 8.1.2.1 for details on persistent connection negotiation. Now if
the client receives a response from Django without 'Connection: close'
and immediately sends a new request (on the same tcp connection) before
our server closes the tcp connection, it will error out at some point
because the connection does get closed a few milli seconds later.

This patch fixes the mentioned issues by always sending 'Connection:
close' if we cannot determine a content length. The code is inefficient
in the sense that it does not allow for persistent connections when
chunked responses are used, but that should not really cause any
problems (Django does not generate those) and it only affects the
development server anyways.

Refs #25619, #28440.

Regression in ac756f16c5bbbe544ad82a8f3ab2eac6cccdb62e.
Backport of 934acf1126995f6e6ccba5947ec8f7561633c27f from master.

comment:13 Changed 11 months ago by Tim Graham <timograham@…>

In 5d327a63:

Refs #29849 -- Forwardported 2.1.4 release note.

comment:14 Changed 11 months ago by Jaap Roes

Thanks Tim!

I'm still wondering why it's so easy to trigger when pyinotify is installed. If anyone can explain that to me, I'd love to know :)

Note: See TracTickets for help on using tickets.
Back to Top