Code

Opened 7 years ago

Closed 6 years ago

#5596 closed (fixed)

"No buffer space available" error when downloading file

Reported by: anonymous Owned by: nobody
Component: HTTP handling Version: master
Severity: Keywords:
Cc: Triage Stage: Ready for checkin
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: yes
Easy pickings: UI/UX:

Description

Development server with django.views.static.serve route, file has 235949884 bytes. When trying to access the file, I get the following error:

Traceback (most recent call last):
  File "D:\Python25\lib\site-packages\django\core\servers\basehttp.py", line 279, in run 
    self.finish_response()
  File "D:\Python25\lib\site-packages\django\core\servers\basehttp.py", line 318, in finish_response
    self.write(data)
  File "D:\Python25\lib\site-packages\django\core\servers\basehttp.py", line 402, in write
    self._write(data)
  File "D:\Python25\lib\socket.py", line 263, in write
    self.flush()
  File "D:\Python25\lib\socket.py", line 250, in flush
    self._sock.sendall(buffer)
error: (10055, 'No buffer space available')

Django seems to be able to read and write the file. In flush() in socket.py the final buffer to flush has the correct size. Please tell me that this isn't a general Python error. It should be possible to write a 230MB file into a socket. Still, it might be better if serve() would *not* try to read in everything at once and then write everything but stream the file in smaller chunks.

Attachments (1)

static.diff (2.8 KB) - added by eibaan 7 years ago.
chunk large files, make application/octet-stream the default

Download all attachments as: .zip

Change History (7)

comment:1 Changed 7 years ago by calvin@…

  • Needs documentation unset
  • Needs tests unset
  • Patch needs improvement unset

Do you mean Django attempts to write ~230MB of data at once via socket.sendall()? That would indeed be a bad idea. You should split the data up in smaller chunks (say 64KB or so) before writing to a network socket. I notice that such a comment is already in basehttp.py:402, so that might be a known problem.

comment:2 Changed 7 years ago by anonymous

Yes, but it's not me but the django.views.static.serve() function. I created a patch (including test cases) to work around the problem. Hopefully, this problem will not occur on a production server which will not use the above function to serve static files. Still, it should work in development mode, too.

When writing the test, I noticed that files with unknown file extensions are served as text/html. A better default would IMHO be application/octet-stream, so I changed this, too. As both changes are in the same files, I didn't really know how to split the patch.

Changed 7 years ago by eibaan

chunk large files, make application/octet-stream the default

comment:3 Changed 6 years ago by mtredinnick

Why have you decided 225 is the right chunk size here?

comment:4 Changed 6 years ago by eibaan

It seems to be a nice value :) 32 MB is large enough so that for typical usage the optimization doesn't kick in and we stay backward compatible. Uploading 32 MB is still fast and doesn't seem to stress the Python garbage collector too much. Using 16 MB or even less as threshold would be fine with me...

comment:5 Changed 6 years ago by Simon G <dev@…>

  • Has patch set
  • Patch needs improvement set
  • Triage Stage changed from Unreviewed to Ready for checkin

Marked as RFC, but might need a few tweaks to sizing etc as per Malcolm's comment

comment:6 Changed 6 years ago by mtredinnick

  • Resolution set to fixed
  • Status changed from new to closed

(In [6939]) Fixed #5596 -- Changed the static view for the development server so that Django doesn't crash if somebody tries to serve a 200MB file. Patch from eibaan.

Add Comment

Modify Ticket

Change Properties
<Author field>
Action
as closed
as The resolution will be set. Next status will be 'closed'
The resolution will be deleted. Next status will be 'new'
Author


E-mail address and user name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.