Opened 9 years ago

Closed 9 years ago

Last modified 9 years ago

#2613 closed defect (fixed)

[patch] memory error in upload big sized file resolved.

Reported by: Jeong-Min Lee Owned by: adrian
Component: Core (Other) Version: master
Severity: major Keywords:
Cc: Triage Stage: Unreviewed
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: UI/UX:

Description

While running in stand alone mode, when big sized file uploaded, this error occured only in the stand alone mode so when coupled with web server this is not triggered..
Trace back like this

Original Traceback (most recent call last):
  File "C:\Python24\lib\site-packages\django\template\__init__.py", line 706, in render_node
    result = node.render(context)
  File "C:\Python24\lib\site-packages\django\template\__init__.py", line 752, in render
    output = self.filter_expression.resolve(context)
  File "C:\Python24\lib\site-packages\django\template\__init__.py", line 548, in resolve
    obj = resolve_variable(self.var, context)
  File "C:\Python24\lib\site-packages\django\template\__init__.py", line 634, in resolve_variable
    current = current[bits[0]]
  File "C:\Python24\lib\site-packages\django\http\__init__.py", line 31, in __getitem__
    for d in (self.POST, self.GET):
  File "C:\Python24\lib\site-packages\django\core\handlers\wsgi.py", line 99, in _get_post
    self._load_post_and_files()
  File "C:\Python24\lib\site-packages\django\core\handlers\wsgi.py", line 77, in _load_post_and_files
    self._post, self._files = http.parse_file_upload(header_dict, self.raw_post_data)
  File "C:\Python24\lib\site-packages\django\core\handlers\wsgi.py", line 122, in _get_raw_post_data
    self._raw_post_data = self.environ['wsgi.input'].read(int(self.environ["CONTENT_LENGTH"]))
  File "C:\Python24\lib\socket.py", line 303, in read
    data = self._sock.recv(recv_size)
MemoryError

So the diff of the django/core/handlers/wsgi.py is..

--- wsgi.py     (revision 3668)
+++ wsgi.py     (working copy)
@@ -50,6 +50,10 @@
     505: 'HTTP VERSION NOT SUPPORTED',
 }
 
+try:
+    from cStringIO import StringIO
+except ImportError:
+    from StringIO import StringIO
 class WSGIRequest(http.HttpRequest):
     def __init__(self, environ):
         self.environ = environ
@@ -119,7 +123,18 @@
         try:
             return self._raw_post_data
         except AttributeError:
-            self._raw_post_data = self.environ['wsgi.input'].read(int(self.environ["CONTENT_LENGTH"]))
+            mem = StringIO()
+            input = self.environ['wsgi.input']
+            content_length = int(self.environ["CONTENT_LENGTH"])
+            chunksize = 65536 # 64KB
+            while 1:
+                remain = content_length - mem.tell()
+                if remain <= 0: break
+                chunk = input.read(min(chunksize, remain))
+                if not chunk: break
+                mem.write(chunk)
+            self._raw_post_data = mem.getvalue()
+            mem.close()
             return self._raw_post_data
 
     GET = property(_get_get, _set_get)

Attachments (2)

wsgi.py.diff (1.1 KB) - added by Jeong-Min Lee 9 years ago.
wsgi MemoryError fix
wsgi.py-memoryerror.diff (1.4 KB) - added by Jeong-Min Lee 9 years ago.
refactored previous patch.

Download all attachments as: .zip

Change History (7)

Changed 9 years ago by Jeong-Min Lee

wsgi MemoryError fix

comment:1 Changed 9 years ago by Anon

Just wanted to say that your patch definitely works.

I had a 12 MB file that wouldn't upload at all due to a 'memory error'. I'd hit submit and the server would instantly crash.

Applied this change and there have been no problems since.

Changed 9 years ago by Jeong-Min Lee

refactored previous patch.

comment:2 Changed 9 years ago by Jeong-Min Lee

  • priority changed from normal to high
  • Severity changed from normal to major

This also affect django applications on SCGI.

comment:3 Changed 9 years ago by SmileyChris

Worked wonders for me too. Thanks Jeong-Min Lee

comment:4 Changed 9 years ago by mtredinnick

  • Resolution set to fixed
  • Status changed from new to closed

(In [3805]) Fixed #2613 -- Fixed an easily triggered memory error in file uploads for WSGI.
Thanks Jeong-Min Lee.

comment:5 Changed 9 years ago by adrian

This patch caused #2924.

Note: See TracTickets for help on using tickets.
Back to Top