Opened 12 years ago

Closed 12 years ago

Last modified 11 years ago

#2613 closed defect (fixed)

[patch] memory error in upload big sized file resolved.

Reported by: Jeong-Min Lee Owned by: Adrian Holovaty
Component: Core (Other) Version: master
Severity: major Keywords:
Cc: Triage Stage: Unreviewed
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: UI/UX:


While running in stand alone mode, when big sized file uploaded, this error occured only in the stand alone mode so when coupled with web server this is not triggered..
Trace back like this

Original Traceback (most recent call last):
  File "C:\Python24\lib\site-packages\django\template\", line 706, in render_node
    result = node.render(context)
  File "C:\Python24\lib\site-packages\django\template\", line 752, in render
    output = self.filter_expression.resolve(context)
  File "C:\Python24\lib\site-packages\django\template\", line 548, in resolve
    obj = resolve_variable(self.var, context)
  File "C:\Python24\lib\site-packages\django\template\", line 634, in resolve_variable
    current = current[bits[0]]
  File "C:\Python24\lib\site-packages\django\http\", line 31, in __getitem__
    for d in (self.POST, self.GET):
  File "C:\Python24\lib\site-packages\django\core\handlers\", line 99, in _get_post
  File "C:\Python24\lib\site-packages\django\core\handlers\", line 77, in _load_post_and_files
    self._post, self._files = http.parse_file_upload(header_dict, self.raw_post_data)
  File "C:\Python24\lib\site-packages\django\core\handlers\", line 122, in _get_raw_post_data
    self._raw_post_data = self.environ['wsgi.input'].read(int(self.environ["CONTENT_LENGTH"]))
  File "C:\Python24\lib\", line 303, in read
    data = self._sock.recv(recv_size)

So the diff of the django/core/handlers/ is..

---     (revision 3668)
+++     (working copy)
@@ -50,6 +50,10 @@
+    from cStringIO import StringIO
+except ImportError:
+    from StringIO import StringIO
 class WSGIRequest(http.HttpRequest):
     def __init__(self, environ):
         self.environ = environ
@@ -119,7 +123,18 @@
             return self._raw_post_data
         except AttributeError:
-            self._raw_post_data = self.environ['wsgi.input'].read(int(self.environ["CONTENT_LENGTH"]))
+            mem = StringIO()
+            input = self.environ['wsgi.input']
+            content_length = int(self.environ["CONTENT_LENGTH"])
+            chunksize = 65536 # 64KB
+            while 1:
+                remain = content_length - mem.tell()
+                if remain <= 0: break
+                chunk =, remain))
+                if not chunk: break
+                mem.write(chunk)
+            self._raw_post_data = mem.getvalue()
+            mem.close()
             return self._raw_post_data
     GET = property(_get_get, _set_get)

Attachments (2) (1.1 KB) - added by Jeong-Min Lee 12 years ago.
wsgi MemoryError fix (1.4 KB) - added by Jeong-Min Lee 12 years ago.
refactored previous patch.

Download all attachments as: .zip

Change History (7)

Changed 12 years ago by Jeong-Min Lee

Attachment: added

wsgi MemoryError fix

comment:1 Changed 12 years ago by Anon

Just wanted to say that your patch definitely works.

I had a 12 MB file that wouldn't upload at all due to a 'memory error'. I'd hit submit and the server would instantly crash.

Applied this change and there have been no problems since.

Changed 12 years ago by Jeong-Min Lee

Attachment: added

refactored previous patch.

comment:2 Changed 12 years ago by Jeong-Min Lee

priority: normalhigh
Severity: normalmajor

This also affect django applications on SCGI.

comment:3 Changed 12 years ago by Chris Beaven

Worked wonders for me too. Thanks Jeong-Min Lee

comment:4 Changed 12 years ago by Malcolm Tredinnick

Resolution: fixed
Status: newclosed

(In [3805]) Fixed #2613 -- Fixed an easily triggered memory error in file uploads for WSGI.
Thanks Jeong-Min Lee.

comment:5 Changed 11 years ago by Adrian Holovaty

This patch caused #2924.

Note: See TracTickets for help on using tickets.
Back to Top