[patch] memory error in upload big sized file resolved.

[Jeong-Min Lee]

While running in stand alone mode, when big sized file uploaded, this error occured only in the stand alone mode so when coupled with web server this is not triggered..
Trace back like this

Original Traceback (most recent call last):
  File "C:\Python24\lib\site-packages\django\template\__init__.py", line 706, in render_node
    result = node.render(context)
  File "C:\Python24\lib\site-packages\django\template\__init__.py", line 752, in render
    output = self.filter_expression.resolve(context)
  File "C:\Python24\lib\site-packages\django\template\__init__.py", line 548, in resolve
    obj = resolve_variable(self.var, context)
  File "C:\Python24\lib\site-packages\django\template\__init__.py", line 634, in resolve_variable
    current = current[bits[0]]
  File "C:\Python24\lib\site-packages\django\http\__init__.py", line 31, in __getitem__
    for d in (self.POST, self.GET):
  File "C:\Python24\lib\site-packages\django\core\handlers\wsgi.py", line 99, in _get_post
    self._load_post_and_files()
  File "C:\Python24\lib\site-packages\django\core\handlers\wsgi.py", line 77, in _load_post_and_files
    self._post, self._files = http.parse_file_upload(header_dict, self.raw_post_data)
  File "C:\Python24\lib\site-packages\django\core\handlers\wsgi.py", line 122, in _get_raw_post_data
    self._raw_post_data = self.environ['wsgi.input'].read(int(self.environ["CONTENT_LENGTH"]))
  File "C:\Python24\lib\socket.py", line 303, in read
    data = self._sock.recv(recv_size)
MemoryError

So the diff of the django/core/handlers/wsgi.py is..

--- wsgi.py     (revision 3668)
+++ wsgi.py     (working copy)
@@ -50,6 +50,10 @@
     505: 'HTTP VERSION NOT SUPPORTED',
 }
 
+try:
+    from cStringIO import StringIO
+except ImportError:
+    from StringIO import StringIO
 class WSGIRequest(http.HttpRequest):
     def __init__(self, environ):
         self.environ = environ
@@ -119,7 +123,18 @@
         try:
             return self._raw_post_data
         except AttributeError:
-            self._raw_post_data = self.environ['wsgi.input'].read(int(self.environ["CONTENT_LENGTH"]))
+            mem = StringIO()
+            input = self.environ['wsgi.input']
+            content_length = int(self.environ["CONTENT_LENGTH"])
+            chunksize = 65536 # 64KB
+            while 1:
+                remain = content_length - mem.tell()
+                if remain <= 0: break
+                chunk = input.read(min(chunksize, remain))
+                if not chunk: break
+                mem.write(chunk)
+            self._raw_post_data = mem.getvalue()
+            mem.close()
             return self._raw_post_data
 
     GET = property(_get_get, _set_get)

[Jeong-Min Lee]

wsgi MemoryError fix

[Anon]

Just wanted to say that your patch definitely works.

I had a 12 MB file that wouldn't upload at all due to a 'memory error'. I'd hit submit and the server would instantly crash.

Applied this change and there have been no problems since.

[Jeong-Min Lee]

refactored previous patch.

[Jeong-Min Lee]

This also affect django applications on SCGI.

[Chris Beaven]

Worked wonders for me too. Thanks Jeong-Min Lee

[Malcolm Tredinnick]

(In [3805]) Fixed #2613 -- Fixed an easily triggered memory error in file uploads for WSGI.
Thanks Jeong-Min Lee.

[Adrian Holovaty]

This patch caused #2924.