Django

Code

Ticket #2613 (closed: fixed)

Opened 2 years ago

Last modified 2 years ago

[patch] memory error in upload big sized file resolved.

Reported by: Jeong-Min Lee Assigned to: adrian
Milestone: Component: Core framework
Version: SVN Keywords:
Cc: Triage Stage: Unreviewed
Has patch: 1 Needs documentation: 0
Needs tests: 0 Patch needs improvement: 0

Description

While running in stand alone mode, when big sized file uploaded, this error occured only in the stand alone mode so when coupled with web server this is not triggered.. Trace back like this

Original Traceback (most recent call last):
  File "C:\Python24\lib\site-packages\django\template\__init__.py", line 706, in render_node
    result = node.render(context)
  File "C:\Python24\lib\site-packages\django\template\__init__.py", line 752, in render
    output = self.filter_expression.resolve(context)
  File "C:\Python24\lib\site-packages\django\template\__init__.py", line 548, in resolve
    obj = resolve_variable(self.var, context)
  File "C:\Python24\lib\site-packages\django\template\__init__.py", line 634, in resolve_variable
    current = current[bits[0]]
  File "C:\Python24\lib\site-packages\django\http\__init__.py", line 31, in __getitem__
    for d in (self.POST, self.GET):
  File "C:\Python24\lib\site-packages\django\core\handlers\wsgi.py", line 99, in _get_post
    self._load_post_and_files()
  File "C:\Python24\lib\site-packages\django\core\handlers\wsgi.py", line 77, in _load_post_and_files
    self._post, self._files = http.parse_file_upload(header_dict, self.raw_post_data)
  File "C:\Python24\lib\site-packages\django\core\handlers\wsgi.py", line 122, in _get_raw_post_data
    self._raw_post_data = self.environ['wsgi.input'].read(int(self.environ["CONTENT_LENGTH"]))
  File "C:\Python24\lib\socket.py", line 303, in read
    data = self._sock.recv(recv_size)
MemoryError

So the diff of the django/core/handlers/wsgi.py is..

--- wsgi.py     (revision 3668)
+++ wsgi.py     (working copy)
@@ -50,6 +50,10 @@
     505: 'HTTP VERSION NOT SUPPORTED',
 }
 
+try:
+    from cStringIO import StringIO
+except ImportError:
+    from StringIO import StringIO
 class WSGIRequest(http.HttpRequest):
     def __init__(self, environ):
         self.environ = environ
@@ -119,7 +123,18 @@
         try:
             return self._raw_post_data
         except AttributeError:
-            self._raw_post_data = self.environ['wsgi.input'].read(int(self.environ["CONTENT_LENGTH"]))
+            mem = StringIO()
+            input = self.environ['wsgi.input']
+            content_length = int(self.environ["CONTENT_LENGTH"])
+            chunksize = 65536 # 64KB
+            while 1:
+                remain = content_length - mem.tell()
+                if remain <= 0: break
+                chunk = input.read(min(chunksize, remain))
+                if not chunk: break
+                mem.write(chunk)
+            self._raw_post_data = mem.getvalue()
+            mem.close()
             return self._raw_post_data
 
     GET = property(_get_get, _set_get)

Attachments

wsgi.py.diff (1.1 kB) - added by Jeong-Min Lee on 09/10/06 09:22:32.
wsgi MemoryError? fix
wsgi.py-memoryerror.diff (1.4 kB) - added by Jeong-Min Lee on 09/16/06 21:05:55.
refactored previous patch.

Change History

09/10/06 09:22:32 changed by Jeong-Min Lee

  • attachment wsgi.py.diff added.

wsgi MemoryError? fix

09/15/06 11:25:00 changed by Anon

Just wanted to say that your patch definitely works.

I had a 12 MB file that wouldn't upload at all due to a 'memory error'. I'd hit submit and the server would instantly crash.

Applied this change and there have been no problems since.

09/16/06 21:05:55 changed by Jeong-Min Lee

  • attachment wsgi.py-memoryerror.diff added.

refactored previous patch.

09/17/06 23:51:31 changed by Jeong-Min Lee

  • priority changed from normal to high.
  • severity changed from normal to major.

This also affect django applications on SCGI.

09/20/06 20:37:18 changed by SmileyChris

Worked wonders for me too. Thanks Jeong-Min Lee

09/23/06 07:41:19 changed by mtredinnick

  • status changed from new to closed.
  • resolution set to fixed.

(In [3805]) Fixed #2613 -- Fixed an easily triggered memory error in file uploads for WSGI. Thanks Jeong-Min Lee.

11/29/06 16:35:59 changed by adrian

This patch caused #2924.


Add/Change #2613 ([patch] memory error in upload big sized file resolved.)




Change Properties
Action