FileField traps error when uploading non-ascii filename

[mizutori]

Uploading non-ascii filename causes error by UnicodeEncodeError.

If the form has two (or more) input fields for uploading files,
and if you fill in either field by non-ascii filename and the other
field empty, then an UnicodeEncodeError occurs when is_valid() is invoked.

To the contrary, no error occurs when
(1) the form has only one input field and fill it by non-ascii filename,
(2) or the form has two (or more) input fields and fill them all by filenames.
(3) or fill one or more input field(s) by ascii filename(s)

I wonder why the error occurs only on condition that
(4) the form has two (or more) input fields and one field is filled by
non-ascii filename and the other field(s) is/are kept empty.

[1] source code

---[ views.py ]---
class UploadForm(forms.Form):
  attach_1 = forms.FileField(required = False)
  attach_2 = forms.FileField(required = False)

def test_upload(self,request):
  if request.method == 'POST':
    formup = UploadForm(request.POST,request.FILES)
    if formup.is_valid():
      result = self.handle_uploaded_file(request.FILES)
      return HttpResponseRedirect('result.html')
---
* the form page is encoded in UTF-8.

[2] error message

UnicodeEncodeError at /mysite1/debug/upload.html
'ascii' codec can't encode characters in position 37-39: ordinal not in range(128)

Unicode error hint
The string that could not be encoded/decoded was: NON-ASCII-UPLOAD-FILENAME

[3] traceback message

Traceback:
File "/usr/lib/python25/Lib/site-packages/django/core/handlers/base.py" in get_response
 86. response = callback(request, *callback_args, **callback_kwargs)
File "/mysite1/debug/views.py" in upload
 70. return MyDebug.toolbox.test_upload(request)
File "/mysite1/debug/util_mysite_debug.py" in test_upload
 177. if formup.is_valid():
File "/usr/lib/python25/Lib/site-packages/django/forms/forms.py" in is_valid
 120. return self.is_bound and not bool(self.errors)
File "/usr/lib/python25/Lib/site-packages/django/forms/forms.py" in _get_errors
 111. self.full_clean()
File "/usr/lib/python25/Lib/site-packages/django/forms/forms.py" in full_clean
 212. value = field.widget.value_from_datadict(self.data, self.files, self.add_prefix(name))
File "/usr/lib/python25/Lib/site-packages/django/forms/widgets.py" in value_from_datadict
 266. return files.get(name, None)
File "/usr/lib/python25/Lib/site-packages/django/utils/datastructures.py" in get
 227. val = self[key]
File "/usr/lib/python25/Lib/site-packages/django/utils/datastructures.py" in __getitem__
 198. raise MultiValueDictKeyError, "Key %r not found in %r" % (key, self)
File "/usr/lib/python25/Lib/site-packages/django/utils/datastructures.py" in __repr__
 188. super(MultiValueDict, self).__repr__())

Exception Type: UnicodeEncodeError at /mysite1/debug/upload.html
Exception Value: 'ascii' codec can't encode characters in position 37-39: ordinal not in range(128)

[mizutori]

A workaround for this problem

The UnicodeEncodeError is caused by printing an exception message that
the field key of empty FileInput field is not found in request.FILES;

---[ in django/utils/datastructures.py ]---
class MultiValueDict(dict):
  def __getitem__(self, key):
    try:
      list_ = super(MultiValueDict, self).__getitem__(key)
    except KeyError:
      raise MultiValueDictKeyError, "Key %r not found in %r" % (key, self)
---

where the self contains request.FILES, whose one filed is empty and other
field is filled by non-ascii filename.

When is_valid() is invoked, the KeyError exception is raised by the empty
field and printing the whole contents of request.FILES. If other field
is filled by non-ascii filename, printing message occurs UnicodeEncodeError.

My idea to fix this problem is to apply one of these workarounds:

WORKAROUND (candidate 1):
in django/utils/datastructures.py, suppress exception message like this.

  raise MultiValueDictKeyError, "Key %r not found in %r" % (key, self)
-->
  raise MultiValueDictKeyError, "Key %r not found" % (key,)

WORKAROUND (candidate 2):
in django/forms/widgets.py, make FileInput widget clever not to dig empty field

class FileInput(Input):
  def value_from_datadict(self, data, files, name):
#   return files.get(name, None)
    if name in files:
      val = files.get(name)
    else:
      val = None
    return val

[Jacob]

(fixed formatting)

[Julien Phalip]

I tried all cases (1 to 4) you've enumerated and I could not reproduce the issue. I you keep having this problem in the latest trunk, please submit a test case as well.

Below is the code I used:

from django import forms
from django.shortcuts import render_to_response
from django.http import HttpResponseRedirect

class UploadForm(forms.Form):
    attach_1 = forms.FileField(required = False)
    attach_2 = forms.FileField(required = False)


def test_upload(request):
    if request.method == 'POST':
        form = UploadForm(request.POST,request.FILES)
        if form.is_valid():
            return HttpResponseRedirect('/ticket8421/success/')
    else:
        form = UploadForm()
    return render_to_response('ticket8421/upload.html', { 'form': form })

[Jacob]

Milestone 1.0 deleted