id	summary	reporter	owner	description	type	status	component	version	severity	resolution	keywords	cc	stage	has_patch	needs_docs	needs_tests	needs_better_patch	easy	ui_ux
5640	Request to improve error handling in utils.encoding.force_unicode.	frank.hoffsummer@…	Malcolm Tredinnick	"migrating a large django project to trunk version > 5609 (i.e. after Unicode branch was merged) can be very painful due to a bug(?!) in utils.encoding.force_unicode

basically, following the instructions here http://code.djangoproject.com/wiki/UnicodeBranch#PortingApplicationsTheQuickChecklist is the right thing to do, but it is very difficult in a larger project to catch all the strings that might need and {{{u''}}} before them. 

Make the slightest oversight, and you lose the ability to debug your models/views/templates as utils.encoding.force_unicode will throw an UnicodeDecodeError exception like this one:


{{{
MOD_PYTHON ERROR

ProcessId:      9493
Interpreter:    'mcc'

ServerName:     'legolas.svt.se'
DocumentRoot:   '/Library/WebServer/Documents'

URI:            '/kluster/errors/'
Location:       '/kluster'
Directory:      None
Filename:       '/Library/WebServer/Documents/kluster'
PathInfo:       '/errors/'

Phase:          'PythonHandler'
Handler:        'django.core.handlers.modpython'

Traceback (most recent call last):

  File ""/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/mod_python/importer.py"", line 1537, in HandlerDispatch
    default=default_handler, arg=req, silent=hlist.silent)

  File ""/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/mod_python/importer.py"", line 1229, in _process_target
    result = _execute_target(config, req, object, arg)

  File ""/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/mod_python/importer.py"", line 1128, in _execute_target
    result = object(arg)

  File ""/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/django/core/handlers/modpython.py"", line 178, in handler
    return ModPythonHandler()(req)

  File ""/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/django/core/handlers/modpython.py"", line 151, in __call__
    response = self.get_response(request)

  File ""/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/django/core/handlers/base.py"", line 53, in get_response
    response = self._real_get_response(request)

  File ""/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/django/core/handlers/base.py"", line 115, in _real_get_response
    return debug.technical_500_response(request, *sys.exc_info())

  File ""/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/django/views/debug.py"", line 151, in technical_500_response
    return HttpResponseServerError(t.render(c), mimetype='text/html')

  File ""/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/django/template/__init__.py"", line 176, in render
    return self.nodelist.render(context)

  File ""/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/django/template/__init__.py"", line 732, in render
    bits.append(self.render_node(node, context))

  File ""/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/django/template/__init__.py"", line 745, in render_node
    return node.render(context)

  File ""/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/django/template/defaulttags.py"", line 135, in render
    nodelist.append(node.render(context))

  File ""/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/django/template/defaulttags.py"", line 229, in render
    return self.nodelist_true.render(context)

  File ""/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/django/template/__init__.py"", line 732, in render
    bits.append(self.render_node(node, context))

  File ""/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/django/template/__init__.py"", line 745, in render_node
    return node.render(context)

  File ""/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/django/template/defaulttags.py"", line 135, in render
    nodelist.append(node.render(context))

  File ""/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/django/template/__init__.py"", line 781, in render
    return self.filter_expression.resolve(context)

  File ""/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/django/template/__init__.py"", line 599, in resolve
    obj = func(obj, *arg_vals)

  File ""/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/django/template/defaultfilters.py"", line 25, in _dec
    args[0] = force_unicode(args[0])

  File ""/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/django/utils/encoding.py"", line 41, in force_unicode
    s = unicode(s, encoding, errors)

UnicodeDecodeError: 'utf8' codec can't decode bytes in position 3679-3681: invalid data
}}}

from this traceback, it is very difficult for laymen (like me) to figure out what is wrong and trace the cause of the error (in my case: a non-ascii string in settings.py that didn't have a u'' in front of it). This severly hampers the migration of larger projects to django trunk post r5609.

My patch just catches the exception in force_unicode and enforces unicode conversion with the 'ignore' flag. This is certainly not the optimal solution, but at least it allows for debugging models/views/templates!

"		closed	Core (Other)	dev		fixed	unicode, force_unicode, UnicodeDecodeError		Design decision needed	0	0	0	0	0	0