encoding.force_unicode exception unclear/vague, update to clarify exception

[monkut]

I just spent over an hour tracking down a bug in my code.

It was a very simple beginner mistake.
I forgot to return a value in a model's unicode method.

class MyModel(models.Model):
    name = models.CharField()

    def __unicode__(self):
        self.name[:25]   

Unfortunatly this isn't caught a propagated in a clear way and results in the following exception, when accessing the admin applicaiton.

TypeError at /admin/specmaker/project/add/
coercing to Unicode: need string or buffer, NoneType foundRequest Method:	POST
Request URL:	http://127.0.0.1:8000/admin/specmaker/project/add/
Exception Type:	TypeError
Exception Value:	coercing to Unicode: need string or buffer, NoneType found
Exception Location:	C:\Python26\Lib\site-packages\django\utils\encoding.py in force_unicode, line 49
Python Executable:	C:\Python26\python.exe
Python Version:	2.6.0
Python Path:	['D:\\PROGRAMMING\\Workspace\\GEODJANGO-TUTORIAL\\geodjango\\specmgr', 'C:\\Python26\\lib\\site-packages\\setuptools-0.6c9-py2.6.egg', 'C:\\Python26\\lib\\site-packages\\sphinx-0.4.3-py2.6.egg', 'C:\\Python26\\lib\\site-packages\\docutils-0.5-py2.6.egg', 'C:\\Python26\\lib\\site-packages\\jinja-1.2-py2.6-win32.egg', 'C:\\Python26\\lib\\site-packages\\pygments-0.11.1-py2.6.egg', 'C:\\Python26\\python26.zip', 'C:\\Python26\\DLLs', 'C:\\Python26\\lib', 'C:\\Python26\\lib\\plat-win', 'C:\\Python26\\lib\\lib-tk', 'C:\\Python26', 'C:\\Python26\\lib\\site-packages', 'C:\\Python26\\lib\\site-packages\\PIL', 'C:\\Python26\\lib\\site-packages\\win32', 'C:\\Python26\\lib\\site-packages\\win32\\lib', 'C:\\Python26\\lib\\site-packages\\Pythonwin']
Server time:	Wed, 25 Feb 2009 17:58:37 +0900

I'm sure regular users of django may no what causes this right-away, but beginners like myself may be stuck trying to debug django code trying to find what this exception is.

I propose something like this (encoding.force_unicode) to clarify this exception:

def force_unicode(s, encoding='utf-8', strings_only=False, errors='strict'):
    """
    Similar to smart_unicode, except that lazy instances are resolved to
    strings, rather than kept as lazy objects.

    If strings_only is True, don't convert (some) non-string-like objects.
    """
    if strings_only and isinstance(s, (types.NoneType, int, long, datetime.datetime, datetime.date, datetime.time, float)):
        return s
    try:
        if not isinstance(s, basestring,):
            if hasattr(s, '__unicode__'):
                try:
                    s = unicode(s)
                except TypeError:
                    msg = u"%s's __unicode__ method not returning proper unicode string" % (repr(type(s)))
                    raise TypeError(msg)

So the error will display as:

TypeError at /admin/specmaker/project/add/
<class 'specmgr.specmaker.models.Project'>'s __unicode__ method not returning proper unicode stringRequest Method:	POST
Request URL:	http://127.0.0.1:8000/admin/specmaker/project/add/
Exception Type:	TypeError
Exception Value:	<class 'specmgr.specmaker.models.Project'>'s __unicode__ method not returning proper unicode string
Exception Location:	C:\Python26\lib\site-packages\django\utils\encoding.py in force_unicode, line 53
Python Executable:	C:\Python26\python.exe
Python Version:	2.6.0
Python Path:	['D:\\PROGRAMMING\\Workspace\\GEODJANGO-TUTORIAL\\geodjango\\specmgr', 'C:\\Python26\\lib\\site-packages\\setuptools-0.6c9-py2.6.egg', 'C:\\Python26\\lib\\site-packages\\sphinx-0.4.3-py2.6.egg', 'C:\\Python26\\lib\\site-packages\\docutils-0.5-py2.6.egg', 'C:\\Python26\\lib\\site-packages\\jinja-1.2-py2.6-win32.egg', 'C:\\Python26\\lib\\site-packages\\pygments-0.11.1-py2.6.egg', 'C:\\Python26\\python26.zip', 'C:\\Python26\\DLLs', 'C:\\Python26\\lib', 'C:\\Python26\\lib\\plat-win', 'C:\\Python26\\lib\\lib-tk', 'C:\\Python26', 'C:\\Python26\\lib\\site-packages', 'C:\\Python26\\lib\\site-packages\\PIL', 'C:\\Python26\\lib\\site-packages\\win32', 'C:\\Python26\\lib\\site-packages\\win32\\lib', 'C:\\Python26\\lib\\site-packages\\Pythonwin']
Server time:	Wed, 25 Feb 2009 18:21:57 +0900

[Jacob]

The problem with this technique is that it overrides the original traceback, leaving you no way of accessing the original traceback object. That's bad form for a number of reasons, the biggest being this: what if the body __unicode__ method itself raises a TypeError? You'd loose that original exception, and be mistaken into thinking it was a returning-the-wrong-type issue.

[monkut]

Replying to jacob:
Good point.

My concern is that new users of django like myself may get turned off of the framework due to confusing exceptions like this one.

Just to clarify your point, your saying that there is an issue if in the body of the unicode method of 's' a TypeError is raised, right?

The issue I see is that the 'real' problem is hidden from the user because the exception points to force_unicode(), making it difficult to quickly debug the problem. It would be nice if the developer could be pointed a little closer to the 'real' problem.

How about changing the message and re-raising the original exception?

def force_unicode(s, encoding='utf-8', strings_only=False, errors='strict'):
    ...
    if strings_only and isinstance(s, (types.NoneType, int, long, datetime.datetime, datetime.date, datetime.time, float)):
        return s
    try:
        if not isinstance(s, basestring,):
            if hasattr(s, '__unicode__'):
                try:
                    s = unicode(s)
                except TypeError, e:
                    msg = u"Found in %s's __unicode__ method" % (repr(type(s)))
                    mutable_args = list(e.args)
                    mutable_args.append(msg)
                    e.args = tuple(mutable_args)
                    raise e
                    

[Jacob]

Please don't reopen tickets closed by a committer. The correct way to revisit issues is to take it up on django-dev.

This change really isn't any cleaner, and depends on the internal details of TypeError which we can't count on. Look, at a certain point we have to trust our users to do the right thing. We can't hand-hold every place someone might make a mistake.

[monkut]

Replying to jacob:

Please don't reopen tickets closed by a committer. The correct way to revisit issues is to take it up on django-dev.

Thanks for the clarification. Sorry about that.

I understand this is a design direction decision, which is not solved by this single suggested fix. I'll look into taking it up on django-dev. Thanks for your time.