smart_unicode does not always return a unicode

[Rachel Willmer]

Django 1.0.2, python 2.6

I'm using BeautifulSoup to parse some XML which I then use as filter into the django database.

If I extract a field from the XML, I get an object of class BeautifulSoup.NavigableString.

If I then call smart_unicode(obj), I still have an object of class BeautifulSoup.NavigableString rather than the unicode I was expecting.

If I then use this as a filter into django, .e.g product = Product.objects.get(obj=obj), I get this error:

  File "/var/lib/python-support/python2.6/django/db/models/manager.py", line 93, in get
    return self.get_query_set().get(*args, **kwargs)
  File "/var/lib/python-support/python2.6/django/db/models/query.py", line 304, in get
    num = len(clone)
  File "/var/lib/python-support/python2.6/django/db/models/query.py", line 160, in __len__
    self._result_cache = list(self.iterator())
  File "/var/lib/python-support/python2.6/django/db/models/query.py", line 275, in iterator
    for row in self.query.results_iter():
  File "/var/lib/python-support/python2.6/django/db/models/sql/query.py", line 206, in results_iter
    for rows in self.execute_sql(MULTI):
  File "/var/lib/python-support/python2.6/django/db/models/sql/query.py", line 1734, in execute_sql
    cursor.execute(sql, params)
  File "/var/lib/python-support/python2.6/django/db/backends/postgresql/base.py", line 52, in execute
    return self.cursor.execute(smart_str(sql, self.charset), self.format_params(params))
  File "/var/lib/python-support/python2.6/django/db/backends/postgresql/base.py", line 49, in format_params
    return tuple([smart_str(p, self.charset, True) for p in params])
  File "/var/lib/python-support/python2.6/django/utils/encoding.py", line 95, in smart_str
    return s.encode(encoding, errors)
TypeError: encode() takes at most 2 arguments (3 given)

[Karen Tracey]

This looks like a problem with BeautifulSoup, not Django. BeautifulSoup's NavigableString class inherits from unicode, so isinstance(x, unicode) where x is a NavigableString returns True. Thus Django's unicode conversion functions simply return it, since it appears to be unicode already. The problem shown in the traceback is that this NavigableString class overrides the standard unicode encode() method, which takes 3 parameters (​http://docs.python.org/library/stdtypes.html#str.encode) with one that only takes 2: ​http://bazaar.launchpad.net/~leonardr/beautifulsoup/trunk/annotate/head%3A/src/beautifulsoup/element.py#L323. If it is going to inherit from unicode and override the encode method, it should support all the arguments defined for the base encode.