Error with admin popups: expected a character buffer object

[heppner.mark@…]

Using Django 1.5 and Python 2.6/2.7, I keep getting an error when using popups in the admin interface. See the example models below. When adding a model having a ForeignKey, the ForeignKey model can be added through a popup. I get the error (traceback below) of 'expected a character buffer overflow'. This only occurs with popups. Adding the ForeignKey separately works as expected. I've seen ​other ​reports of this error, but no solid explanation. The user from that StackOverflow question reports downgrading to Django 1.4 and the errors disappear.

class OrgUnit(models.Model):
    api_id = models.IntegerField()
    name = models.CharField()

    def __unicode__(self):
        return '%s' % self.name

class Module(models.Model):
    org_unit = models.ForeignKey(OrgUnit)
    api_id = models.IntegerField()
    name = models.CharField(max_length=255, blank=True)

    def __unicode__(self):
        return '%s (%s)' % (self.name, self.api_id)

ERROR Internal Server Error: /distance/admin/pilot_api/orgunit/add/
Traceback (most recent call last):
  File "/usr/local/www/django/distance/lib/python2.7/site-packages/django/core/handlers/base.py", line 115, in get_response
    response = callback(request, *callback_args, **callback_kwargs)
  File "/usr/local/www/django/distance/lib/python2.7/site-packages/django/contrib/admin/options.py", line 372, in wrapper
    return self.admin_site.admin_view(view)(*args, **kwargs)
  File "/usr/local/www/django/distance/lib/python2.7/site-packages/django/utils/decorators.py", line 91, in _wrapped_view
    response = view_func(request, *args, **kwargs)
  File "/usr/local/www/django/distance/lib/python2.7/site-packages/django/views/decorators/cache.py", line 89, in _wrapped_view_func
    response = view_func(request, *args, **kwargs)
  File "/usr/local/www/django/distance/lib/python2.7/site-packages/django/contrib/admin/sites.py", line 202, in inner
    return view(request, *args, **kwargs)
  File "/usr/local/www/django/distance/lib/python2.7/site-packages/django/utils/decorators.py", line 25, in _wrapper
    return bound_func(*args, **kwargs)
  File "/usr/local/www/django/distance/lib/python2.7/site-packages/django/utils/decorators.py", line 91, in _wrapped_view
    response = view_func(request, *args, **kwargs)
  File "/usr/local/www/django/distance/lib/python2.7/site-packages/django/utils/decorators.py", line 21, in bound_func
    return func(self, *args2, **kwargs2)
  File "/usr/local/www/django/distance/lib/python2.7/site-packages/django/db/transaction.py", line 223, in inner
    return func(*args, **kwargs)
  File "/usr/local/www/django/distance/lib/python2.7/site-packages/django/contrib/admin/options.py", line 1010, in add_view
    return self.response_add(request, new_object)
  File "/usr/local/www/django/distance/lib/python2.7/site-packages/django/contrib/admin/options.py", line 833, in response_add
    (escape(pk_value), escapejs(obj)))
  File "/usr/local/www/django/distance/lib/python2.7/site-packages/django/utils/functional.py", line 194, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/www/django/distance/lib/python2.7/site-packages/django/utils/html.py", line 65, in escapejs
    return mark_safe(force_text(value).translate(_js_escapes))
TypeError: expected a character buffer object

[Tim Graham]

Your __unicode__ method needs to return a unicode string, e.g. return u'%s (%s)' % (self.name, self.api_id).

[heppner.mark@…]

Making the unicode function return a unicode string still doesn't fix it. There is some regression between 1.5 and 1.4 because I did not have this issue using Django 1.4. And if the problem was with the unicode strings, I should still be getting an error with the normal admin interface, but it only appears in popups.

[Tim Graham]

What's the value that force_text is called with? That should help you debug the problem.

[heppner.mark@…]

There was a ternary operator in the __unicode__ function where one option wasn't being cast as a unicode string. I missed it the first time. However, I still couldn't figure out why it was throwing the error only in the popup. Adding the object through the separate admin page worked fine. This error only appeared after updating from 1.4 to 1.5.

[Natim87]

I have the exact same problem when I set my primary key to be UUID.

   >>> value
   UUID('c7e0b63c79974727a659428c8d734db3')
   >>> force_text(value)
   'c7e0b63c79974727a659428c8d734db3'
   >>> _js_escapes
   {0: u'\\u0000', 1: u'\\u0001', 2: u'\\u0002', 3: u'\\u0003', 4: u'\\u0004', 5: u'\\u0005', 6: u'\\u0006', 7: u'\\u0007', 8: u'\\u0008', 9: u'\\u0009', 10: u'\\u000A', 11: u'\\u000B', 12: u'\\u000C', 13: u'\\u000D', 14: u'\\u000E', 15: u'\\u000F', 16: u'\\u0010', 17: u'\\u0011', 18: u'\\u0012', 19: u'\\u0013', 20: u'\\u0014', 21: u'\\u0015', 22: u'\\u0016', 23: u'\\u0017', 24: u'\\u0018', 25: u'\\u0019', 26: u'\\u001A', 27: u'\\u001B', 28: u'\\u001C', 29: u'\\u001D', 30: u'\\u001E', 31: u'\\u001F', 34: u'\\u0022', 38: u'\\u0026', 39: u'\\u0027', 8232: u'\\u2028', 8233: u'\\u2029', 45: u'\\u002D', 59: u'\\u003B', 60: u'\\u003C', 61: u'\\u003D', 62: u'\\u003E', 92: u'\\u005C'}
    >>> force_text(value).translate(_js_escapes)
    *** TypeError: expected a character buffer object

It looks like sometimes, the force_text doesn't return a unicode object but a str one, then this error is raised.

django/utils/html.py in escapejs line 62 with Django 1.6.1

[Natim87]

What I don't understand is that if I run the code behind from a clean shell, force_text(value) returns me an unicode string.
But when adding a PDB and running this, I get a str as written before.

From the shell:

>>> from django.utils.encoding import force_text
>>> from uuid import UUID
>>> value = UUID('c7e0b63c79974727a659428c8d734db3')
>>> force_text(value)
u'c7e0b63c-7997-4727-a659-428c8d734db3'

From PDB:

(Pdb) value
UUID('c7e0b63c79974727a659428c8d734db3')
(Pdb) force_text(value)
'c7e0b63c79974727a659428c8d734db3'
(Pdb) l
 57  	_js_escapes.update((ord('%c' % z), '\\u%04X' % z) for z in range(32))
 58  	
 59  	def escapejs(value):
 60  	    """Hex encodes characters for use in JavaScript strings."""
 61  	    import pdb; pdb.set_trace()
 62  ->	    return mark_safe(force_text(value).translate(_js_escapes))
 63  	escapejs = allow_lazy(escapejs, six.text_type)
 64  	
 65  	def conditional_escape(text):
 66  	    """
 67  	    Similar to escape(), except that it doesn't operate on pre-escaped strings.
(Pdb)

[Natim87]

A clue:

(Pdb) value.__class__
<class 'uuidfield.fields.StringUUID'>

[Natim87]

I could reproduce using:

>>> from django.utils.encoding import force_text
>>> from uuidfield.fields import StringUUID
>>> value = StringUUID('c7e0b63c79974727a659428c8d734db3')
>>> force_text(value)
'c7e0b63c-7997-4727-a659-428c8d734db3'

[Natim87]

I think we need to change s.__unicode__() with unicode(s), in django.utils.encoding line 102, to fix the problem.

[Natim87]

--- django/utils/encoding.py.a	2014-01-17 12:22:19.314939101 +0100
+++ django/utils/encoding.py	2014-01-17 12:23:34.658416678 +0100
@@ -98,5 +98,5 @@ def force_text(s, encoding='utf-8', stri
         if not isinstance(s, six.string_types):
             if hasattr(s, '__unicode__'):
-                s = s.__unicode__()
+                s = six.text_type(s)
             else:
                 if six.PY3:

[Claude Paroz]

This was probably a bug in django-uuidfield, may be fixed by ​https://github.com/dcramer/django-uuidfield/commit/19af35938cf75c84a1ae21082fdfdaa2f2a56477

[anonymous]

Still getting these issues in django 1.6.1 and admin pop ups

TypeError: expected a character buffer object

Exception Location: /Users/me/my_env/lib/python2.7/site-packages/django/utils/html.py in escapejs, line 61
Python Executable: /Users/me/my_env/bin/python
Python Version: 2.7.5

[anonymous]

Having the same issue as the above about admin pop ups.

[anonymous]

Same issue with python 2.7.3 on django version 1.6.1

[Tim Graham]

How can we reproduce the issue?

[rob.jones@…]

I've experienced this issue in 1.6.2. The workaround suggested in ​http://stackoverflow.com/questions/15162673/expected-a-character-buffer-object (Michael Gendin) was successful in my case.

He suggested that the return call from unicode() method of the class that was being called in the popup be wrapped with unicode().

In my case I have the (simplified) model: (Already using the described workaround)

class AspectRatio(models.Model):
    x = models.PositiveIntegerField
    y = models.PositiveIntegerField
    @property
    def aspect_ratio_decimal(self):
        return self.x/self.y
    
    def __unicode__(self):
        if self.aspect_ratio_minimised != self.aspect_ratio:
            return unicode("%s (%s)" % (self.aspect_ratio, self.aspect_ratio_minimised))
        else:
            return unicode("%s" % (self.aspect_ratio))

This model is referenced through a ForeignKey in this (simplified) model:

class MobileDeviceDisplaySize(models.Model):
    diagonal = models.FloatField()
    aspect_ratio = models.ForeignKey(AspectRatio)

    @property
    def x(self):
         return '%8.2f mm' % (self.diagonal * math.sin(math.atan(self.aspect_ratio.aspect_ratio_decimal)))
    @property
    def y(self):
         return '%8.2f mm' % (self.diagonal * math.cos(math.atan(self.aspect_ratio.aspect_ratio_decimal)))
    def __unicode__(self):
        return "%8.2f mm" % (self.diagonal)
    class Meta:
        unique_together = ('diagonal', 'aspect_ratio',)

Relevant bits of admin.py:

class AspectRatioAdmin(admin.ModelAdmin):
    list_display = ('x', 'y', 'aspect_ratio', 'aspect_ratio_decimal',)
admin.site.register(AspectRatio, AspectRatioAdmin)

class MobileDeviceDisplaySizeAdmin(admin.ModelAdmin):
    list_display = ('diagonal', 'aspect_ratio', 'x', 'y')
admin.site.register(MobileDeviceDisplaySize, MobileDeviceDisplaySizeAdmin)

In the admin interface, if I do the following I see the reported error:
-Add a new mobiledisplaysize object
-Use the '+' on the aspectratio foreignkey field to open a popup
-In the popup (new aspectratio object) enter valid data
-OK on the form
-'expected a character buffer object'

[Claude Paroz]

The above example is not usable as is (missing self.aspect_ratio / self.aspect_ratio_minimised in AspectRatio).
Did you try returning unicode in MobileDeviceDisplaySize.__unicode__ (return u"%8.2f mm" % (self.diagonal))?

[undergroundrob]

Replying to claudep:

The above example is not usable as is (missing self.aspect_ratio / self.aspect_ratio_minimised in AspectRatio).
Did you try returning unicode in MobileDeviceDisplaySize.__unicode__ (return u"%8.2f mm" % (self.diagonal))?

You're quite right; I made a hash of decluttering it. Take 2:

class AspectRatio(models.Model):
    x = models.PositiveIntegerField()
    y = models.PositiveIntegerField()

    def __unicode__(self):
            return "%d:%d" % (self.x, self.y)

class MobileDeviceDisplaySize(models.Model):
    diagonal = models.FloatField()
    aspect_ratio = models.ForeignKey(AspectRatio)

    @property
    def x(self):
         return '%8.2f mm' % (self.diagonal * math.sin(math.atan(self.aspect_ratio.x / self.aspect_ratio.y )))
    @property
    def y(self):
         return '%8.2f mm' % (self.diagonal * math.cos(math.atan(self.aspect_ratio.x / self.aspect_ratio.y)))
    
    def __unicode__(self):
        return "%8.2f mm" % (self.diagonal)

#Admin.py
class MobileDeviceDisplaySizeAdmin(admin.ModelAdmin):
    list_display = ('diagonal', 'aspect_ratio', 'x', 'y')
admin.site.register(MobileDeviceDisplaySize, MobileDeviceDisplaySizeAdmin)

class AspectRatioAdmin(admin.ModelAdmin):
    list_display = ('x', 'y')
admin.site.register(AspectRatio, AspectRatioAdmin)

To answer your question, no, wrapping MobileDeviceDisplaySize.unicode in unicode does not help, nor does it have a detrimental effect.

Only returning unicode from AspectRatio.unicode works, either by:

return unicode(("%d:%d") % (self.x, self.y))

or

return "%s:%s" % (unicode(self.x), unicode(self.y))

[Claude Paroz]

Thanks, I was able to reproduce the error. However, the answer is: __unicode__ should return unicode and not bytestring.
It might be that previous versions of Django were more permissive regarding this requirement.
We could of course add some warning like this:

--- a/django/utils/encoding.py
+++ b/django/utils/encoding.py
@@ -97,7 +97,12 @@ def force_text(s, encoding='utf-8', strings_only=False, errors='strict'):
     try:
         if not isinstance(s, six.string_types):
             if hasattr(s, '__unicode__'):
+                s_class = s.__class__.__name__
                 s = s.__unicode__()
+                if not isinstance(s, six.text_type):
+                    warnings.warn(
+                        "The __unicode__ method of an %s object didn't return "
+                        "unicode content." % s_class, UnicodeWarning)
             else:
                 if six.PY3:
                     if isinstance(s, bytes):

But this would add a slight overhead in a frequently-used method for a relatively obvious error. Hence I'm -0 for adding this. Others may disagree, of course.

[Florian Apolloner]

This was fixed as a sideeffect of #20812, people on 1.6 will have to live with it or write proper __unicode__ methods :)

[Dan Loewenherz]

This bug still affects Django 1.5.6 as well. I think the fact that it affects two Django versions is enough reason to backport a fix. Thoughts?

[Tim Graham]

1.5 is in security fix only mode. You can fix it by writing proper __unicode__() methods in your application as described above.

[anonymous]

Actually, there is an issue with this. Actually two

When you open raw_id_field in a popup, you get url like: /admin/a/b/?_popup=1
THEN when you want to add new object, url changes to : /admin/a/b/add/?_popup=1&_changelist_filters=_popup%3D1

Well, first of all, the url is wrong, look at _changelist_filters
Second of all, as long as you leave popup=1 parameter, you get the above error.

This has nothing to do with unicode methods, this is broken even with return 'a' ;)

[matija@…]

The error is django/contrib/admin/options.py
Line 1093:
'obj': escapejs(obj)

escapejs does not automatically call unicode()

'obj': escape(obj.unicode()) actually works.

[matija]

sorry,

of xcourse, it is unicode()

[Tim Graham]

You can disable the changelist filters stuff if you want; see ​ModelAdmin.preserve_filters.

I don't think it's broken. escapejs calls force_text() which will invoke the proper method on an object. If you find otherwise, please open a new ticket with a failing test case for Django's test suite.

Please don't reopen this ticket, thanks.