Opened 3 years ago

Closed 3 years ago

#20856 closed Bug (invalid)

Error with admin popups: expected a character buffer object

Reported by: heppner.mark@… Owned by: nobody
Component: contrib.admin Version: 1.6
Severity: Normal Keywords: admin, str, character, inline, popup
Cc: Natim87 Triage Stage: Unreviewed
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

Using Django 1.5 and Python 2.6/2.7, I keep getting an error when using popups in the admin interface. See the example models below. When adding a model having a ForeignKey, the ForeignKey model can be added through a popup. I get the error (traceback below) of 'expected a character buffer overflow'. This only occurs with popups. Adding the ForeignKey separately works as expected. I've seen other reports of this error, but no solid explanation. The user from that StackOverflow question reports downgrading to Django 1.4 and the errors disappear.

class OrgUnit(models.Model):
    api_id = models.IntegerField()
    name = models.CharField()

    def __unicode__(self):
        return '%s' % self.name

class Module(models.Model):
    org_unit = models.ForeignKey(OrgUnit)
    api_id = models.IntegerField()
    name = models.CharField(max_length=255, blank=True)

    def __unicode__(self):
        return '%s (%s)' % (self.name, self.api_id)
ERROR Internal Server Error: /distance/admin/pilot_api/orgunit/add/
Traceback (most recent call last):
  File "/usr/local/www/django/distance/lib/python2.7/site-packages/django/core/handlers/base.py", line 115, in get_response
    response = callback(request, *callback_args, **callback_kwargs)
  File "/usr/local/www/django/distance/lib/python2.7/site-packages/django/contrib/admin/options.py", line 372, in wrapper
    return self.admin_site.admin_view(view)(*args, **kwargs)
  File "/usr/local/www/django/distance/lib/python2.7/site-packages/django/utils/decorators.py", line 91, in _wrapped_view
    response = view_func(request, *args, **kwargs)
  File "/usr/local/www/django/distance/lib/python2.7/site-packages/django/views/decorators/cache.py", line 89, in _wrapped_view_func
    response = view_func(request, *args, **kwargs)
  File "/usr/local/www/django/distance/lib/python2.7/site-packages/django/contrib/admin/sites.py", line 202, in inner
    return view(request, *args, **kwargs)
  File "/usr/local/www/django/distance/lib/python2.7/site-packages/django/utils/decorators.py", line 25, in _wrapper
    return bound_func(*args, **kwargs)
  File "/usr/local/www/django/distance/lib/python2.7/site-packages/django/utils/decorators.py", line 91, in _wrapped_view
    response = view_func(request, *args, **kwargs)
  File "/usr/local/www/django/distance/lib/python2.7/site-packages/django/utils/decorators.py", line 21, in bound_func
    return func(self, *args2, **kwargs2)
  File "/usr/local/www/django/distance/lib/python2.7/site-packages/django/db/transaction.py", line 223, in inner
    return func(*args, **kwargs)
  File "/usr/local/www/django/distance/lib/python2.7/site-packages/django/contrib/admin/options.py", line 1010, in add_view
    return self.response_add(request, new_object)
  File "/usr/local/www/django/distance/lib/python2.7/site-packages/django/contrib/admin/options.py", line 833, in response_add
    (escape(pk_value), escapejs(obj)))
  File "/usr/local/www/django/distance/lib/python2.7/site-packages/django/utils/functional.py", line 194, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/www/django/distance/lib/python2.7/site-packages/django/utils/html.py", line 65, in escapejs
    return mark_safe(force_text(value).translate(_js_escapes))
TypeError: expected a character buffer object

Change History (29)

comment:1 Changed 3 years ago by Tim Graham

Resolution: invalid
Status: newclosed

Your __unicode__ method needs to return a unicode string, e.g. return u'%s (%s)' % (self.name, self.api_id).

comment:2 Changed 3 years ago by heppner.mark@…

Resolution: invalid
Status: closednew

Making the unicode function return a unicode string still doesn't fix it. There is some regression between 1.5 and 1.4 because I did not have this issue using Django 1.4. And if the problem was with the unicode strings, I should still be getting an error with the normal admin interface, but it only appears in popups.

comment:3 Changed 3 years ago by Tim Graham

What's the value that force_text is called with? That should help you debug the problem.

comment:4 Changed 3 years ago by Tim Graham

Resolution: needsinfo
Status: newclosed

comment:5 Changed 3 years ago by heppner.mark@…

There was a ternary operator in the __unicode__ function where one option wasn't being cast as a unicode string. I missed it the first time. However, I still couldn't figure out why it was throwing the error only in the popup. Adding the object through the separate admin page worked fine. This error only appeared after updating from 1.4 to 1.5.

comment:6 Changed 3 years ago by Natim87

Cc: Natim87 added
Resolution: needsinfo
Status: closednew
Version: 1.51.6

I have the exact same problem when I set my primary key to be UUID.

   >>> value
   UUID('c7e0b63c79974727a659428c8d734db3')
   >>> force_text(value)
   'c7e0b63c79974727a659428c8d734db3'
   >>> _js_escapes
   {0: u'\\u0000', 1: u'\\u0001', 2: u'\\u0002', 3: u'\\u0003', 4: u'\\u0004', 5: u'\\u0005', 6: u'\\u0006', 7: u'\\u0007', 8: u'\\u0008', 9: u'\\u0009', 10: u'\\u000A', 11: u'\\u000B', 12: u'\\u000C', 13: u'\\u000D', 14: u'\\u000E', 15: u'\\u000F', 16: u'\\u0010', 17: u'\\u0011', 18: u'\\u0012', 19: u'\\u0013', 20: u'\\u0014', 21: u'\\u0015', 22: u'\\u0016', 23: u'\\u0017', 24: u'\\u0018', 25: u'\\u0019', 26: u'\\u001A', 27: u'\\u001B', 28: u'\\u001C', 29: u'\\u001D', 30: u'\\u001E', 31: u'\\u001F', 34: u'\\u0022', 38: u'\\u0026', 39: u'\\u0027', 8232: u'\\u2028', 8233: u'\\u2029', 45: u'\\u002D', 59: u'\\u003B', 60: u'\\u003C', 61: u'\\u003D', 62: u'\\u003E', 92: u'\\u005C'}
    >>> force_text(value).translate(_js_escapes)
    *** TypeError: expected a character buffer object

It looks like sometimes, the force_text doesn't return a unicode object but a str one, then this error is raised.

django/utils/html.py in escapejs line 62 with Django 1.6.1

comment:7 Changed 3 years ago by Natim87

What I don't understand is that if I run the code behind from a clean shell, force_text(value) returns me an unicode string.
But when adding a PDB and running this, I get a str as written before.

From the shell:

>>> from django.utils.encoding import force_text
>>> from uuid import UUID
>>> value = UUID('c7e0b63c79974727a659428c8d734db3')
>>> force_text(value)
u'c7e0b63c-7997-4727-a659-428c8d734db3'

From PDB:

(Pdb) value
UUID('c7e0b63c79974727a659428c8d734db3')
(Pdb) force_text(value)
'c7e0b63c79974727a659428c8d734db3'
(Pdb) l
 57  	_js_escapes.update((ord('%c' % z), '\\u%04X' % z) for z in range(32))
 58  	
 59  	def escapejs(value):
 60  	    """Hex encodes characters for use in JavaScript strings."""
 61  	    import pdb; pdb.set_trace()
 62  ->	    return mark_safe(force_text(value).translate(_js_escapes))
 63  	escapejs = allow_lazy(escapejs, six.text_type)
 64  	
 65  	def conditional_escape(text):
 66  	    """
 67  	    Similar to escape(), except that it doesn't operate on pre-escaped strings.
(Pdb)


Last edited 3 years ago by Natim87 (previous) (diff)

comment:8 Changed 3 years ago by Natim87

A clue:

(Pdb) value.__class__
<class 'uuidfield.fields.StringUUID'>

comment:9 Changed 3 years ago by Natim87

I could reproduce using:

>>> from django.utils.encoding import force_text
>>> from uuidfield.fields import StringUUID
>>> value = StringUUID('c7e0b63c79974727a659428c8d734db3')
>>> force_text(value)
'c7e0b63c-7997-4727-a659-428c8d734db3'

comment:10 Changed 3 years ago by Natim87

I think we need to change s.__unicode__() with unicode(s), in django.utils.encoding line 102, to fix the problem.

Last edited 3 years ago by Natim87 (previous) (diff)

comment:11 Changed 3 years ago by Natim87

--- django/utils/encoding.py.a	2014-01-17 12:22:19.314939101 +0100
+++ django/utils/encoding.py	2014-01-17 12:23:34.658416678 +0100
@@ -98,5 +98,5 @@ def force_text(s, encoding='utf-8', stri
         if not isinstance(s, six.string_types):
             if hasattr(s, '__unicode__'):
-                s = s.__unicode__()
+                s = six.text_type(s)
             else:
                 if six.PY3:

comment:12 Changed 3 years ago by Claude Paroz

Resolution: invalid
Status: newclosed

This was probably a bug in django-uuidfield, may be fixed by https://github.com/dcramer/django-uuidfield/commit/19af35938cf75c84a1ae21082fdfdaa2f2a56477

comment:13 Changed 3 years ago by anonymous

Still getting these issues in django 1.6.1 and admin pop ups

TypeError: expected a character buffer object

Exception Location: /Users/me/my_env/lib/python2.7/site-packages/django/utils/html.py in escapejs, line 61
Python Executable: /Users/me/my_env/bin/python
Python Version: 2.7.5

comment:14 Changed 3 years ago by anonymous

Having the same issue as the above about admin pop ups.

comment:15 Changed 3 years ago by anonymous

Resolution: invalid
Status: closednew

Same issue with python 2.7.3 on django version 1.6.1

comment:16 Changed 3 years ago by Tim Graham

How can we reproduce the issue?

comment:17 Changed 3 years ago by Claude Paroz

Resolution: needsinfo
Status: newclosed

comment:18 Changed 3 years ago by rob.jones@…

I've experienced this issue in 1.6.2. The workaround suggested in http://stackoverflow.com/questions/15162673/expected-a-character-buffer-object (Michael Gendin) was successful in my case.

He suggested that the return call from unicode() method of the class that was being called in the popup be wrapped with unicode().

In my case I have the (simplified) model: (Already using the described workaround)

class AspectRatio(models.Model):
    x = models.PositiveIntegerField
    y = models.PositiveIntegerField
    @property
    def aspect_ratio_decimal(self):
        return self.x/self.y
    
    def __unicode__(self):
        if self.aspect_ratio_minimised != self.aspect_ratio:
            return unicode("%s (%s)" % (self.aspect_ratio, self.aspect_ratio_minimised))
        else:
            return unicode("%s" % (self.aspect_ratio))

This model is referenced through a ForeignKey in this (simplified) model:

class MobileDeviceDisplaySize(models.Model):
    diagonal = models.FloatField()
    aspect_ratio = models.ForeignKey(AspectRatio)

    @property
    def x(self):
         return '%8.2f mm' % (self.diagonal * math.sin(math.atan(self.aspect_ratio.aspect_ratio_decimal)))
    @property
    def y(self):
         return '%8.2f mm' % (self.diagonal * math.cos(math.atan(self.aspect_ratio.aspect_ratio_decimal)))
    def __unicode__(self):
        return "%8.2f mm" % (self.diagonal)
    class Meta:
        unique_together = ('diagonal', 'aspect_ratio',)

Relevant bits of admin.py:

class AspectRatioAdmin(admin.ModelAdmin):
    list_display = ('x', 'y', 'aspect_ratio', 'aspect_ratio_decimal',)
admin.site.register(AspectRatio, AspectRatioAdmin)

class MobileDeviceDisplaySizeAdmin(admin.ModelAdmin):
    list_display = ('diagonal', 'aspect_ratio', 'x', 'y')
admin.site.register(MobileDeviceDisplaySize, MobileDeviceDisplaySizeAdmin)

In the admin interface, if I do the following I see the reported error:
-Add a new mobiledisplaysize object
-Use the '+' on the aspectratio foreignkey field to open a popup
-In the popup (new aspectratio object) enter valid data
-OK on the form
-'expected a character buffer object'

comment:19 Changed 3 years ago by Claude Paroz

The above example is not usable as is (missing self.aspect_ratio / self.aspect_ratio_minimised in AspectRatio).
Did you try returning unicode in MobileDeviceDisplaySize.__unicode__ (return u"%8.2f mm" % (self.diagonal))?

comment:20 in reply to:  19 Changed 3 years ago by undergroundrob

Replying to claudep:

The above example is not usable as is (missing self.aspect_ratio / self.aspect_ratio_minimised in AspectRatio).
Did you try returning unicode in MobileDeviceDisplaySize.__unicode__ (return u"%8.2f mm" % (self.diagonal))?

You're quite right; I made a hash of decluttering it. Take 2:

class AspectRatio(models.Model):
    x = models.PositiveIntegerField()
    y = models.PositiveIntegerField()

    def __unicode__(self):
            return "%d:%d" % (self.x, self.y)

class MobileDeviceDisplaySize(models.Model):
    diagonal = models.FloatField()
    aspect_ratio = models.ForeignKey(AspectRatio)

    @property
    def x(self):
         return '%8.2f mm' % (self.diagonal * math.sin(math.atan(self.aspect_ratio.x / self.aspect_ratio.y )))
    @property
    def y(self):
         return '%8.2f mm' % (self.diagonal * math.cos(math.atan(self.aspect_ratio.x / self.aspect_ratio.y)))
    
    def __unicode__(self):
        return "%8.2f mm" % (self.diagonal)

#Admin.py
class MobileDeviceDisplaySizeAdmin(admin.ModelAdmin):
    list_display = ('diagonal', 'aspect_ratio', 'x', 'y')
admin.site.register(MobileDeviceDisplaySize, MobileDeviceDisplaySizeAdmin)

class AspectRatioAdmin(admin.ModelAdmin):
    list_display = ('x', 'y')
admin.site.register(AspectRatio, AspectRatioAdmin)

To answer your question, no, wrapping MobileDeviceDisplaySize.unicode in unicode does not help, nor does it have a detrimental effect.

Only returning unicode from AspectRatio.unicode works, either by:

return unicode(("%d:%d") % (self.x, self.y))

or

return "%s:%s" % (unicode(self.x), unicode(self.y))

comment:21 Changed 3 years ago by Claude Paroz

Thanks, I was able to reproduce the error. However, the answer is: __unicode__ should return unicode and not bytestring.
It might be that previous versions of Django were more permissive regarding this requirement.
We could of course add some warning like this:

--- a/django/utils/encoding.py
+++ b/django/utils/encoding.py
@@ -97,7 +97,12 @@ def force_text(s, encoding='utf-8', strings_only=False, errors='strict'):
     try:
         if not isinstance(s, six.string_types):
             if hasattr(s, '__unicode__'):
+                s_class = s.__class__.__name__
                 s = s.__unicode__()
+                if not isinstance(s, six.text_type):
+                    warnings.warn(
+                        "The __unicode__ method of an %s object didn't return "
+                        "unicode content." % s_class, UnicodeWarning)
             else:
                 if six.PY3:
                     if isinstance(s, bytes):

But this would add a slight overhead in a frequently-used method for a relatively obvious error. Hence I'm -0 for adding this. Others may disagree, of course.

comment:22 Changed 3 years ago by Florian Apolloner

Resolution: needsinfoinvalid

This was fixed as a sideeffect of #20812, people on 1.6 will have to live with it or write proper __unicode__ methods :)

comment:23 Changed 3 years ago by Dan Loewenherz

Resolution: invalid
Status: closednew

This bug still affects Django 1.5.6 as well. I think the fact that it affects two Django versions is enough reason to backport a fix. Thoughts?

comment:24 Changed 3 years ago by Tim Graham

Resolution: invalid
Status: newclosed

1.5 is in security fix only mode. You can fix it by writing proper __unicode__() methods in your application as described above.

comment:25 Changed 3 years ago by anonymous

Resolution: invalid
Status: closednew

Actually, there is an issue with this. Actually two

When you open raw_id_field in a popup, you get url like: /admin/a/b/?_popup=1
THEN when you want to add new object, url changes to : /admin/a/b/add/?_popup=1&_changelist_filters=_popup%3D1

Well, first of all, the url is wrong, look at _changelist_filters
Second of all, as long as you leave popup=1 parameter, you get the above error.

This has nothing to do with unicode methods, this is broken even with return 'a' ;)

comment:26 Changed 3 years ago by matija@…

The error is django/contrib/admin/options.py
Line 1093:
'obj': escapejs(obj)

escapejs does not automatically call unicode()

'obj': escape(obj.unicode()) actually works.

comment:27 Changed 3 years ago by matija

sorry,

of xcourse, it is unicode()

comment:28 Changed 3 years ago by Tim Graham

You can disable the changelist filters stuff if you want; see ModelAdmin.preserve_filters.

I don't think it's broken. escapejs calls force_text() which will invoke the proper method on an object. If you find otherwise, please open a new ticket with a failing test case for Django's test suite.

Please don't reopen this ticket, thanks.

comment:29 Changed 3 years ago by Tim Graham

Resolution: invalid
Status: newclosed
Note: See TracTickets for help on using tickets.
Back to Top