Code

Opened 9 months ago

Closed 2 months ago

Last modified 2 months ago

#20856 closed Bug (needsinfo)

Error with admin popups: expected a character buffer object

Reported by: heppner.mark@… Owned by: nobody
Component: contrib.admin Version: 1.6
Severity: Normal Keywords: admin, str, character, inline, popup
Cc: Natim87 Triage Stage: Unreviewed
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

Using Django 1.5 and Python 2.6/2.7, I keep getting an error when using popups in the admin interface. See the example models below. When adding a model having a ForeignKey, the ForeignKey model can be added through a popup. I get the error (traceback below) of 'expected a character buffer overflow'. This only occurs with popups. Adding the ForeignKey separately works as expected. I've seen other reports of this error, but no solid explanation. The user from that StackOverflow question reports downgrading to Django 1.4 and the errors disappear.

class OrgUnit(models.Model):
    api_id = models.IntegerField()
    name = models.CharField()

    def __unicode__(self):
        return '%s' % self.name

class Module(models.Model):
    org_unit = models.ForeignKey(OrgUnit)
    api_id = models.IntegerField()
    name = models.CharField(max_length=255, blank=True)

    def __unicode__(self):
        return '%s (%s)' % (self.name, self.api_id)
ERROR Internal Server Error: /distance/admin/pilot_api/orgunit/add/
Traceback (most recent call last):
  File "/usr/local/www/django/distance/lib/python2.7/site-packages/django/core/handlers/base.py", line 115, in get_response
    response = callback(request, *callback_args, **callback_kwargs)
  File "/usr/local/www/django/distance/lib/python2.7/site-packages/django/contrib/admin/options.py", line 372, in wrapper
    return self.admin_site.admin_view(view)(*args, **kwargs)
  File "/usr/local/www/django/distance/lib/python2.7/site-packages/django/utils/decorators.py", line 91, in _wrapped_view
    response = view_func(request, *args, **kwargs)
  File "/usr/local/www/django/distance/lib/python2.7/site-packages/django/views/decorators/cache.py", line 89, in _wrapped_view_func
    response = view_func(request, *args, **kwargs)
  File "/usr/local/www/django/distance/lib/python2.7/site-packages/django/contrib/admin/sites.py", line 202, in inner
    return view(request, *args, **kwargs)
  File "/usr/local/www/django/distance/lib/python2.7/site-packages/django/utils/decorators.py", line 25, in _wrapper
    return bound_func(*args, **kwargs)
  File "/usr/local/www/django/distance/lib/python2.7/site-packages/django/utils/decorators.py", line 91, in _wrapped_view
    response = view_func(request, *args, **kwargs)
  File "/usr/local/www/django/distance/lib/python2.7/site-packages/django/utils/decorators.py", line 21, in bound_func
    return func(self, *args2, **kwargs2)
  File "/usr/local/www/django/distance/lib/python2.7/site-packages/django/db/transaction.py", line 223, in inner
    return func(*args, **kwargs)
  File "/usr/local/www/django/distance/lib/python2.7/site-packages/django/contrib/admin/options.py", line 1010, in add_view
    return self.response_add(request, new_object)
  File "/usr/local/www/django/distance/lib/python2.7/site-packages/django/contrib/admin/options.py", line 833, in response_add
    (escape(pk_value), escapejs(obj)))
  File "/usr/local/www/django/distance/lib/python2.7/site-packages/django/utils/functional.py", line 194, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/www/django/distance/lib/python2.7/site-packages/django/utils/html.py", line 65, in escapejs
    return mark_safe(force_text(value).translate(_js_escapes))
TypeError: expected a character buffer object

Attachments (0)

Change History (21)

comment:1 Changed 9 months ago by timo

  • Needs documentation unset
  • Needs tests unset
  • Patch needs improvement unset
  • Resolution set to invalid
  • Status changed from new to closed

Your __unicode__ method needs to return a unicode string, e.g. return u'%s (%s)' % (self.name, self.api_id).

comment:2 Changed 9 months ago by heppner.mark@…

  • Resolution invalid deleted
  • Status changed from closed to new

Making the unicode function return a unicode string still doesn't fix it. There is some regression between 1.5 and 1.4 because I did not have this issue using Django 1.4. And if the problem was with the unicode strings, I should still be getting an error with the normal admin interface, but it only appears in popups.

comment:3 Changed 9 months ago by timo

What's the value that force_text is called with? That should help you debug the problem.

comment:4 Changed 9 months ago by timo

  • Resolution set to needsinfo
  • Status changed from new to closed

comment:5 Changed 9 months ago by heppner.mark@…

There was a ternary operator in the __unicode__ function where one option wasn't being cast as a unicode string. I missed it the first time. However, I still couldn't figure out why it was throwing the error only in the popup. Adding the object through the separate admin page worked fine. This error only appeared after updating from 1.4 to 1.5.

comment:6 Changed 3 months ago by Natim87

  • Cc Natim87 added
  • Resolution needsinfo deleted
  • Status changed from closed to new
  • Version changed from 1.5 to 1.6

I have the exact same problem when I set my primary key to be UUID.

   >>> value
   UUID('c7e0b63c79974727a659428c8d734db3')
   >>> force_text(value)
   'c7e0b63c79974727a659428c8d734db3'
   >>> _js_escapes
   {0: u'\\u0000', 1: u'\\u0001', 2: u'\\u0002', 3: u'\\u0003', 4: u'\\u0004', 5: u'\\u0005', 6: u'\\u0006', 7: u'\\u0007', 8: u'\\u0008', 9: u'\\u0009', 10: u'\\u000A', 11: u'\\u000B', 12: u'\\u000C', 13: u'\\u000D', 14: u'\\u000E', 15: u'\\u000F', 16: u'\\u0010', 17: u'\\u0011', 18: u'\\u0012', 19: u'\\u0013', 20: u'\\u0014', 21: u'\\u0015', 22: u'\\u0016', 23: u'\\u0017', 24: u'\\u0018', 25: u'\\u0019', 26: u'\\u001A', 27: u'\\u001B', 28: u'\\u001C', 29: u'\\u001D', 30: u'\\u001E', 31: u'\\u001F', 34: u'\\u0022', 38: u'\\u0026', 39: u'\\u0027', 8232: u'\\u2028', 8233: u'\\u2029', 45: u'\\u002D', 59: u'\\u003B', 60: u'\\u003C', 61: u'\\u003D', 62: u'\\u003E', 92: u'\\u005C'}
    >>> force_text(value).translate(_js_escapes)
    *** TypeError: expected a character buffer object

It looks like sometimes, the force_text doesn't return a unicode object but a str one, then this error is raised.

django/utils/html.py in escapejs line 62 with Django 1.6.1

comment:7 Changed 3 months ago by Natim87

What I don't understand is that if I run the code behind from a clean shell, force_text(value) returns me an unicode string.
But when adding a PDB and running this, I get a str as written before.

From the shell:

>>> from django.utils.encoding import force_text
>>> from uuid import UUID
>>> value = UUID('c7e0b63c79974727a659428c8d734db3')
>>> force_text(value)
u'c7e0b63c-7997-4727-a659-428c8d734db3'

From PDB:

(Pdb) value
UUID('c7e0b63c79974727a659428c8d734db3')
(Pdb) force_text(value)
'c7e0b63c79974727a659428c8d734db3'
(Pdb) l
 57  	_js_escapes.update((ord('%c' % z), '\\u%04X' % z) for z in range(32))
 58  	
 59  	def escapejs(value):
 60  	    """Hex encodes characters for use in JavaScript strings."""
 61  	    import pdb; pdb.set_trace()
 62  ->	    return mark_safe(force_text(value).translate(_js_escapes))
 63  	escapejs = allow_lazy(escapejs, six.text_type)
 64  	
 65  	def conditional_escape(text):
 66  	    """
 67  	    Similar to escape(), except that it doesn't operate on pre-escaped strings.
(Pdb)


Last edited 3 months ago by Natim87 (previous) (diff)

comment:8 Changed 3 months ago by Natim87

A clue:

(Pdb) value.__class__
<class 'uuidfield.fields.StringUUID'>

comment:9 Changed 3 months ago by Natim87

I could reproduce using:

>>> from django.utils.encoding import force_text
>>> from uuidfield.fields import StringUUID
>>> value = StringUUID('c7e0b63c79974727a659428c8d734db3')
>>> force_text(value)
'c7e0b63c-7997-4727-a659-428c8d734db3'

comment:10 Changed 3 months ago by Natim87

I think we need to change s.__unicode__() with unicode(s), in django.utils.encoding line 102, to fix the problem.

Last edited 3 months ago by Natim87 (previous) (diff)

comment:11 Changed 3 months ago by Natim87

--- django/utils/encoding.py.a	2014-01-17 12:22:19.314939101 +0100
+++ django/utils/encoding.py	2014-01-17 12:23:34.658416678 +0100
@@ -98,5 +98,5 @@ def force_text(s, encoding='utf-8', stri
         if not isinstance(s, six.string_types):
             if hasattr(s, '__unicode__'):
-                s = s.__unicode__()
+                s = six.text_type(s)
             else:
                 if six.PY3:

comment:12 Changed 3 months ago by claudep

  • Resolution set to invalid
  • Status changed from new to closed

This was probably a bug in django-uuidfield, may be fixed by https://github.com/dcramer/django-uuidfield/commit/19af35938cf75c84a1ae21082fdfdaa2f2a56477

comment:13 Changed 3 months ago by anonymous

Still getting these issues in django 1.6.1 and admin pop ups

TypeError: expected a character buffer object

Exception Location: /Users/me/my_env/lib/python2.7/site-packages/django/utils/html.py in escapejs, line 61
Python Executable: /Users/me/my_env/bin/python
Python Version: 2.7.5

comment:14 Changed 2 months ago by anonymous

Having the same issue as the above about admin pop ups.

comment:15 Changed 2 months ago by anonymous

  • Resolution invalid deleted
  • Status changed from closed to new

Same issue with python 2.7.3 on django version 1.6.1

comment:16 Changed 2 months ago by timo

How can we reproduce the issue?

comment:17 Changed 2 months ago by claudep

  • Resolution set to needsinfo
  • Status changed from new to closed

comment:18 Changed 2 months ago by rob.jones@…

I've experienced this issue in 1.6.2. The workaround suggested in http://stackoverflow.com/questions/15162673/expected-a-character-buffer-object (Michael Gendin) was successful in my case.

He suggested that the return call from unicode() method of the class that was being called in the popup be wrapped with unicode().

In my case I have the (simplified) model: (Already using the described workaround)

class AspectRatio(models.Model):
    x = models.PositiveIntegerField
    y = models.PositiveIntegerField
    @property
    def aspect_ratio_decimal(self):
        return self.x/self.y
    
    def __unicode__(self):
        if self.aspect_ratio_minimised != self.aspect_ratio:
            return unicode("%s (%s)" % (self.aspect_ratio, self.aspect_ratio_minimised))
        else:
            return unicode("%s" % (self.aspect_ratio))

This model is referenced through a ForeignKey in this (simplified) model:

class MobileDeviceDisplaySize(models.Model):
    diagonal = models.FloatField()
    aspect_ratio = models.ForeignKey(AspectRatio)

    @property
    def x(self):
         return '%8.2f mm' % (self.diagonal * math.sin(math.atan(self.aspect_ratio.aspect_ratio_decimal)))
    @property
    def y(self):
         return '%8.2f mm' % (self.diagonal * math.cos(math.atan(self.aspect_ratio.aspect_ratio_decimal)))
    def __unicode__(self):
        return "%8.2f mm" % (self.diagonal)
    class Meta:
        unique_together = ('diagonal', 'aspect_ratio',)

Relevant bits of admin.py:

class AspectRatioAdmin(admin.ModelAdmin):
    list_display = ('x', 'y', 'aspect_ratio', 'aspect_ratio_decimal',)
admin.site.register(AspectRatio, AspectRatioAdmin)

class MobileDeviceDisplaySizeAdmin(admin.ModelAdmin):
    list_display = ('diagonal', 'aspect_ratio', 'x', 'y')
admin.site.register(MobileDeviceDisplaySize, MobileDeviceDisplaySizeAdmin)

In the admin interface, if I do the following I see the reported error:
-Add a new mobiledisplaysize object
-Use the '+' on the aspectratio foreignkey field to open a popup
-In the popup (new aspectratio object) enter valid data
-OK on the form
-'expected a character buffer object'

comment:19 follow-up: Changed 2 months ago by claudep

The above example is not usable as is (missing self.aspect_ratio / self.aspect_ratio_minimised in AspectRatio).
Did you try returning unicode in MobileDeviceDisplaySize.__unicode__ (return u"%8.2f mm" % (self.diagonal))?

comment:20 in reply to: ↑ 19 Changed 2 months ago by undergroundrob

Replying to claudep:

The above example is not usable as is (missing self.aspect_ratio / self.aspect_ratio_minimised in AspectRatio).
Did you try returning unicode in MobileDeviceDisplaySize.__unicode__ (return u"%8.2f mm" % (self.diagonal))?

You're quite right; I made a hash of decluttering it. Take 2:

class AspectRatio(models.Model):
    x = models.PositiveIntegerField()
    y = models.PositiveIntegerField()

    def __unicode__(self):
            return "%d:%d" % (self.x, self.y)

class MobileDeviceDisplaySize(models.Model):
    diagonal = models.FloatField()
    aspect_ratio = models.ForeignKey(AspectRatio)

    @property
    def x(self):
         return '%8.2f mm' % (self.diagonal * math.sin(math.atan(self.aspect_ratio.x / self.aspect_ratio.y )))
    @property
    def y(self):
         return '%8.2f mm' % (self.diagonal * math.cos(math.atan(self.aspect_ratio.x / self.aspect_ratio.y)))
    
    def __unicode__(self):
        return "%8.2f mm" % (self.diagonal)

#Admin.py
class MobileDeviceDisplaySizeAdmin(admin.ModelAdmin):
    list_display = ('diagonal', 'aspect_ratio', 'x', 'y')
admin.site.register(MobileDeviceDisplaySize, MobileDeviceDisplaySizeAdmin)

class AspectRatioAdmin(admin.ModelAdmin):
    list_display = ('x', 'y')
admin.site.register(AspectRatio, AspectRatioAdmin)

To answer your question, no, wrapping MobileDeviceDisplaySize.unicode in unicode does not help, nor does it have a detrimental effect.

Only returning unicode from AspectRatio.unicode works, either by:

return unicode(("%d:%d") % (self.x, self.y))

or

return "%s:%s" % (unicode(self.x), unicode(self.y))

comment:21 Changed 2 months ago by claudep

Thanks, I was able to reproduce the error. However, the answer is: __unicode__ should return unicode and not bytestring.
It might be that previous versions of Django were more permissive regarding this requirement.
We could of course add some warning like this:

--- a/django/utils/encoding.py
+++ b/django/utils/encoding.py
@@ -97,7 +97,12 @@ def force_text(s, encoding='utf-8', strings_only=False, errors='strict'):
     try:
         if not isinstance(s, six.string_types):
             if hasattr(s, '__unicode__'):
+                s_class = s.__class__.__name__
                 s = s.__unicode__()
+                if not isinstance(s, six.text_type):
+                    warnings.warn(
+                        "The __unicode__ method of an %s object didn't return "
+                        "unicode content." % s_class, UnicodeWarning)
             else:
                 if six.PY3:
                     if isinstance(s, bytes):

But this would add a slight overhead in a frequently-used method for a relatively obvious error. Hence I'm -0 for adding this. Others may disagree, of course.

Add Comment

Modify Ticket

Change Properties
<Author field>
Action
as closed
as The resolution will be set. Next status will be 'closed'
The resolution will be deleted. Next status will be 'new'
Author


E-mail address and user name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.