Opened 17 years ago

Closed 17 years ago

Last modified 12 years ago

#4796 closed Uncategorized (fixed)

force_unicode error

Reported by: fpierfed@… Owned by:
Component: Core (Other) Version: dev
Severity: Normal Keywords: fcgi unicode __proxy__ databrowse
Cc: jeff@…, mmulley@…, me@… Triage Stage: Accepted
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description (last modified by Malcolm Tredinnick)

Hi,

I am using the newforms-admin branch from SVN (revision 5632) under apache 2.0.54, flup SVN (revision 2354) and Python 2.4.1. I have a webapp that uses custom admin sites via myadmin = admin.AdminSite() and myadmin.register(...)

If I access the admin site via the django internal development server, everything works. If I use fcgi the admin pages work only if I am an anonymous user (in that case I am redirected to the login screen, then to the admin pages and all is good). If I use fcgi and I am already logged in, accessing the admin pages gives this error:

TypeError at /tips/admin/
coercing to Unicode: need string or buffer, __proxy__ found
Request Method: 	GET
Request URL: 	http://www.peertraveller.com/tips/admin/
Exception Type: 	TypeError
Exception Value: 	coercing to Unicode: need string or buffer, __proxy__ found
Exception Location: 	/home/fpierfed/lib/python2.4/site-packages/django/utils/encoding.py in force_unicode, line 38
Python Executable: 	/home/fpierfed/bin/python
Python Version: 	2.4.1

Attachments (4)

TypeError at _databrowse_.html (49.8 KB ) - added by Jeff Forcier <jeff@…> 17 years ago.
4796-hopeful-fix.diff (2.2 KB ) - added by Malcolm Tredinnick 17 years ago.
If somebody can reproduce the problem, does this patch fix it?
bug4796.tar.bz2 (1.1 KB ) - added by durdinator 17 years ago.
Minimal testcase
4796-fix.diff (1.1 KB ) - added by durdinator 17 years ago.
Patch, with regression test (fix typo)

Download all attachments as: .zip

Change History (40)

comment:1 by Malcolm Tredinnick, 17 years ago

Description: modified (diff)

Fixed description.

comment:2 by Malcolm Tredinnick, 17 years ago

We'll need the full traceback to be able to debug this -- you've only provided the last line. Look a bit further down the debug screen and find the link that says "cut-and-paste traceback". Click on that and the paste in the output (remember to wrap it in {{{ and }}} markup so that it's formatted correctly in Trac's wiki syntax.

comment:3 by fpierfed@…, 17 years ago

Here is the full traceback:

Traceback (most recent call last):
File "/home/fpierfed/lib/python2.4/site-packages/django/core/handlers/base.py" in get_response
  77. response = callback(request, *callback_args, **callback_kwargs)
File "/home/fpierfed/lib/python2.4/site-packages/django/contrib/admin/sites.py" in root
  115. return self.index(request)
File "/home/fpierfed/lib/python2.4/site-packages/django/contrib/admin/sites.py" in index
  273. app['models'].sort(lambda x, y: cmp(x['name'], y['name']))
File "/home/fpierfed/lib/python2.4/site-packages/django/contrib/admin/sites.py" in
  273. app['models'].sort(lambda x, y: cmp(x['name'], y['name']))
File "/home/fpierfed/lib/python2.4/site-packages/django/utils/functional.py" in __cmp__
  86. s = unicode(self.__func(*self.__args, **self.__kw))
File "/home/fpierfed/lib/python2.4/site-packages/django/utils/text.py" in
  7. capfirst = lambda x: x and force_unicode(x)[0].upper() + force_unicode(x)[1:]
File "/home/fpierfed/lib/python2.4/site-packages/django/utils/encoding.py" in force_unicode
  38. s = unicode(s)

  TypeError at /tips/add/
  coercing to Unicode: need string or buffer, __proxy__ found

comment:4 by Malcolm Tredinnick, 17 years ago

So this traceback means the __unicode__ method on a model is returning a lazy translation object, which is illegal Python code (__unicode__ must return a Unicode object).

It's not possible to tell from the traceback which model is involved, although as far as I can see, no models in Django's source are doing this.

Have a look at your debug traceback screen again and examine the local variables in the second last line (the "capfirst" line). That should tell you which object is involved. Or, if that doesn't, have a look at the local variables in one of the previous lines (the ones from sites.py).

If the object involved is in Django's code, please let us know which one.

comment:5 by Simon G. <dev@…>, 17 years ago

Resolution: invalid
Status: newclosed

Resolving as invalid as it doesn't appear to be Django-specific. Please reopen if I'm wrong!

comment:6 by Jeff Forcier <jeff@…>, 17 years ago

Resolution: invalid
Status: closedreopened

I'm getting this too, as of SVN changeset 5677, with databrowse. Fairly stock Apache2 + mod_python setup, Python 2.5, Linux.

In my situation, I'm just trying to use databrowse on a brand-new app with three simple model classes, and get a similar error. The offending lazy object is one of my Model objects' nonexistent Meta.verbose_name_plural - in other words, the databrowse template is trying to access MyModel._meta.verbose_name_plural (which is not defined), and at least in the Python shell, that attribute results in a lazy object. My other two classes in the same file do define verbose_name_plural, and accessing the attribute on them results in regular strings.

I've toyed around in the shell and read over the debug page and can't figure out exactly what's going wrong. It appears that in normal usage, calling force_unicode or unicode on a lazy object works as you'd expect (i.e. it takes the output of the stored function and makes it into a Unicode string); even when I del a lazy object's __unicode__ attribute/method, unicode still works by returns a string representation of the object, i.e. u'<django.utils.functional.__proxy__ object at 0x8606a2c>'.

by Jeff Forcier <jeff@…>, 17 years ago

comment:7 by Jeff Forcier <jeff@…>, 17 years ago

Component: Admin interfaceInternationalization
Version: newforms-adminSVN

Oh, I should probably note that I am on regular trunk and not newforms-admin.

comment:8 by Jeff Forcier <jeff@…>, 17 years ago

Component: InternationalizationAdmin interface
Version: SVNnewforms-admin

FFS. Didn't mean to save the changes I'd made to the dropdowns.

comment:9 by Jeff Forcier <jeff@…>, 17 years ago

Final comment for the night: the issue does not occur under the development runserver, and it is also avoided by explicitly defining verbose_name_plural on the third object in my model. Haven't been able to figure out the ramifications of either.

comment:10 by fpierfed@…, 17 years ago

I cannot find anything wrong in the traceback page: all string variables are unicode objects as one would expect.

comment:11 by Jeff Forcier <reg@…>, 17 years ago

Yes, that's the rub. The traceback page's printing of s causes it to happily show up as u'jobs'. However, during my investigation, I at one point had a line just prior to the s = unicode(s), to the effect of foo = s.__unicode__() and that made the process work - the __proxy__ related error did not appear and databrowse worked fine.

So in this case what I think is happening is the s = unicode(s) line is erroring out, but at the same time it's "unblocking" the lazy aspect of s, so that when it is asked to yield itself as a (Unicode?) string in the debug printout, it does so.

comment:12 by Chris Beaven, 17 years ago

Component: Admin interfaceCore framework
Owner: changed from Adrian Holovaty to Malcolm Tredinnick
Status: reopenednew
Summary: Accessing admin pages as a logged in user crashes under apache + fcgiforce_unicode error
Triage Stage: UnreviewedAccepted
Version: newforms-admin

I'm promoting to accepted - it does look like a valid bug. Also noted in #5146, which I closed as a dupe of this.

by Malcolm Tredinnick, 17 years ago

Attachment: 4796-hopeful-fix.diff added

If somebody can reproduce the problem, does this patch fix it?

comment:13 by Malcolm Tredinnick, 17 years ago

Cc: jeff@… added

I cannot reproduce this problem at all. Staring at the code for a long time doesn't make things any clearer either. The only thing I can imagine it might be is a race condition or some other problem at import time. This might have been fixed in [5919].

If anybody can repeat the problem with latest subversion code, does the 4796-hopeful-fix.diff patch make it go away? It removes a potential race condition in i18n initialisation, although I seriously doubt that is the problem here -- it's too reliably repeatable for some people for that to be the case. If you can repeat the problem with something that isn't confidential or megabytes in size, can you attach a copy of the models that fail (as a plain text file -- gzipped files won't upload to Trac), please? Without more information or a way to repeat this, I'm a bit stuck at the moment.

Oh .. and if you can repeat it, what is the type of EasyModel.verbose_name_plural (put a print statement in contrib/databrowse/datastructures.py in EasyModel.__init__)?

(Temporarily adding Jeff Forcier to the cc list, since he had the best information last time. Feel free to remove yourself, Jeff, if you're not following this any longer.)

comment:14 by fpierfed@…, 17 years ago

Nope, the patch is a no go for me:

Traceback (most recent call last):
File "/home/fpierfed/lib/python2.4/site-packages/django/core/handlers/base.py" in get_response
  77. response = callback(request, *callback_args, **callback_kwargs)
File "/home/fpierfed/lib/python2.4/site-packages/django/contrib/admin/sites.py" in root
  115. return self.index(request)
File "/home/fpierfed/lib/python2.4/site-packages/django/contrib/admin/sites.py" in index
  273. app['models'].sort(lambda x, y: cmp(x['name'], y['name']))
File "/home/fpierfed/lib/python2.4/site-packages/django/contrib/admin/sites.py" in
  273. app['models'].sort(lambda x, y: cmp(x['name'], y['name']))
File "/home/fpierfed/lib/python2.4/site-packages/django/utils/functional.py" in __cmp__
  88. s = unicode(self.__func(*self.__args, **self.__kw))
File "/home/fpierfed/lib/python2.4/site-packages/django/utils/text.py" in
  8. capfirst = lambda x: x and force_unicode(x)[0].upper() + force_unicode(x)[1:]
File "/home/fpierfed/lib/python2.4/site-packages/django/utils/encoding.py" in force_unicode
  37. s = unicode(s)

  TypeError at /tips/add/
  coercing to Unicode: need string or buffer, __proxy__ found

Funny thing is that I put a

        f = file('/tmp/pippo', 'w')
        f.write(self.verbose_name_plural)
        f.write('\n')
        f.close()

at the botton of

EasyModel.__init__()

but that never gets called and the file is simply not there. Maybe I am doing something wrong here...

Anyway, here is my model

class Tag(models.Model):
    name = models.CharField(maxlength=255)
    
    def __str__(self):
        return(self.name)


class Location(models.Model):
    address = models.CharField(maxlength=255, blank=True, null=True)
    city = models.CharField(maxlength=255, blank=True, null=True)
    state = models.CharField(maxlength=255, blank=True, null=True)
    country = models.CharField(maxlength=255)
    region = models.CharField(maxlength=30)
    zip_code = models.IntegerField(blank=True, null=True)
    longitude = models.FloatField(blank=True, null=True)
    latitude = models.FloatField(blank=True, null=True)
    last_seen = models.DateTimeField(default=datetime.now())
    expired = models.BooleanField()

    def __str__(self):
        label = ''
        if(self.address):
            label = '%s, ' %(self.address)
        if(self.city):
            label += '%s ' %(self.city)
        if(self.state):
            label += '%s ' %(self.state)
        if(self.zip_code):
            label += '%s, ' %(self.zip_code)
        if(self.country and not label):
            label = self.country
        elif(self.country):
            label = label.strip() + ', %s' %(self.country)
        return(label)


class Tip(models.Model):
    title = models.CharField(maxlength=255)
    description = models.TextField()
    
    author = models.ForeignKey(User)
    date = models.DateTimeField(auto_now_add=True)
    
    rating = models.FloatField()
    expired = models.BooleanField()
    
    tags = models.ManyToManyField(Tag)
    location = models.ForeignKey(Location)

    def __str__(self):
        return(self.title)


class Vote(models.Model):
    rating = models.IntegerField()
    comment = models.TextField()
    user = models.ForeignKey(User)
    tip = models.ForeignKey(Tip)

    def __str__(self):
        return(str(self.rating))


class UserProfile(models.Model):
    user = models.ForeignKey(User, unique=True)
    picture = models.ImageField(upload_to='/tmp', blank=True, null=True)
    locations = models.ManyToManyField(Location)

comment:15 by Malcolm Tredinnick, 17 years ago

@fpierfed: since the changes from [5919] haven't been merged into newforms-admin yet, your test isn't going to be exercising those. And the request for information from EasyModel was only relevant for those seeing the problem via the databrowse app (which was happening to like Jeff, who were using trunk).

However, this has given me the idea to try and repeat it on the newforms-admin branch. What does your admin registration code look like for these models (just paste the extra bit, I'll merge it with what you've pasted above)?

comment:16 by Malcolm Tredinnick, 17 years ago

#4813 might be the same issue in another disguise. Or it might be totally unrelated.

comment:17 by fpierfed@…, 17 years ago

Here it is:

tips_admin = admin.AdminSite()
tips_admin.register(Location)
tips_admin.register(Tag)
tips_admin.register(Tip)

and then, in urls.py

(r'^add/(.*)', tips_admin.root),

comment:18 by Jeff Forcier <jeff@…>, 17 years ago

I am sad to report that after doing an svn up (to [5947]) the issue persists, and applying the specified diff makes no apparent difference. The error remains exactly the same as what I reported before.

comment:19 by daniel@…, 17 years ago

I'm having what I believe to be at least a related issue, using SVN r5993. The patch doesn't help at all, and the bt remains the same.

Here's my models:

class ExtendedUser(models.Model):
    user = models.ForeignKey(User)
    first_name = models.CharField(_("First name"), maxlength=128)
    surname = models.CharField(_("Surname"), maxlength=128)
    surname_first = models.BooleanField(_("Surname first"), default=False)
    url = models.URLField(_("URL"), blank=True, null=True)
    confirmation_string = models.CharField(_("Email validation string"),
                                           maxlength=32, blank=True, null=True)

    # The user's identifier is their name.
    def __unicode__(self):
        if self.surname_first:
            return "%s %s" % (self.surname, self.first_name)
        else:
            return "%s %s" % (self.first_name, self.surname)
    
    # Allow administration, with all the defaults.
    class Admin:
        pass

class Attendee(models.Model):
    user = models.ForeignKey(ExtendedUser)
    [...]

Running the following code from a view:

def detail(request, attendee_id):
    if not request.user.is_anonymous():
        user = ExtendedUser.objects.get(user=request.user)
    else:
        user = None
    print "%s" % user

    attendee = get_object_or_404(Attendee, id=attendee_id)
    print "%s" % attendee.user
    print "%s" % attendee

Will print 'Daniel Stone' twice (with the first and second prints), but tank with the following backtrace on the third:

Traceback (most recent call last):
File "/usr/lib/python2.5/site-packages/django/core/handlers/base.py" in get_response
  77. response = callback(request, *callback_args, **callback_kwargs)
File "/home/daniels/x/xds/site/endtroducing/../endtroducing/xds2007/attendees/views.py" in detail
  47. print "%s" % attendee
File "/usr/lib/python2.5/site-packages/django/db/models/base.py" in __str__
  91. return force_unicode(self).encode('utf-8')
File "/usr/lib/python2.5/site-packages/django/utils/encoding.py" in force_unicode
  37. s = unicode(s)

  TypeError at /xds2007/attendees/1/
  coercing to Unicode: need string or buffer, ExtendedUser found

Hope that helps.

comment:20 by Myers Carpenter, 17 years ago

Owner: changed from nobody to Myers Carpenter

comment:21 by Nis Jørgensen <nis@…>, 17 years ago

Cc: daniel@… added

Daniel, your example does not include the definition of the str or unicode method of Attendee. It looks like you might be having

class Attendee(models.Model):

[...]
def unicode(self):

return self.user

(this gives me a stacktrace resembling yours)

Which should be something like

class Attendee(models.Model):

[...]
def unicode(self):

return unicode(self.user)

comment:22 by daniel@…, 17 years ago

Yeah, your deduction is correct. I can't try it at the moment, but if that's the case, should it not be documented?

comment:23 by Myers Carpenter, 17 years ago

Daniel: self.user is a model; _ _ unicode _ _ needs to return a unicode object.

As far as Jeff's issue goes, I'm reproducing it with apache 2.2, mod_python, django_newforms-admin (latest), and python 2.5. If I sit on the admin page and click my reload button repeatedly, the admin page alternates with the "coercing to Unicode" TypeError traceback. Question is, can I fix it :)

comment:24 by Myers Carpenter, 17 years ago

Cc: daniel@… removed
  • If I change Apache from a worker to a prefork MPM, it consistently barfs every time (instead of every other time).
  • As Jeff indicates in comment #11, adding foo = s.__unicode__() just above the s = unicode(s) statement in force_unicode magically makes things work.
  • the above patch doesn't have any effect.
  • it only appears to happen the first time force_unicode is called (per request) -- when I tried logging foo & s like so:
        if not isinstance(s, basestring,):
            if hasattr(s, '__unicode__'):
                foo = s.__unicode__()
                file('wtf.log','a').write('repr(foo): %s\n' % repr(foo))
                file('wtf.log','a').write('repr(s): %s\n\n' % repr(s))
                s = unicode(s)
            else:
                ...
    

... which resulted in:

repr(foo): <django.utils.functional.__proxy__ object at 0x86f1cec>
repr(s): <django.utils.functional.__proxy__ object at 0x86853ec>

repr(foo): u'locations'
repr(s): <django.utils.functional.__proxy__ object at 0x86853ec>

repr(foo): u'tags'
repr(s): <django.utils.functional.__proxy__ object at 0x86815cc>

...

Interestingly, the proxy object from the first __unicode__() call is different from the one we supplied. A further __unicode__() call to both the foo and s objects returns the same string, however.

I've thrown a tarball of the barebones skeleton project I'm using (based on fpierfed's model above) up here if anyone else wants to take a poke at it. This is a tough one.

comment:25 by Myers Carpenter, 17 years ago

(taking myself off this one, in case someone else feels like sprinting on it instead)

comment:26 by Myers Carpenter, 17 years ago

Owner: Myers Carpenter removed

argh! actually reassigning helps :)

comment:27 by anonymous, 17 years ago

Cc: mmulley@… added

comment:28 by William McVey <wam+djangobug@…>, 17 years ago

Keywords: databrowse added
Version: SVN

Another datapoint for everyone. I'm having the same problem with unicode/proxy objects while using Databrowse as well. I've confirmed that all my str() methods are in fact returned as strings. The problem only happens if I leave MyModel.META.verbose_name_plural undefined. I've gone in and have defined verbose_name_plural for all my models and databrowse now works. I'm running the mainline development Django svn release 6026.

by durdinator, 17 years ago

Attachment: bug4796.tar.bz2 added

Minimal testcase

comment:29 by durdinator, 17 years ago

Cc: me@… added
Has patch: set

The patch 4796-fix.diff fixes the problem for me.

However, I couldn't quite pin down what the cause of the error was. Apparently mod_python does some weird shit when handling globals() or loading modules or something like that...

comment:30 by durdinator, 17 years ago

Just for clarification, the minimal testcase was running against trunk, revision 6380.

comment:31 by Myers Carpenter, 17 years ago

Took care of it for newforms-admin rev. 6403 also.

by durdinator, 17 years ago

Attachment: 4796-fix.diff added

Patch, with regression test (fix typo)

comment:32 by Malcolm Tredinnick, 17 years ago

Resolution: fixed
Status: newclosed

(In [6446]) Fixed #4796. Fixed a problem when using i18n support for the first time -- in
particular when string_concat() was the first call made. Thanks, Andy Durdin.

comment:33 by niosop@…, 17 years ago

Resolution: fixed
Status: closedreopened

I can confirm that this problem is still present in the newforms-admin branch rev 6453. Adding a try block around the offending s = unicode(s) and re-executing s = unicode(s) upon exception fixes it, in accordance with Jeffs explanation of it being "de-proxied" when accessed the first time.

I'm using apache2.2.3 and mod_proxy 3.2.10-3ub on Ubuntu.

comment:34 by niosop@…, 17 years ago

Resolution: fixed
Status: reopenedclosed

Disregard previous, still a problem in latest newforms-admin svn, but applying above patch fixes the problem.

comment:35 by James Bennett, 17 years ago

The newforms-admin branch is periodically synced with trunk so that it can pick up bugfixes. The next time that happens it'll pick up the fix for this, and in the meantime it's really not worth adding additional patches or messing with the ticket status.

comment:36 by joseph8th@…, 12 years ago

Easy pickings: unset
Severity: Normal
Type: Uncategorized
UI/UX: unset

I had the same problem but found that I was returning the object in my unicode instead of returning unicode(self.object) in the model. Fixed it right up.

Note: See TracTickets for help on using tickets.
Back to Top