Opened 9 years ago

Closed 9 years ago

Last modified 4 years ago

#4796 closed Uncategorized (fixed)

force_unicode error

Reported by: fpierfed@… Owned by:
Component: Core (Other) Version: master
Severity: Normal Keywords: fcgi unicode __proxy__ databrowse
Cc: jeff@…, mmulley@…, me@… Triage Stage: Accepted
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description (last modified by Malcolm Tredinnick)

Hi,

I am using the newforms-admin branch from SVN (revision 5632) under apache 2.0.54, flup SVN (revision 2354) and Python 2.4.1. I have a webapp that uses custom admin sites via myadmin = admin.AdminSite() and myadmin.register(...)

If I access the admin site via the django internal development server, everything works. If I use fcgi the admin pages work only if I am an anonymous user (in that case I am redirected to the login screen, then to the admin pages and all is good). If I use fcgi and I am already logged in, accessing the admin pages gives this error:

TypeError at /tips/admin/
coercing to Unicode: need string or buffer, __proxy__ found
Request Method: 	GET
Request URL: 	http://www.peertraveller.com/tips/admin/
Exception Type: 	TypeError
Exception Value: 	coercing to Unicode: need string or buffer, __proxy__ found
Exception Location: 	/home/fpierfed/lib/python2.4/site-packages/django/utils/encoding.py in force_unicode, line 38
Python Executable: 	/home/fpierfed/bin/python
Python Version: 	2.4.1

Attachments (4)

TypeError at _databrowse_.html (49.8 KB) - added by Jeff Forcier <jeff@…> 9 years ago.
4796-hopeful-fix.diff (2.2 KB) - added by Malcolm Tredinnick 9 years ago.
If somebody can reproduce the problem, does this patch fix it?
bug4796.tar.bz2 (1.1 KB) - added by durdinator 9 years ago.
Minimal testcase
4796-fix.diff (1.1 KB) - added by durdinator 9 years ago.
Patch, with regression test (fix typo)

Download all attachments as: .zip

Change History (40)

comment:1 Changed 9 years ago by Malcolm Tredinnick

Description: modified (diff)

Fixed description.

comment:2 Changed 9 years ago by Malcolm Tredinnick

We'll need the full traceback to be able to debug this -- you've only provided the last line. Look a bit further down the debug screen and find the link that says "cut-and-paste traceback". Click on that and the paste in the output (remember to wrap it in {{{ and }}} markup so that it's formatted correctly in Trac's wiki syntax.

comment:3 Changed 9 years ago by fpierfed@…

Here is the full traceback:

Traceback (most recent call last):
File "/home/fpierfed/lib/python2.4/site-packages/django/core/handlers/base.py" in get_response
  77. response = callback(request, *callback_args, **callback_kwargs)
File "/home/fpierfed/lib/python2.4/site-packages/django/contrib/admin/sites.py" in root
  115. return self.index(request)
File "/home/fpierfed/lib/python2.4/site-packages/django/contrib/admin/sites.py" in index
  273. app['models'].sort(lambda x, y: cmp(x['name'], y['name']))
File "/home/fpierfed/lib/python2.4/site-packages/django/contrib/admin/sites.py" in
  273. app['models'].sort(lambda x, y: cmp(x['name'], y['name']))
File "/home/fpierfed/lib/python2.4/site-packages/django/utils/functional.py" in __cmp__
  86. s = unicode(self.__func(*self.__args, **self.__kw))
File "/home/fpierfed/lib/python2.4/site-packages/django/utils/text.py" in
  7. capfirst = lambda x: x and force_unicode(x)[0].upper() + force_unicode(x)[1:]
File "/home/fpierfed/lib/python2.4/site-packages/django/utils/encoding.py" in force_unicode
  38. s = unicode(s)

  TypeError at /tips/add/
  coercing to Unicode: need string or buffer, __proxy__ found

comment:4 Changed 9 years ago by Malcolm Tredinnick

So this traceback means the __unicode__ method on a model is returning a lazy translation object, which is illegal Python code (__unicode__ must return a Unicode object).

It's not possible to tell from the traceback which model is involved, although as far as I can see, no models in Django's source are doing this.

Have a look at your debug traceback screen again and examine the local variables in the second last line (the "capfirst" line). That should tell you which object is involved. Or, if that doesn't, have a look at the local variables in one of the previous lines (the ones from sites.py).

If the object involved is in Django's code, please let us know which one.

comment:5 Changed 9 years ago by Simon G. <dev@…>

Resolution: invalid
Status: newclosed

Resolving as invalid as it doesn't appear to be Django-specific. Please reopen if I'm wrong!

comment:6 Changed 9 years ago by Jeff Forcier <jeff@…>

Resolution: invalid
Status: closedreopened

I'm getting this too, as of SVN changeset 5677, with databrowse. Fairly stock Apache2 + mod_python setup, Python 2.5, Linux.

In my situation, I'm just trying to use databrowse on a brand-new app with three simple model classes, and get a similar error. The offending lazy object is one of my Model objects' nonexistent Meta.verbose_name_plural - in other words, the databrowse template is trying to access MyModel._meta.verbose_name_plural (which is not defined), and at least in the Python shell, that attribute results in a lazy object. My other two classes in the same file do define verbose_name_plural, and accessing the attribute on them results in regular strings.

I've toyed around in the shell and read over the debug page and can't figure out exactly what's going wrong. It appears that in normal usage, calling force_unicode or unicode on a lazy object works as you'd expect (i.e. it takes the output of the stored function and makes it into a Unicode string); even when I del a lazy object's __unicode__ attribute/method, unicode still works by returns a string representation of the object, i.e. u'<django.utils.functional.__proxy__ object at 0x8606a2c>'.

Changed 9 years ago by Jeff Forcier <jeff@…>

comment:7 Changed 9 years ago by Jeff Forcier <jeff@…>

Component: Admin interfaceInternationalization
Version: newforms-adminSVN

Oh, I should probably note that I am on regular trunk and not newforms-admin.

comment:8 Changed 9 years ago by Jeff Forcier <jeff@…>

Component: InternationalizationAdmin interface
Version: SVNnewforms-admin

FFS. Didn't mean to save the changes I'd made to the dropdowns.

comment:9 Changed 9 years ago by Jeff Forcier <jeff@…>

Final comment for the night: the issue does not occur under the development runserver, and it is also avoided by explicitly defining verbose_name_plural on the third object in my model. Haven't been able to figure out the ramifications of either.

comment:10 Changed 9 years ago by fpierfed@…

I cannot find anything wrong in the traceback page: all string variables are unicode objects as one would expect.

comment:11 Changed 9 years ago by Jeff Forcier <reg@…>

Yes, that's the rub. The traceback page's printing of s causes it to happily show up as u'jobs'. However, during my investigation, I at one point had a line just prior to the s = unicode(s), to the effect of foo = s.__unicode__() and that made the process work - the __proxy__ related error did not appear and databrowse worked fine.

So in this case what I think is happening is the s = unicode(s) line is erroring out, but at the same time it's "unblocking" the lazy aspect of s, so that when it is asked to yield itself as a (Unicode?) string in the debug printout, it does so.

comment:12 Changed 9 years ago by Chris Beaven

Component: Admin interfaceCore framework
Owner: changed from Adrian Holovaty to Malcolm Tredinnick
Status: reopenednew
Summary: Accessing admin pages as a logged in user crashes under apache + fcgiforce_unicode error
Triage Stage: UnreviewedAccepted
Version: newforms-admin

I'm promoting to accepted - it does look like a valid bug. Also noted in #5146, which I closed as a dupe of this.

Changed 9 years ago by Malcolm Tredinnick

Attachment: 4796-hopeful-fix.diff added

If somebody can reproduce the problem, does this patch fix it?

comment:13 Changed 9 years ago by Malcolm Tredinnick

Cc: jeff@… added

I cannot reproduce this problem at all. Staring at the code for a long time doesn't make things any clearer either. The only thing I can imagine it might be is a race condition or some other problem at import time. This might have been fixed in [5919].

If anybody can repeat the problem with latest subversion code, does the 4796-hopeful-fix.diff patch make it go away? It removes a potential race condition in i18n initialisation, although I seriously doubt that is the problem here -- it's too reliably repeatable for some people for that to be the case. If you can repeat the problem with something that isn't confidential or megabytes in size, can you attach a copy of the models that fail (as a plain text file -- gzipped files won't upload to Trac), please? Without more information or a way to repeat this, I'm a bit stuck at the moment.

Oh .. and if you can repeat it, what is the type of EasyModel.verbose_name_plural (put a print statement in contrib/databrowse/datastructures.py in EasyModel.__init__)?

(Temporarily adding Jeff Forcier to the cc list, since he had the best information last time. Feel free to remove yourself, Jeff, if you're not following this any longer.)

comment:14 Changed 9 years ago by fpierfed@…

Nope, the patch is a no go for me:

Traceback (most recent call last):
File "/home/fpierfed/lib/python2.4/site-packages/django/core/handlers/base.py" in get_response
  77. response = callback(request, *callback_args, **callback_kwargs)
File "/home/fpierfed/lib/python2.4/site-packages/django/contrib/admin/sites.py" in root
  115. return self.index(request)
File "/home/fpierfed/lib/python2.4/site-packages/django/contrib/admin/sites.py" in index
  273. app['models'].sort(lambda x, y: cmp(x['name'], y['name']))
File "/home/fpierfed/lib/python2.4/site-packages/django/contrib/admin/sites.py" in
  273. app['models'].sort(lambda x, y: cmp(x['name'], y['name']))
File "/home/fpierfed/lib/python2.4/site-packages/django/utils/functional.py" in __cmp__
  88. s = unicode(self.__func(*self.__args, **self.__kw))
File "/home/fpierfed/lib/python2.4/site-packages/django/utils/text.py" in
  8. capfirst = lambda x: x and force_unicode(x)[0].upper() + force_unicode(x)[1:]
File "/home/fpierfed/lib/python2.4/site-packages/django/utils/encoding.py" in force_unicode
  37. s = unicode(s)

  TypeError at /tips/add/
  coercing to Unicode: need string or buffer, __proxy__ found

Funny thing is that I put a

        f = file('/tmp/pippo', 'w')
        f.write(self.verbose_name_plural)
        f.write('\n')
        f.close()

at the botton of

EasyModel.__init__()

but that never gets called and the file is simply not there. Maybe I am doing something wrong here...

Anyway, here is my model

class Tag(models.Model):
    name = models.CharField(maxlength=255)
    
    def __str__(self):
        return(self.name)


class Location(models.Model):
    address = models.CharField(maxlength=255, blank=True, null=True)
    city = models.CharField(maxlength=255, blank=True, null=True)
    state = models.CharField(maxlength=255, blank=True, null=True)
    country = models.CharField(maxlength=255)
    region = models.CharField(maxlength=30)
    zip_code = models.IntegerField(blank=True, null=True)
    longitude = models.FloatField(blank=True, null=True)
    latitude = models.FloatField(blank=True, null=True)
    last_seen = models.DateTimeField(default=datetime.now())
    expired = models.BooleanField()

    def __str__(self):
        label = ''
        if(self.address):
            label = '%s, ' %(self.address)
        if(self.city):
            label += '%s ' %(self.city)
        if(self.state):
            label += '%s ' %(self.state)
        if(self.zip_code):
            label += '%s, ' %(self.zip_code)
        if(self.country and not label):
            label = self.country
        elif(self.country):
            label = label.strip() + ', %s' %(self.country)
        return(label)


class Tip(models.Model):
    title = models.CharField(maxlength=255)
    description = models.TextField()
    
    author = models.ForeignKey(User)
    date = models.DateTimeField(auto_now_add=True)
    
    rating = models.FloatField()
    expired = models.BooleanField()
    
    tags = models.ManyToManyField(Tag)
    location = models.ForeignKey(Location)

    def __str__(self):
        return(self.title)


class Vote(models.Model):
    rating = models.IntegerField()
    comment = models.TextField()
    user = models.ForeignKey(User)
    tip = models.ForeignKey(Tip)

    def __str__(self):
        return(str(self.rating))


class UserProfile(models.Model):
    user = models.ForeignKey(User, unique=True)
    picture = models.ImageField(upload_to='/tmp', blank=True, null=True)
    locations = models.ManyToManyField(Location)

comment:15 Changed 9 years ago by Malcolm Tredinnick

@fpierfed: since the changes from [5919] haven't been merged into newforms-admin yet, your test isn't going to be exercising those. And the request for information from EasyModel was only relevant for those seeing the problem via the databrowse app (which was happening to like Jeff, who were using trunk).

However, this has given me the idea to try and repeat it on the newforms-admin branch. What does your admin registration code look like for these models (just paste the extra bit, I'll merge it with what you've pasted above)?

comment:16 Changed 9 years ago by Malcolm Tredinnick

#4813 might be the same issue in another disguise. Or it might be totally unrelated.

comment:17 Changed 9 years ago by fpierfed@…

Here it is:

tips_admin = admin.AdminSite()
tips_admin.register(Location)
tips_admin.register(Tag)
tips_admin.register(Tip)

and then, in urls.py

(r'^add/(.*)', tips_admin.root),

comment:18 Changed 9 years ago by Jeff Forcier <jeff@…>

I am sad to report that after doing an svn up (to [5947]) the issue persists, and applying the specified diff makes no apparent difference. The error remains exactly the same as what I reported before.

comment:19 Changed 9 years ago by daniel@…

I'm having what I believe to be at least a related issue, using SVN r5993. The patch doesn't help at all, and the bt remains the same.

Here's my models:

class ExtendedUser(models.Model):
    user = models.ForeignKey(User)
    first_name = models.CharField(_("First name"), maxlength=128)
    surname = models.CharField(_("Surname"), maxlength=128)
    surname_first = models.BooleanField(_("Surname first"), default=False)
    url = models.URLField(_("URL"), blank=True, null=True)
    confirmation_string = models.CharField(_("Email validation string"),
                                           maxlength=32, blank=True, null=True)

    # The user's identifier is their name.
    def __unicode__(self):
        if self.surname_first:
            return "%s %s" % (self.surname, self.first_name)
        else:
            return "%s %s" % (self.first_name, self.surname)
    
    # Allow administration, with all the defaults.
    class Admin:
        pass

class Attendee(models.Model):
    user = models.ForeignKey(ExtendedUser)
    [...]

Running the following code from a view:

def detail(request, attendee_id):
    if not request.user.is_anonymous():
        user = ExtendedUser.objects.get(user=request.user)
    else:
        user = None
    print "%s" % user

    attendee = get_object_or_404(Attendee, id=attendee_id)
    print "%s" % attendee.user
    print "%s" % attendee

Will print 'Daniel Stone' twice (with the first and second prints), but tank with the following backtrace on the third:

Traceback (most recent call last):
File "/usr/lib/python2.5/site-packages/django/core/handlers/base.py" in get_response
  77. response = callback(request, *callback_args, **callback_kwargs)
File "/home/daniels/x/xds/site/endtroducing/../endtroducing/xds2007/attendees/views.py" in detail
  47. print "%s" % attendee
File "/usr/lib/python2.5/site-packages/django/db/models/base.py" in __str__
  91. return force_unicode(self).encode('utf-8')
File "/usr/lib/python2.5/site-packages/django/utils/encoding.py" in force_unicode
  37. s = unicode(s)

  TypeError at /xds2007/attendees/1/
  coercing to Unicode: need string or buffer, ExtendedUser found

Hope that helps.

comment:20 Changed 9 years ago by Andrew Myers

Owner: changed from nobody to Andrew Myers

comment:21 Changed 9 years ago by Nis Jørgensen <nis@…>

Cc: daniel@… added

Daniel, your example does not include the definition of the str or unicode method of Attendee. It looks like you might be having

class Attendee(models.Model):

[...]
def unicode(self):

return self.user

(this gives me a stacktrace resembling yours)

Which should be something like

class Attendee(models.Model):

[...]
def unicode(self):

return unicode(self.user)

comment:22 Changed 9 years ago by daniel@…

Yeah, your deduction is correct. I can't try it at the moment, but if that's the case, should it not be documented?

comment:23 Changed 9 years ago by Andrew Myers

Daniel: self.user is a model; _ _ unicode _ _ needs to return a unicode object.

As far as Jeff's issue goes, I'm reproducing it with apache 2.2, mod_python, django_newforms-admin (latest), and python 2.5. If I sit on the admin page and click my reload button repeatedly, the admin page alternates with the "coercing to Unicode" TypeError traceback. Question is, can I fix it :)

comment:24 Changed 9 years ago by Andrew Myers

Cc: daniel@… removed
  • If I change Apache from a worker to a prefork MPM, it consistently barfs every time (instead of every other time).
  • As Jeff indicates in comment #11, adding foo = s.__unicode__() just above the s = unicode(s) statement in force_unicode magically makes things work.
  • the above patch doesn't have any effect.
  • it only appears to happen the first time force_unicode is called (per request) -- when I tried logging foo & s like so:
        if not isinstance(s, basestring,):
            if hasattr(s, '__unicode__'):
                foo = s.__unicode__()
                file('wtf.log','a').write('repr(foo): %s\n' % repr(foo))
                file('wtf.log','a').write('repr(s): %s\n\n' % repr(s))
                s = unicode(s)
            else:
                ...
    

... which resulted in:

repr(foo): <django.utils.functional.__proxy__ object at 0x86f1cec>
repr(s): <django.utils.functional.__proxy__ object at 0x86853ec>

repr(foo): u'locations'
repr(s): <django.utils.functional.__proxy__ object at 0x86853ec>

repr(foo): u'tags'
repr(s): <django.utils.functional.__proxy__ object at 0x86815cc>

...

Interestingly, the proxy object from the first __unicode__() call is different from the one we supplied. A further __unicode__() call to both the foo and s objects returns the same string, however.

I've thrown a tarball of the barebones skeleton project I'm using (based on fpierfed's model above) up here if anyone else wants to take a poke at it. This is a tough one.

comment:25 Changed 9 years ago by Andrew Myers

(taking myself off this one, in case someone else feels like sprinting on it instead)

comment:26 Changed 9 years ago by Andrew Myers

Owner: Andrew Myers deleted

argh! actually reassigning helps :)

comment:27 Changed 9 years ago by anonymous

Cc: mmulley@… added

comment:28 Changed 9 years ago by William McVey <wam+djangobug@…>

Keywords: databrowse added
Version: SVN

Another datapoint for everyone. I'm having the same problem with unicode/proxy objects while using Databrowse as well. I've confirmed that all my str() methods are in fact returned as strings. The problem only happens if I leave MyModel.META.verbose_name_plural undefined. I've gone in and have defined verbose_name_plural for all my models and databrowse now works. I'm running the mainline development Django svn release 6026.

Changed 9 years ago by durdinator

Attachment: bug4796.tar.bz2 added

Minimal testcase

comment:29 Changed 9 years ago by durdinator

Cc: me@… added
Has patch: set

The patch 4796-fix.diff fixes the problem for me.

However, I couldn't quite pin down what the cause of the error was. Apparently mod_python does some weird shit when handling globals() or loading modules or something like that...

comment:30 Changed 9 years ago by durdinator

Just for clarification, the minimal testcase was running against trunk, revision 6380.

comment:31 Changed 9 years ago by Andrew Myers

Took care of it for newforms-admin rev. 6403 also.

Changed 9 years ago by durdinator

Attachment: 4796-fix.diff added

Patch, with regression test (fix typo)

comment:32 Changed 9 years ago by Malcolm Tredinnick

Resolution: fixed
Status: newclosed

(In [6446]) Fixed #4796. Fixed a problem when using i18n support for the first time -- in
particular when string_concat() was the first call made. Thanks, Andy Durdin.

comment:33 Changed 9 years ago by niosop@…

Resolution: fixed
Status: closedreopened

I can confirm that this problem is still present in the newforms-admin branch rev 6453. Adding a try block around the offending s = unicode(s) and re-executing s = unicode(s) upon exception fixes it, in accordance with Jeffs explanation of it being "de-proxied" when accessed the first time.

I'm using apache2.2.3 and mod_proxy 3.2.10-3ub on Ubuntu.

comment:34 Changed 9 years ago by niosop@…

Resolution: fixed
Status: reopenedclosed

Disregard previous, still a problem in latest newforms-admin svn, but applying above patch fixes the problem.

comment:35 Changed 9 years ago by James Bennett

The newforms-admin branch is periodically synced with trunk so that it can pick up bugfixes. The next time that happens it'll pick up the fix for this, and in the meantime it's really not worth adding additional patches or messing with the ticket status.

comment:36 Changed 4 years ago by joseph8th@…

Easy pickings: unset
Severity: Normal
Type: Uncategorized
UI/UX: unset

I had the same problem but found that I was returning the object in my unicode instead of returning unicode(self.object) in the model. Fixed it right up.

Note: See TracTickets for help on using tickets.
Back to Top