Opened 8 years ago

Closed 7 years ago

Last modified 3 years ago

#4796 closed Uncategorized (fixed)

force_unicode error

Reported by: fpierfed@… Owned by:
Component: Core (Other) Version: master
Severity: Normal Keywords: fcgi unicode __proxy__ databrowse
Cc: jeff@…, mmulley@…, me@… Triage Stage: Accepted
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description (last modified by mtredinnick)

Hi,

I am using the newforms-admin branch from SVN (revision 5632) under apache 2.0.54, flup SVN (revision 2354) and Python 2.4.1. I have a webapp that uses custom admin sites via myadmin = admin.AdminSite() and myadmin.register(...)

If I access the admin site via the django internal development server, everything works. If I use fcgi the admin pages work only if I am an anonymous user (in that case I am redirected to the login screen, then to the admin pages and all is good). If I use fcgi and I am already logged in, accessing the admin pages gives this error:

TypeError at /tips/admin/
coercing to Unicode: need string or buffer, __proxy__ found
Request Method: 	GET
Request URL: 	http://www.peertraveller.com/tips/admin/
Exception Type: 	TypeError
Exception Value: 	coercing to Unicode: need string or buffer, __proxy__ found
Exception Location: 	/home/fpierfed/lib/python2.4/site-packages/django/utils/encoding.py in force_unicode, line 38
Python Executable: 	/home/fpierfed/bin/python
Python Version: 	2.4.1

Attachments (4)

TypeError at _databrowse_.html (49.8 KB) - added by Jeff Forcier <jeff@…> 8 years ago.
4796-hopeful-fix.diff (2.2 KB) - added by mtredinnick 8 years ago.
If somebody can reproduce the problem, does this patch fix it?
bug4796.tar.bz2 (1.1 KB) - added by durdinator 8 years ago.
Minimal testcase
4796-fix.diff (1.1 KB) - added by durdinator 8 years ago.
Patch, with regression test (fix typo)

Download all attachments as: .zip

Change History (40)

comment:1 Changed 8 years ago by mtredinnick

  • Description modified (diff)
  • Needs documentation unset
  • Needs tests unset
  • Patch needs improvement unset

Fixed description.

comment:2 Changed 8 years ago by mtredinnick

We'll need the full traceback to be able to debug this -- you've only provided the last line. Look a bit further down the debug screen and find the link that says "cut-and-paste traceback". Click on that and the paste in the output (remember to wrap it in {{{ and }}} markup so that it's formatted correctly in Trac's wiki syntax.

comment:3 Changed 8 years ago by fpierfed@…

Here is the full traceback:

Traceback (most recent call last):
File "/home/fpierfed/lib/python2.4/site-packages/django/core/handlers/base.py" in get_response
  77. response = callback(request, *callback_args, **callback_kwargs)
File "/home/fpierfed/lib/python2.4/site-packages/django/contrib/admin/sites.py" in root
  115. return self.index(request)
File "/home/fpierfed/lib/python2.4/site-packages/django/contrib/admin/sites.py" in index
  273. app['models'].sort(lambda x, y: cmp(x['name'], y['name']))
File "/home/fpierfed/lib/python2.4/site-packages/django/contrib/admin/sites.py" in
  273. app['models'].sort(lambda x, y: cmp(x['name'], y['name']))
File "/home/fpierfed/lib/python2.4/site-packages/django/utils/functional.py" in __cmp__
  86. s = unicode(self.__func(*self.__args, **self.__kw))
File "/home/fpierfed/lib/python2.4/site-packages/django/utils/text.py" in
  7. capfirst = lambda x: x and force_unicode(x)[0].upper() + force_unicode(x)[1:]
File "/home/fpierfed/lib/python2.4/site-packages/django/utils/encoding.py" in force_unicode
  38. s = unicode(s)

  TypeError at /tips/add/
  coercing to Unicode: need string or buffer, __proxy__ found

comment:4 Changed 8 years ago by mtredinnick

So this traceback means the __unicode__ method on a model is returning a lazy translation object, which is illegal Python code (__unicode__ must return a Unicode object).

It's not possible to tell from the traceback which model is involved, although as far as I can see, no models in Django's source are doing this.

Have a look at your debug traceback screen again and examine the local variables in the second last line (the "capfirst" line). That should tell you which object is involved. Or, if that doesn't, have a look at the local variables in one of the previous lines (the ones from sites.py).

If the object involved is in Django's code, please let us know which one.

comment:5 Changed 8 years ago by Simon G. <dev@…>

  • Resolution set to invalid
  • Status changed from new to closed

Resolving as invalid as it doesn't appear to be Django-specific. Please reopen if I'm wrong!

comment:6 Changed 8 years ago by Jeff Forcier <jeff@…>

  • Resolution invalid deleted
  • Status changed from closed to reopened

I'm getting this too, as of SVN changeset 5677, with databrowse. Fairly stock Apache2 + mod_python setup, Python 2.5, Linux.

In my situation, I'm just trying to use databrowse on a brand-new app with three simple model classes, and get a similar error. The offending lazy object is one of my Model objects' nonexistent Meta.verbose_name_plural - in other words, the databrowse template is trying to access MyModel._meta.verbose_name_plural (which is not defined), and at least in the Python shell, that attribute results in a lazy object. My other two classes in the same file do define verbose_name_plural, and accessing the attribute on them results in regular strings.

I've toyed around in the shell and read over the debug page and can't figure out exactly what's going wrong. It appears that in normal usage, calling force_unicode or unicode on a lazy object works as you'd expect (i.e. it takes the output of the stored function and makes it into a Unicode string); even when I del a lazy object's __unicode__ attribute/method, unicode still works by returns a string representation of the object, i.e. u'<django.utils.functional.__proxy__ object at 0x8606a2c>'.

Changed 8 years ago by Jeff Forcier <jeff@…>

comment:7 Changed 8 years ago by Jeff Forcier <jeff@…>

  • Component changed from Admin interface to Internationalization
  • Version changed from newforms-admin to SVN

Oh, I should probably note that I am on regular trunk and not newforms-admin.

comment:8 Changed 8 years ago by Jeff Forcier <jeff@…>

  • Component changed from Internationalization to Admin interface
  • Version changed from SVN to newforms-admin

FFS. Didn't mean to save the changes I'd made to the dropdowns.

comment:9 Changed 8 years ago by Jeff Forcier <jeff@…>

Final comment for the night: the issue does not occur under the development runserver, and it is also avoided by explicitly defining verbose_name_plural on the third object in my model. Haven't been able to figure out the ramifications of either.

comment:10 Changed 8 years ago by fpierfed@…

I cannot find anything wrong in the traceback page: all string variables are unicode objects as one would expect.

comment:11 Changed 8 years ago by Jeff Forcier <reg@…>

Yes, that's the rub. The traceback page's printing of s causes it to happily show up as u'jobs'. However, during my investigation, I at one point had a line just prior to the s = unicode(s), to the effect of foo = s.__unicode__() and that made the process work - the __proxy__ related error did not appear and databrowse worked fine.

So in this case what I think is happening is the s = unicode(s) line is erroring out, but at the same time it's "unblocking" the lazy aspect of s, so that when it is asked to yield itself as a (Unicode?) string in the debug printout, it does so.

comment:12 Changed 8 years ago by SmileyChris

  • Component changed from Admin interface to Core framework
  • Owner changed from adrian to mtredinnick
  • Status changed from reopened to new
  • Summary changed from Accessing admin pages as a logged in user crashes under apache + fcgi to force_unicode error
  • Triage Stage changed from Unreviewed to Accepted
  • Version newforms-admin deleted

I'm promoting to accepted - it does look like a valid bug. Also noted in #5146, which I closed as a dupe of this.

Changed 8 years ago by mtredinnick

If somebody can reproduce the problem, does this patch fix it?

comment:13 Changed 8 years ago by mtredinnick

  • Cc jeff@… added

I cannot reproduce this problem at all. Staring at the code for a long time doesn't make things any clearer either. The only thing I can imagine it might be is a race condition or some other problem at import time. This might have been fixed in [5919].

If anybody can repeat the problem with latest subversion code, does the 4796-hopeful-fix.diff patch make it go away? It removes a potential race condition in i18n initialisation, although I seriously doubt that is the problem here -- it's too reliably repeatable for some people for that to be the case. If you can repeat the problem with something that isn't confidential or megabytes in size, can you attach a copy of the models that fail (as a plain text file -- gzipped files won't upload to Trac), please? Without more information or a way to repeat this, I'm a bit stuck at the moment.

Oh .. and if you can repeat it, what is the type of EasyModel.verbose_name_plural (put a print statement in contrib/databrowse/datastructures.py in EasyModel.__init__)?

(Temporarily adding Jeff Forcier to the cc list, since he had the best information last time. Feel free to remove yourself, Jeff, if you're not following this any longer.)

comment:14 Changed 8 years ago by fpierfed@…

Nope, the patch is a no go for me:

Traceback (most recent call last):
File "/home/fpierfed/lib/python2.4/site-packages/django/core/handlers/base.py" in get_response
  77. response = callback(request, *callback_args, **callback_kwargs)
File "/home/fpierfed/lib/python2.4/site-packages/django/contrib/admin/sites.py" in root
  115. return self.index(request)
File "/home/fpierfed/lib/python2.4/site-packages/django/contrib/admin/sites.py" in index
  273. app['models'].sort(lambda x, y: cmp(x['name'], y['name']))
File "/home/fpierfed/lib/python2.4/site-packages/django/contrib/admin/sites.py" in
  273. app['models'].sort(lambda x, y: cmp(x['name'], y['name']))
File "/home/fpierfed/lib/python2.4/site-packages/django/utils/functional.py" in __cmp__
  88. s = unicode(self.__func(*self.__args, **self.__kw))
File "/home/fpierfed/lib/python2.4/site-packages/django/utils/text.py" in
  8. capfirst = lambda x: x and force_unicode(x)[0].upper() + force_unicode(x)[1:]
File "/home/fpierfed/lib/python2.4/site-packages/django/utils/encoding.py" in force_unicode
  37. s = unicode(s)

  TypeError at /tips/add/
  coercing to Unicode: need string or buffer, __proxy__ found

Funny thing is that I put a

        f = file('/tmp/pippo', 'w')
        f.write(self.verbose_name_plural)
        f.write('\n')
        f.close()

at the botton of

EasyModel.__init__()

but that never gets called and the file is simply not there. Maybe I am doing something wrong here...

Anyway, here is my model

class Tag(models.Model):
    name = models.CharField(maxlength=255)
    
    def __str__(self):
        return(self.name)


class Location(models.Model):
    address = models.CharField(maxlength=255, blank=True, null=True)
    city = models.CharField(maxlength=255, blank=True, null=True)
    state = models.CharField(maxlength=255, blank=True, null=True)
    country = models.CharField(maxlength=255)
    region = models.CharField(maxlength=30)
    zip_code = models.IntegerField(blank=True, null=True)
    longitude = models.FloatField(blank=True, null=True)
    latitude = models.FloatField(blank=True, null=True)
    last_seen = models.DateTimeField(default=datetime.now())
    expired = models.BooleanField()

    def __str__(self):
        label = ''
        if(self.address):
            label = '%s, ' %(self.address)
        if(self.city):
            label += '%s ' %(self.city)
        if(self.state):
            label += '%s ' %(self.state)
        if(self.zip_code):
            label += '%s, ' %(self.zip_code)
        if(self.country and not label):
            label = self.country
        elif(self.country):
            label = label.strip() + ', %s' %(self.country)
        return(label)


class Tip(models.Model):
    title = models.CharField(maxlength=255)
    description = models.TextField()
    
    author = models.ForeignKey(User)
    date = models.DateTimeField(auto_now_add=True)
    
    rating = models.FloatField()
    expired = models.BooleanField()
    
    tags = models.ManyToManyField(Tag)
    location = models.ForeignKey(Location)

    def __str__(self):
        return(self.title)


class Vote(models.Model):
    rating = models.IntegerField()
    comment = models.TextField()
    user = models.ForeignKey(User)
    tip = models.ForeignKey(Tip)

    def __str__(self):
        return(str(self.rating))


class UserProfile(models.Model):
    user = models.ForeignKey(User, unique=True)
    picture = models.ImageField(upload_to='/tmp', blank=True, null=True)
    locations = models.ManyToManyField(Location)

comment:15 Changed 8 years ago by mtredinnick

@fpierfed: since the changes from [5919] haven't been merged into newforms-admin yet, your test isn't going to be exercising those. And the request for information from EasyModel was only relevant for those seeing the problem via the databrowse app (which was happening to like Jeff, who were using trunk).

However, this has given me the idea to try and repeat it on the newforms-admin branch. What does your admin registration code look like for these models (just paste the extra bit, I'll merge it with what you've pasted above)?

comment:16 Changed 8 years ago by mtredinnick

#4813 might be the same issue in another disguise. Or it might be totally unrelated.

comment:17 Changed 8 years ago by fpierfed@…

Here it is:

tips_admin = admin.AdminSite()
tips_admin.register(Location)
tips_admin.register(Tag)
tips_admin.register(Tip)

and then, in urls.py

(r'^add/(.*)', tips_admin.root),

comment:18 Changed 8 years ago by Jeff Forcier <jeff@…>

I am sad to report that after doing an svn up (to [5947]) the issue persists, and applying the specified diff makes no apparent difference. The error remains exactly the same as what I reported before.

comment:19 Changed 8 years ago by daniel@…

I'm having what I believe to be at least a related issue, using SVN r5993. The patch doesn't help at all, and the bt remains the same.

Here's my models:

class ExtendedUser(models.Model):
    user = models.ForeignKey(User)
    first_name = models.CharField(_("First name"), maxlength=128)
    surname = models.CharField(_("Surname"), maxlength=128)
    surname_first = models.BooleanField(_("Surname first"), default=False)
    url = models.URLField(_("URL"), blank=True, null=True)
    confirmation_string = models.CharField(_("Email validation string"),
                                           maxlength=32, blank=True, null=True)

    # The user's identifier is their name.
    def __unicode__(self):
        if self.surname_first:
            return "%s %s" % (self.surname, self.first_name)
        else:
            return "%s %s" % (self.first_name, self.surname)
    
    # Allow administration, with all the defaults.
    class Admin:
        pass

class Attendee(models.Model):
    user = models.ForeignKey(ExtendedUser)
    [...]

Running the following code from a view:

def detail(request, attendee_id):
    if not request.user.is_anonymous():
        user = ExtendedUser.objects.get(user=request.user)
    else:
        user = None
    print "%s" % user

    attendee = get_object_or_404(Attendee, id=attendee_id)
    print "%s" % attendee.user
    print "%s" % attendee

Will print 'Daniel Stone' twice (with the first and second prints), but tank with the following backtrace on the third:

Traceback (most recent call last):
File "/usr/lib/python2.5/site-packages/django/core/handlers/base.py" in get_response
  77. response = callback(request, *callback_args, **callback_kwargs)
File "/home/daniels/x/xds/site/endtroducing/../endtroducing/xds2007/attendees/views.py" in detail
  47. print "%s" % attendee
File "/usr/lib/python2.5/site-packages/django/db/models/base.py" in __str__
  91. return force_unicode(self).encode('utf-8')
File "/usr/lib/python2.5/site-packages/django/utils/encoding.py" in force_unicode
  37. s = unicode(s)

  TypeError at /xds2007/attendees/1/
  coercing to Unicode: need string or buffer, ExtendedUser found

Hope that helps.

comment:20 Changed 8 years ago by myers

  • Owner changed from nobody to myers

comment:21 Changed 8 years ago by Nis Jørgensen <nis@…>

  • Cc daniel@… added

Daniel, your example does not include the definition of the str or unicode method of Attendee. It looks like you might be having

class Attendee(models.Model):

[...]
def unicode(self):

return self.user

(this gives me a stacktrace resembling yours)

Which should be something like

class Attendee(models.Model):

[...]
def unicode(self):

return unicode(self.user)

comment:22 Changed 8 years ago by daniel@…

Yeah, your deduction is correct. I can't try it at the moment, but if that's the case, should it not be documented?

comment:23 Changed 8 years ago by myers

Daniel: self.user is a model; _ _ unicode _ _ needs to return a unicode object.

As far as Jeff's issue goes, I'm reproducing it with apache 2.2, mod_python, django_newforms-admin (latest), and python 2.5. If I sit on the admin page and click my reload button repeatedly, the admin page alternates with the "coercing to Unicode" TypeError traceback. Question is, can I fix it :)

comment:24 Changed 8 years ago by myers

  • Cc daniel@… removed
  • If I change Apache from a worker to a prefork MPM, it consistently barfs every time (instead of every other time).
  • As Jeff indicates in comment #11, adding foo = s.__unicode__() just above the s = unicode(s) statement in force_unicode magically makes things work.
  • the above patch doesn't have any effect.
  • it only appears to happen the first time force_unicode is called (per request) -- when I tried logging foo & s like so:
        if not isinstance(s, basestring,):
            if hasattr(s, '__unicode__'):
                foo = s.__unicode__()
                file('wtf.log','a').write('repr(foo): %s\n' % repr(foo))
                file('wtf.log','a').write('repr(s): %s\n\n' % repr(s))
                s = unicode(s)
            else:
                ...
    

... which resulted in:

repr(foo): <django.utils.functional.__proxy__ object at 0x86f1cec>
repr(s): <django.utils.functional.__proxy__ object at 0x86853ec>

repr(foo): u'locations'
repr(s): <django.utils.functional.__proxy__ object at 0x86853ec>

repr(foo): u'tags'
repr(s): <django.utils.functional.__proxy__ object at 0x86815cc>

...

Interestingly, the proxy object from the first __unicode__() call is different from the one we supplied. A further __unicode__() call to both the foo and s objects returns the same string, however.

I've thrown a tarball of the barebones skeleton project I'm using (based on fpierfed's model above) up here if anyone else wants to take a poke at it. This is a tough one.

comment:25 Changed 8 years ago by myers

(taking myself off this one, in case someone else feels like sprinting on it instead)

comment:26 Changed 8 years ago by myers

  • Owner myers deleted

argh! actually reassigning helps :)

comment:27 Changed 8 years ago by anonymous

  • Cc mmulley@… added

comment:28 Changed 8 years ago by William McVey <wam+djangobug@…>

  • Keywords databrowse added
  • Version set to SVN

Another datapoint for everyone. I'm having the same problem with unicode/proxy objects while using Databrowse as well. I've confirmed that all my str() methods are in fact returned as strings. The problem only happens if I leave MyModel.META.verbose_name_plural undefined. I've gone in and have defined verbose_name_plural for all my models and databrowse now works. I'm running the mainline development Django svn release 6026.

Changed 8 years ago by durdinator

Minimal testcase

comment:29 Changed 8 years ago by durdinator

  • Cc me@… added
  • Has patch set

The patch 4796-fix.diff fixes the problem for me.

However, I couldn't quite pin down what the cause of the error was. Apparently mod_python does some weird shit when handling globals() or loading modules or something like that...

comment:30 Changed 8 years ago by durdinator

Just for clarification, the minimal testcase was running against trunk, revision 6380.

comment:31 Changed 8 years ago by myers

Took care of it for newforms-admin rev. 6403 also.

Changed 8 years ago by durdinator

Patch, with regression test (fix typo)

comment:32 Changed 8 years ago by mtredinnick

  • Resolution set to fixed
  • Status changed from new to closed

(In [6446]) Fixed #4796. Fixed a problem when using i18n support for the first time -- in
particular when string_concat() was the first call made. Thanks, Andy Durdin.

comment:33 Changed 7 years ago by niosop@…

  • Resolution fixed deleted
  • Status changed from closed to reopened

I can confirm that this problem is still present in the newforms-admin branch rev 6453. Adding a try block around the offending s = unicode(s) and re-executing s = unicode(s) upon exception fixes it, in accordance with Jeffs explanation of it being "de-proxied" when accessed the first time.

I'm using apache2.2.3 and mod_proxy 3.2.10-3ub on Ubuntu.

comment:34 Changed 7 years ago by niosop@…

  • Resolution set to fixed
  • Status changed from reopened to closed

Disregard previous, still a problem in latest newforms-admin svn, but applying above patch fixes the problem.

comment:35 Changed 7 years ago by ubernostrum

The newforms-admin branch is periodically synced with trunk so that it can pick up bugfixes. The next time that happens it'll pick up the fix for this, and in the meantime it's really not worth adding additional patches or messing with the ticket status.

comment:36 Changed 3 years ago by joseph8th@…

  • Easy pickings unset
  • Severity set to Normal
  • Type set to Uncategorized
  • UI/UX unset

I had the same problem but found that I was returning the object in my unicode instead of returning unicode(self.object) in the model. Fixed it right up.

Note: See TracTickets for help on using tickets.
Back to Top