force_unicode error

[fpierfed@…]

Hi,

I am using the newforms-admin branch from SVN (revision 5632) under apache 2.0.54, flup SVN (revision 2354) and Python 2.4.1. I have a webapp that uses custom admin sites via myadmin = admin.AdminSite() and myadmin.register(...)

If I access the admin site via the django internal development server, everything works. If I use fcgi the admin pages work only if I am an anonymous user (in that case I am redirected to the login screen, then to the admin pages and all is good). If I use fcgi and I am already logged in, accessing the admin pages gives this error:

TypeError at /tips/admin/
coercing to Unicode: need string or buffer, __proxy__ found
Request Method: 	GET
Request URL: 	http://www.peertraveller.com/tips/admin/
Exception Type: 	TypeError
Exception Value: 	coercing to Unicode: need string or buffer, __proxy__ found
Exception Location: 	/home/fpierfed/lib/python2.4/site-packages/django/utils/encoding.py in force_unicode, line 38
Python Executable: 	/home/fpierfed/bin/python
Python Version: 	2.4.1

[Malcolm Tredinnick]

Fixed description.

[Malcolm Tredinnick]

We'll need the full traceback to be able to debug this -- you've only provided the last line. Look a bit further down the debug screen and find the link that says "cut-and-paste traceback". Click on that and the paste in the output (remember to wrap it in {{{ and }}} markup so that it's formatted correctly in Trac's wiki syntax.

[fpierfed@…]

Here is the full traceback:

Traceback (most recent call last):
File "/home/fpierfed/lib/python2.4/site-packages/django/core/handlers/base.py" in get_response
  77. response = callback(request, *callback_args, **callback_kwargs)
File "/home/fpierfed/lib/python2.4/site-packages/django/contrib/admin/sites.py" in root
  115. return self.index(request)
File "/home/fpierfed/lib/python2.4/site-packages/django/contrib/admin/sites.py" in index
  273. app['models'].sort(lambda x, y: cmp(x['name'], y['name']))
File "/home/fpierfed/lib/python2.4/site-packages/django/contrib/admin/sites.py" in
  273. app['models'].sort(lambda x, y: cmp(x['name'], y['name']))
File "/home/fpierfed/lib/python2.4/site-packages/django/utils/functional.py" in __cmp__
  86. s = unicode(self.__func(*self.__args, **self.__kw))
File "/home/fpierfed/lib/python2.4/site-packages/django/utils/text.py" in
  7. capfirst = lambda x: x and force_unicode(x)[0].upper() + force_unicode(x)[1:]
File "/home/fpierfed/lib/python2.4/site-packages/django/utils/encoding.py" in force_unicode
  38. s = unicode(s)

  TypeError at /tips/add/
  coercing to Unicode: need string or buffer, __proxy__ found

[Malcolm Tredinnick]

So this traceback means the __unicode__ method on a model is returning a lazy translation object, which is illegal Python code (__unicode__ must return a Unicode object).

It's not possible to tell from the traceback which model is involved, although as far as I can see, no models in Django's source are doing this.

Have a look at your debug traceback screen again and examine the local variables in the second last line (the "capfirst" line). That should tell you which object is involved. Or, if that doesn't, have a look at the local variables in one of the previous lines (the ones from sites.py).

If the object involved is in Django's code, please let us know which one.

[Simon G. <dev@…>]

Resolving as invalid as it doesn't appear to be Django-specific. Please reopen if I'm wrong!

[Jeff Forcier <jeff@…>]

I'm getting this too, as of SVN changeset 5677, with databrowse. Fairly stock Apache2 + mod_python setup, Python 2.5, Linux.

In my situation, I'm just trying to use databrowse on a brand-new app with three simple model classes, and get a similar error. The offending lazy object is one of my Model objects' nonexistent Meta.verbose_name_plural - in other words, the databrowse template is trying to access MyModel._meta.verbose_name_plural (which is not defined), and at least in the Python shell, that attribute results in a lazy object. My other two classes in the same file do define verbose_name_plural, and accessing the attribute on them results in regular strings.

I've toyed around in the shell and read over the debug page and can't figure out exactly what's going wrong. It appears that in normal usage, calling force_unicode or unicode on a lazy object works as you'd expect (i.e. it takes the output of the stored function and makes it into a Unicode string); even when I del a lazy object's __unicode__ attribute/method, unicode still works by returns a string representation of the object, i.e. u'<django.utils.functional.__proxy__ object at 0x8606a2c>'.

[Jeff Forcier <jeff@…>]

Oh, I should probably note that I am on regular trunk and not newforms-admin.

[Jeff Forcier <jeff@…>]

FFS. Didn't mean to save the changes I'd made to the dropdowns.

[Jeff Forcier <jeff@…>]

Final comment for the night: the issue does not occur under the development runserver, and it is also avoided by explicitly defining verbose_name_plural on the third object in my model. Haven't been able to figure out the ramifications of either.

[fpierfed@…]

I cannot find anything wrong in the traceback page: all string variables are unicode objects as one would expect.

[Jeff Forcier <reg@…>]

Yes, that's the rub. The traceback page's printing of s causes it to happily show up as u'jobs'. However, during my investigation, I at one point had a line just prior to the s = unicode(s), to the effect of foo = s.__unicode__() and that made the process work - the __proxy__ related error did not appear and databrowse worked fine.

So in this case what I think is happening is the s = unicode(s) line is erroring out, but at the same time it's "unblocking" the lazy aspect of s, so that when it is asked to yield itself as a (Unicode?) string in the debug printout, it does so.

[Chris Beaven]

I'm promoting to accepted - it does look like a valid bug. Also noted in #5146, which I closed as a dupe of this.

[Malcolm Tredinnick]

If somebody can reproduce the problem, does this patch fix it?

[Malcolm Tredinnick]

I cannot reproduce this problem at all. Staring at the code for a long time doesn't make things any clearer either. The only thing I can imagine it might be is a race condition or some other problem at import time. This might have been fixed in [5919].

If anybody can repeat the problem with latest subversion code, does the 4796-hopeful-fix.diff patch make it go away? It removes a potential race condition in i18n initialisation, although I seriously doubt that is the problem here -- it's too reliably repeatable for some people for that to be the case. If you can repeat the problem with something that isn't confidential or megabytes in size, can you attach a copy of the models that fail (as a plain text file -- gzipped files won't upload to Trac), please? Without more information or a way to repeat this, I'm a bit stuck at the moment.

Oh .. and if you can repeat it, what is the type of EasyModel.verbose_name_plural (put a print statement in contrib/databrowse/datastructures.py in EasyModel.__init__)?

(Temporarily adding Jeff Forcier to the cc list, since he had the best information last time. Feel free to remove yourself, Jeff, if you're not following this any longer.)

[fpierfed@…]

Nope, the patch is a no go for me:

Traceback (most recent call last):
File "/home/fpierfed/lib/python2.4/site-packages/django/core/handlers/base.py" in get_response
  77. response = callback(request, *callback_args, **callback_kwargs)
File "/home/fpierfed/lib/python2.4/site-packages/django/contrib/admin/sites.py" in root
  115. return self.index(request)
File "/home/fpierfed/lib/python2.4/site-packages/django/contrib/admin/sites.py" in index
  273. app['models'].sort(lambda x, y: cmp(x['name'], y['name']))
File "/home/fpierfed/lib/python2.4/site-packages/django/contrib/admin/sites.py" in
  273. app['models'].sort(lambda x, y: cmp(x['name'], y['name']))
File "/home/fpierfed/lib/python2.4/site-packages/django/utils/functional.py" in __cmp__
  88. s = unicode(self.__func(*self.__args, **self.__kw))
File "/home/fpierfed/lib/python2.4/site-packages/django/utils/text.py" in
  8. capfirst = lambda x: x and force_unicode(x)[0].upper() + force_unicode(x)[1:]
File "/home/fpierfed/lib/python2.4/site-packages/django/utils/encoding.py" in force_unicode
  37. s = unicode(s)

  TypeError at /tips/add/
  coercing to Unicode: need string or buffer, __proxy__ found

Funny thing is that I put a

        f = file('/tmp/pippo', 'w')
        f.write(self.verbose_name_plural)
        f.write('\n')
        f.close()

at the botton of

EasyModel.__init__()

but that never gets called and the file is simply not there. Maybe I am doing something wrong here...

Anyway, here is my model

class Tag(models.Model):
    name = models.CharField(maxlength=255)
    
    def __str__(self):
        return(self.name)


class Location(models.Model):
    address = models.CharField(maxlength=255, blank=True, null=True)
    city = models.CharField(maxlength=255, blank=True, null=True)
    state = models.CharField(maxlength=255, blank=True, null=True)
    country = models.CharField(maxlength=255)
    region = models.CharField(maxlength=30)
    zip_code = models.IntegerField(blank=True, null=True)
    longitude = models.FloatField(blank=True, null=True)
    latitude = models.FloatField(blank=True, null=True)
    last_seen = models.DateTimeField(default=datetime.now())
    expired = models.BooleanField()

    def __str__(self):
        label = ''
        if(self.address):
            label = '%s, ' %(self.address)
        if(self.city):
            label += '%s ' %(self.city)
        if(self.state):
            label += '%s ' %(self.state)
        if(self.zip_code):
            label += '%s, ' %(self.zip_code)
        if(self.country and not label):
            label = self.country
        elif(self.country):
            label = label.strip() + ', %s' %(self.country)
        return(label)


class Tip(models.Model):
    title = models.CharField(maxlength=255)
    description = models.TextField()
    
    author = models.ForeignKey(User)
    date = models.DateTimeField(auto_now_add=True)
    
    rating = models.FloatField()
    expired = models.BooleanField()
    
    tags = models.ManyToManyField(Tag)
    location = models.ForeignKey(Location)

    def __str__(self):
        return(self.title)


class Vote(models.Model):
    rating = models.IntegerField()
    comment = models.TextField()
    user = models.ForeignKey(User)
    tip = models.ForeignKey(Tip)

    def __str__(self):
        return(str(self.rating))


class UserProfile(models.Model):
    user = models.ForeignKey(User, unique=True)
    picture = models.ImageField(upload_to='/tmp', blank=True, null=True)
    locations = models.ManyToManyField(Location)

[Malcolm Tredinnick]

@fpierfed: since the changes from [5919] haven't been merged into newforms-admin yet, your test isn't going to be exercising those. And the request for information from EasyModel was only relevant for those seeing the problem via the databrowse app (which was happening to like Jeff, who were using trunk).

However, this has given me the idea to try and repeat it on the newforms-admin branch. What does your admin registration code look like for these models (just paste the extra bit, I'll merge it with what you've pasted above)?

[Malcolm Tredinnick]

#4813 might be the same issue in another disguise. Or it might be totally unrelated.

[fpierfed@…]

Here it is:

tips_admin = admin.AdminSite()
tips_admin.register(Location)
tips_admin.register(Tag)
tips_admin.register(Tip)

and then, in urls.py

(r'^add/(.*)', tips_admin.root),

[Jeff Forcier <jeff@…>]

I am sad to report that after doing an svn up (to [5947]) the issue persists, and applying the specified diff makes no apparent difference. The error remains exactly the same as what I reported before.

[daniel@…]

I'm having what I believe to be at least a related issue, using SVN r5993. The patch doesn't help at all, and the bt remains the same.

Here's my models:

class ExtendedUser(models.Model):
    user = models.ForeignKey(User)
    first_name = models.CharField(_("First name"), maxlength=128)
    surname = models.CharField(_("Surname"), maxlength=128)
    surname_first = models.BooleanField(_("Surname first"), default=False)
    url = models.URLField(_("URL"), blank=True, null=True)
    confirmation_string = models.CharField(_("Email validation string"),
                                           maxlength=32, blank=True, null=True)

    # The user's identifier is their name.
    def __unicode__(self):
        if self.surname_first:
            return "%s %s" % (self.surname, self.first_name)
        else:
            return "%s %s" % (self.first_name, self.surname)
    
    # Allow administration, with all the defaults.
    class Admin:
        pass

class Attendee(models.Model):
    user = models.ForeignKey(ExtendedUser)
    [...]

Running the following code from a view:

def detail(request, attendee_id):
    if not request.user.is_anonymous():
        user = ExtendedUser.objects.get(user=request.user)
    else:
        user = None
    print "%s" % user

    attendee = get_object_or_404(Attendee, id=attendee_id)
    print "%s" % attendee.user
    print "%s" % attendee

Will print 'Daniel Stone' twice (with the first and second prints), but tank with the following backtrace on the third:

Traceback (most recent call last):
File "/usr/lib/python2.5/site-packages/django/core/handlers/base.py" in get_response
  77. response = callback(request, *callback_args, **callback_kwargs)
File "/home/daniels/x/xds/site/endtroducing/../endtroducing/xds2007/attendees/views.py" in detail
  47. print "%s" % attendee
File "/usr/lib/python2.5/site-packages/django/db/models/base.py" in __str__
  91. return force_unicode(self).encode('utf-8')
File "/usr/lib/python2.5/site-packages/django/utils/encoding.py" in force_unicode
  37. s = unicode(s)

  TypeError at /xds2007/attendees/1/
  coercing to Unicode: need string or buffer, ExtendedUser found

Hope that helps.

[Nis Jørgensen <nis@…>]

Daniel, your example does not include the definition of the str or unicode method of Attendee. It looks like you might be having

class Attendee(models.Model):

[...]
def unicode(self):

return self.user

(this gives me a stacktrace resembling yours)

Which should be something like

class Attendee(models.Model):

[...]
def unicode(self):

return unicode(self.user)

[daniel@…]

Yeah, your deduction is correct. I can't try it at the moment, but if that's the case, should it not be documented?

[Myers Carpenter]

Daniel: self.user is a model; _ _ unicode _ _ needs to return a unicode object.

As far as Jeff's issue goes, I'm reproducing it with apache 2.2, mod_python, django_newforms-admin (latest), and python 2.5. If I sit on the admin page and click my reload button repeatedly, the admin page alternates with the "coercing to Unicode" TypeError traceback. Question is, can I fix it :)

[Myers Carpenter]

• If I change Apache from a worker to a prefork MPM, it consistently barfs every time (instead of every other time).

• As Jeff indicates in comment #11, adding foo = s.__unicode__() just above the s = unicode(s) statement in force_unicode magically makes things work.

• the above patch doesn't have any effect.

• it only appears to happen the first time force_unicode is called (per request) -- when I tried logging foo & s like so:

      if not isinstance(s, basestring,):
          if hasattr(s, '__unicode__'):
              foo = s.__unicode__()
              file('wtf.log','a').write('repr(foo): %s\n' % repr(foo))
              file('wtf.log','a').write('repr(s): %s\n\n' % repr(s))
              s = unicode(s)
          else:
              ...
  

... which resulted in:

repr(foo): <django.utils.functional.__proxy__ object at 0x86f1cec>
repr(s): <django.utils.functional.__proxy__ object at 0x86853ec>

repr(foo): u'locations'
repr(s): <django.utils.functional.__proxy__ object at 0x86853ec>

repr(foo): u'tags'
repr(s): <django.utils.functional.__proxy__ object at 0x86815cc>

...

Interestingly, the proxy object from the first __unicode__() call is different from the one we supplied. A further __unicode__() call to both the foo and s objects returns the same string, however.

I've thrown a tarball of the barebones skeleton project I'm using (based on fpierfed's model above) up ​here if anyone else wants to take a poke at it. This is a tough one.

[Myers Carpenter]

(taking myself off this one, in case someone else feels like sprinting on it instead)

[Myers Carpenter]

argh! actually reassigning helps :)

[William McVey <wam+djangobug@…>]

Another datapoint for everyone. I'm having the same problem with unicode/proxy objects while using Databrowse as well. I've confirmed that all my str() methods are in fact returned as strings. The problem only happens if I leave MyModel.META.verbose_name_plural undefined. I've gone in and have defined verbose_name_plural for all my models and databrowse now works. I'm running the mainline development Django svn release 6026.

[durdinator]

Minimal testcase

[durdinator]

The patch 4796-fix.diff fixes the problem for me.

However, I couldn't quite pin down what the cause of the error was. Apparently mod_python does some weird shit when handling globals() or loading modules or something like that...

[durdinator]

Just for clarification, the minimal testcase was running against trunk, revision 6380.

[Myers Carpenter]

Took care of it for newforms-admin rev. 6403 also.

[durdinator]

Patch, with regression test (fix typo)

[Malcolm Tredinnick]

(In [6446]) Fixed #4796. Fixed a problem when using i18n support for the first time -- in
particular when string_concat() was the first call made. Thanks, Andy Durdin.

[niosop@…]

I can confirm that this problem is still present in the newforms-admin branch rev 6453. Adding a try block around the offending s = unicode(s) and re-executing s = unicode(s) upon exception fixes it, in accordance with Jeffs explanation of it being "de-proxied" when accessed the first time.

I'm using apache2.2.3 and mod_proxy 3.2.10-3ub on Ubuntu.

[niosop@…]

Disregard previous, still a problem in latest newforms-admin svn, but applying above patch fixes the problem.

[James Bennett]

The newforms-admin branch is periodically synced with trunk so that it can pick up bugfixes. The next time that happens it'll pick up the fix for this, and in the meantime it's really not worth adding additional patches or messing with the ticket status.

[joseph8th@…]

I had the same problem but found that I was returning the object in my unicode instead of returning unicode(self.object) in the model. Fixed it right up.