Email subjects not encoded properly

Ugly patch that fixes the problem for me

Yes. Good catch.

I'll have to check the behaviour of Header. This might need some tweaking from memory. The point is that when I was writing the current code, there were times when headers were being pointlessly encoded even when they could be represented directly (particularly ASCII text). So, providing it doesn't try to wrap ASCII up in anything fancy, this is the right fix. Otherwise, we need to check that the data really is non-ASCII before making a Header() out of it.

You're right, it does some pointless encoding when the string is pure ASCII, for example:

From: =?utf-8?q?Trivialibre?= <trivialibre@enix.org>

In that case, the =?utf-8?q? stuff is useless.

So, instead of verifying if the string is Unicode (with force_unicode), the code should probably check if the string is ASCII 7bits only or not. Do you want me to provide an improved fix, or are you going to do it ?

I was thinking about this some more and I'm not sure I completely understand the problem any longer. force_unicode() forces the input to a unicode object and str() will raise an error for any data that isn't ASCII.

So, for example

>>> str(force_unicode('\xc3\x85ngstr\xc3\xb6m'))    # A UTF-8 bytestring
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeEncodeError: 'ascii' codec can't encode character u'\xc5' in position 0: ordinal not in range(128)

The only way I could see this failing is if somebody had changed Python's default encoding, which is well advertised as being something that shouldn't be done, for exactly this sort of reason.

What is an example of a header string that is causing the problem? And what does sys.getdefaultencoding() return?

From a raw Python shell, with PYTHONPATH=/path/to/django:

thomas@toulibre:/srv/www/trivialibre.humanoidz.org$ python
Python 2.4.4 (#2, Apr  5 2007, 20:11:18) 
[GCC 4.1.2 20061115 (prerelease) (Debian 4.1.1-21)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> print sys.getdefaultencoding()
ascii
>>> from django.utils.encoding import force_unicode
>>> str(force_unicode('\xc3\x85ngstr\xc3\xb6m'))
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
UnicodeEncodeError: 'ascii' codec can't encode character u'\xc5' in position 0: ordinal not in range(128)
>>> str(force_unicode('Nouvelle question "Et ça marche bien é ?"'))
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe7' in position 22: ordinal not in range(128)

So here, it works properly. Now, from a Python shell ran using "manage.py shell", still with PYTHONPATH=/path/to/django/:

thomas@toulibre:/srv/www/trivialibre.humanoidz.org$ ./trivialibre/tvl/manage.py shell
Python 2.4.4 (#2, Apr  5 2007, 20:11:18) 
[GCC 4.1.2 20061115 (prerelease) (Debian 4.1.1-21)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> import sys
>>> print sys.getdefaultencoding()
utf-8
>>> from django.utils.encoding import force_unicode
>>> str(force_unicode('\xc3\x85ngstr\xc3\xb6m'))
'\xc3\x85ngstr\xc3\xb6m'
>>> str(force_unicode('Nouvelle question "Et ça marche bien é ?"'))
'Nouvelle question "Et \xc3\xa7a marche bien \xc3\xa9 ?"'
>>> 

The second string tested above is the one I was using for my tests. But yours also perfectly shows the problem.

Oh, that's tricky and not very nice behaviour. :-(

Okay, now I'm convinced. Will fix it in a minute.

(In [6551]) Fixed #5778 -- Changed the way we detect if a string is non-ASCII when creating email headers. This fixes a problem that was showing up on some (but not all) systems.

Tested, works perfectly. Thanks!