UnicodeEncodeError in template path after migration from Django 0.96 to 1.0

[anonymous]

My templates are in a dir with a non-ascii char. Something like: u"/foo/bar/paraíso/templates".

With Django-0.96 works fine. With Django-1.0 internal testing server (./manage.py runserver) works fine too. But with apache and mod_python throws an UnicodeEncodeError ('ascii' codec can't encode character u'\xed' in position 13: ordinal not in range(128)) in /var/lib/python-support/python2.5/django/template/loaders/filesystem.py, in load_template_source:

return (open(filepath).read().decode(settings.FILE_CHARSET), filepath)

settings.FILE_CHARSET is "utf-8". filepath is something like u"/foo/bar/paraíso/templates/foobar.html". It comes from TEMPLATE_DIRS in settings.py, that it looks like:

TEMPLATE_DIRS = (unicode(os.path.join(os.path.dirname(__file__), 'templates').replace('\\','/'), "utf-8"), )

If I put template dir without unicode conversion both runserver and mod_python+apache gives a "cannot extend foobar.html because template doesn't exist" error.

[Daniel Pope <dan@…>]

settings.FILE_CHARSET is irrelevant: that's the encoding for file data, not paths.

Under Linux paths are effectively stored as binary. Though characters like : and / are reserved, everything else is stored as-is, which means that when you have non-ASCII characters in paths they are just stored in the character set defined by the application's locale. A locale mismatch between the Apache process and your shell could cause this.

For the purposes of this ticket, could please type out the paths you want Django to use in full and let us know the results of writing the setting as

TEMPLATE_DIRS = [u"/foo/bar/para\xedso/templates/"]

versus

import locale
TEMPLATE_DIRS = [u"/foo/bar/para\xedso/templates/".encode(locale.getpreferredencoding())]

?

[anonymous]

With:

TEMPLATE_DIRS = [u"/home/test1/Q-DJANGO1/Para\xedso/fp/web/fpqweb/templates"]

Works in Django test server, but not in apache+mod_python:

UnicodeEncodeError at /

'ascii' codec can't encode character u'\xed' in position 26: ordinal not in range(128)

Request Method: 	GET
Request URL: 	http://localhost/
Exception Type: 	UnicodeEncodeError
Exception Value: 	

'ascii' codec can't encode character u'\xed' in position 26: ordinal not in range(128)

Exception Location: 	/var/lib/python-support/python2.5/django/template/loaders/filesystem.py in load_template_source, line 23
Python Executable: 	/usr/bin/python
Python Version: 	2.5.2
Python Path: 	['home/test1/Q-DJANGO1/Para\xc3\xadso/fp/web', '/usr/lib/python2.5', '/usr/lib/python2.5/plat-linux2', '/usr/lib/python2.5/lib-tk', '/usr/lib/python2.5/lib-dynload', '/usr/local/lib/python2.5/site-packages', '/usr/lib/python2.5/site-packages', '/usr/lib/python2.5/site-packages/Numeric', '/usr/lib/python2.5/site-packages/PIL', '/usr/lib/python2.5/site-packages/gst-0.10', '/var/lib/python-support/python2.5', '/usr/lib/python2.5/site-packages/gtk-2.0', '/var/lib/python-support/python2.5/gtk-2.0', '/usr/lib/python2.5/site-packages/wx-2.8-gtk2-unicode']
Server time: 	lun, 8 Sep 2008 17:05:22 +0200
Unicode error hint

The string that could not be encoded/decoded was: /Paraíso/fp

With:

import locale
TEMPLATE_DIRS = [u"/home/test1/Q-DJANGO1/Para\xedso/fp/web/fpqweb/templates".encode(locale.getpreferredencoding())]

In Django test server:

TemplateSyntaxError at /

Template u'base.html' cannot be extended, because it doesn't exist

Request Method: 	GET
Request URL: 	http://localhost:8000/
Exception Type: 	TemplateSyntaxError
Exception Value: 	

Template u'base.html' cannot be extended, because it doesn't exist

Exception Location: 	/var/lib/python-support/python2.5/django/template/loader_tags.py in get_parent, line 66
Python Executable: 	/usr/bin/python
Python Version: 	2.5.2
[...]
Template error

In template /home/test1/Q-DJANGO1/Paraíso/fp/web/fpqweb/templates/index.html, error at line 1
Template u'base.html' cannot be extended, because it doesn't exist
1 	{% extends "base.html" %}
2       
3 	{% block 
[...]

And in Apache2:

MOD_PYTHON ERROR

[...]
URI:            '/'
Location:       '/'
Directory:      None
Filename:       '/var/www/'
PathInfo:       ''

Phase:          'PythonHandler'
Handler:        'django.core.handlers.modpython'

Traceback (most recent call last):

  File "/usr/lib/python2.5/site-packages/mod_python/importer.py", line 1537, in HandlerDispatch
    default=default_handler, arg=req, silent=hlist.silent)

  File "/usr/lib/python2.5/site-packages/mod_python/importer.py", line 1229, in _process_target
    result = _execute_target(config, req, object, arg)

  File "/usr/lib/python2.5/site-packages/mod_python/importer.py", line 1128, in _execute_target
    result = object(arg)

  File "/var/lib/python-support/python2.5/django/core/handlers/modpython.py", line 222, in handler
    return ModPythonHandler()(req)

  File "/var/lib/python-support/python2.5/django/core/handlers/modpython.py", line 185, in __call__
    self.load_middleware()

  File "/var/lib/python-support/python2.5/django/core/handlers/base.py", line 31, in load_middleware
    for middleware_path in settings.MIDDLEWARE_CLASSES:

  File "/var/lib/python-support/python2.5/django/conf/__init__.py", line 28, in __getattr__
    self._import_settings()

  File "/var/lib/python-support/python2.5/django/conf/__init__.py", line 59, in _import_settings
    self._target = Settings(settings_module)

  File "/var/lib/python-support/python2.5/django/conf/__init__.py", line 92, in __init__
    mod = __import__(self.SETTINGS_MODULE, {}, {}, [''])

  File "/home/test1/Q-DJANGO1/Paraíso/fp/web/fpqweb/settings.py", line 103, in <module>
    TEMPLATE_DIRS = [u"/home/test1/Q-DJANGO1/Para\xedso/fp/web/fpqweb/templates".encode(locale.getpreferredencoding())]

UnicodeEncodeError: 'ascii' codec can't encode character u'\xed' in position 26: ordinal not in range(128)

[Daniel Pope <dan@…>]

Apache is running in the wrong locale: locale.getpreferredencoding() is returning an ASCII character set.

I've had a look at my servers and both Debian and Ubuntu include the line:

ENV="env -i LANG=C PATH=/usr/local/bin:/usr/bin:/bin"

as one of the first lines in /etc/init.d/apache2.

Replace LANG=C with LC_ALL=*your locale* and restart Apache, and mod_python should then be able to open Unicode paths.

The

Template u'base.html' cannot be extended, because it doesn't exist

error is subtle, but simply put, you cannot specify non-ASCII bytestrings in TEMPLATE_DIRS.

Full explanation: {% extends "base.html" %} is now interpreted as a unicode template filename u'base.html' which causes a UnicodeDecodeError when os.path.joined with the template dir as the loader searches for the template. Unfortunately it ​gets swallowed because UnicodeDecodeError is a subclass of ValueError, and ValueError is also raised when safe_join detects that the join is not safe.

The errors from safe_join should not be silently ignored. Suppressing them makes it hard to debug. There could also be a better error message when searching TEMPLATE_DIRS raises UnicodeDecodeError.

[Daniel Pope <dan@…>]

Patch to allow errors from safe_join to propagate

[anonymous]

Thanks a lot. Certainly that was the problem. Not a Django bug after all. You can close this ticket if you want to.

[Malcolm Tredinnick]

If debugging one problem reveals a different bug, please open a new ticket next time. The patch attached here doesn't have anything to do with the original problem report (which wasn't a Django bug), but it was overlooked because it wasn't in an appropriately named ticket.

The patch is appreciated, but it would be nice if we knew it existed. So new ticket for a new issue, please.

[Malcolm Tredinnick]

This patch turned out not to work as intended (it didn't pass the existing tests and failed some new ones I wrote as well). I'm going to add some different error reporting in a commit shortly.

[Malcolm Tredinnick]

(In [9161]) Added some better error reporting and path handling when creating template paths.

We now raise UnicodeDecodeError for non-UTF-8 bytestrings (thanks to Daniel
Pope for diagnosing this was being swallowed by ValueError) and allow UTF-8
bytestrings as template directories.

Refs #8965.

[Malcolm Tredinnick]

(In [9162]) [1.0.X] Added some better error reporting and path handling when creating template paths.

We now raise UnicodeDecodeError for non-UTF-8 bytestrings (thanks to Daniel
Pope for diagnosing this was being swallowed by ValueError) and allow UTF-8
bytestrings as template directories. (The last bit is arguably a feature-add,
but we allow UTF-8 bytestrings everywhere else, so I'm counting it as a bugfix.)

Refs #8965.

Backport of r9161 from trunk.