SuspiciousOperation exception is thrown if application static path contains non-ascii characters

[andkit@…]

AppStaticStorage sets its location based on __file__ of the specific application,
but when FileSystemStore.path checks the location the bytestring will be converted to unicode (by decoding from utf-8) leading to a SuspiciousOperation exception (at least on a windows-machine with a german locale, but I believe this will happen on all systems with non-utf8 filesystem encoding)

Attached patch uses the same approach as the template loaders and will decode the path using the filesystem encoding before further processing happens

[Jannis Leidel]

I can't reproduce the problem, can you provide a test case demonstrating it?

[andkit@…]

Small sample project to reproduce the problem

[andkit@…]

Replying to jezdez:

I can't reproduce the problem, can you provide a test case demonstrating it?

I have attached a sample project. The zip file should contains a directory with a german u-umlaut in latin1 encoding. Set your encoding (export LC_CTYPE=de_DE.ISO-8859-1 should do), unpack anywhere run manage.py runserver and access ​http://127.0.0.1:8000/ in your browser. (in case unziping doesnt preserve the encoding, you can create a suitable directory with os.mkdir(u"kap\xfct".encode("latin1")))

[Aymeric Augustin]

I tried the sample project but runserver didn't even start:

(django-dev)myk@mYk bug18404 % locale                                                                                ~/Downloads/kap%FCt/bug18404
LANG="fr_FR.ISO-8859-1"
LC_COLLATE="fr_FR"
LC_CTYPE="fr_FR"
LC_MESSAGES="fr_FR"
LC_MONETARY="fr_FR"
LC_NUMERIC="fr_FR"
LC_TIME="fr_FR"
LC_ALL="fr_FR"
(django-dev)myk@mYk bug18404 % python manage.py runserver --traceback                                                ~/Downloads/kap%FCt/bug18404
Traceback (most recent call last):
  File "/Users/myk/Documents/dev/django-trunk/django/core/management/base.py", line 222, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "/Users/myk/Documents/dev/django-trunk/django/core/management/base.py", line 247, in execute
    translation.activate('en-us')
  File "/Users/myk/Documents/dev/django-trunk/django/utils/translation/__init__.py", line 89, in activate
    return _trans.activate(language)
  File "/Users/myk/Documents/dev/django-trunk/django/utils/translation/trans_real.py", line 179, in activate
    _active.value = translation(language)
  File "/Users/myk/Documents/dev/django-trunk/django/utils/translation/trans_real.py", line 168, in translation
    default_translation = _fetch(settings.LANGUAGE_CODE)
  File "/Users/myk/Documents/dev/django-trunk/django/utils/translation/trans_real.py", line 151, in _fetch
    apppath = os.path.join(os.path.dirname(app.__file__), 'locale')
  File "/Users/myk/.virtualenvs/django-dev/bin/../lib/python2.7/posixpath.py", line 71, in join
    path += '/' + b
UnicodeDecodeError: 'ascii' codec can't decode byte 0xfc in position 24: ordinal not in range(128)

Apparently my terminal (under OS X) doesn't support export LC_ALL=fr_FR.ISO-8859-1; that activates the C locale instead.

[Melvyn Sopacua]

Replying to aaugustin:

Apparently my terminal (under OS X) doesn't support export LC_ALL=fr_FR.ISO-8859-1; that activates the C locale instead.

That's cause on FreeBSD derived systems the locale would be fr_FR.ISO8859-1.

The root cause is that u umlaut in latin-1 conflicts with UTF-8 encoding, where the value is used as part of the variable size encoding scheme:

>>> u = unicode('\xfc', 'latin1')
>>> u.encode('utf-8')
'\xc3\xbc'

[Łukasz Rekucki]

Unfortunately, the abspathu used inside FileSystemStorage.__init doesn't return unicode if not given unicode:

>>> import sys
>>> sys.path
['', 'C:\\Temp\\g\xb9ska', ...snip... ]
>>> import foo
>>> foo.__file__
'C:\\Temp\\g\xb9ska\\foo\\__init__.py'
>>> from django.core.files.storage import *
>>> st = FileSystemStorage(os.path.join(os.path.dirname(foo.__file__), "static"), base_url=u"/foo")
>>> st.path("bar")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "django\core\files\storage.py", line 259, in path
    raise SuspiciousOperation("Attempted access to '%s' denied." % name)
django.core.exceptions.SuspiciousOperation: Attempted access to 'bar' denied.

So this is definitly a bug. Most likely FileSystemStorage should accept only unicode locations and AppStaticStorage should pass a unicode path as done in the patch. It would also be good is path didn't confuse UnicodeDecodeError with ValueError raised by safe_join. Maybe safe_join could raise a more meaningful subclass of ValueError instead?

Note that on master (1.5) this will fail earlier (i.e. on every os.path.join), because of unicode_literals switch.

[fhahn]

I've added a test for the patch and created a pull request: ​https://github.com/django/django/pull/814

I put to decoding in FileSystemStorage.init, because AppStaticStorage is only a subclass of FileSystemStorage.

[fhahn]

I had another look and reworked my patch. I've added a test for AppStaticStorage with non ascii characters in the module path.

I think this bug was fixed by the changes introduced by #19357.

upath is used to convert the file name to an unicode string: ​https://github.com/fhahn/django/blob/ticket_18404/django/contrib/staticfiles/storage.py#L300

[Tim Graham <timograham@…>]

In ca39c0a6becf497880181b3a5ef6db84130f9723:

Fixed #18404 -- Added test for AppStaticStorage with non ascii path

(bug was already fixed in #19357)