StaticFileStorage Encoding error on '\xf4'

[Klemen Sever]

While using the collectstatic command with django-pipeline and a i18n js library (containing lots of foreign characters), I stumbled on an encoding error:

Traceback (most recent call last):
  File "./dev_manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/Users/achedeuzot/Documents/Projects/project/www.project.net/lib/python3.4/site-packages/django/core/management/__init__.py", line 385, in execute_from_command_line
    utility.execute()
  File "/Users/achedeuzot/Documents/Projects/project/www.project.net/lib/python3.4/site-packages/django/core/management/__init__.py", line 377, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/Users/achedeuzot/Documents/Projects/project/www.project.net/lib/python3.4/site-packages/django/core/management/base.py", line 288, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "/Users/achedeuzot/Documents/Projects/project/www.project.net/lib/python3.4/site-packages/django/core/management/base.py", line 337, in execute
    output = self.handle(*args, **options)
  File "/Users/achedeuzot/Documents/Projects/project/www.project.net/lib/python3.4/site-packages/django/core/management/base.py", line 532, in handle
    return self.handle_noargs(**options)
  File "/Users/achedeuzot/Documents/Projects/project/www.project.net/lib/python3.4/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 168, in handle_noargs
    collected = self.collect()
  File "/Users/achedeuzot/Documents/Projects/project/www.project.net/lib/python3.4/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 114, in collect
    for original_path, processed_path, processed in processor:
  File "/Users/achedeuzot/Documents/Projects/project/www.project.net/lib/python3.4/site-packages/pipeline/storage.py", line 32, in post_process
    packager.pack_stylesheets(package)
  File "/Users/achedeuzot/Documents/Projects/project/www.project.net/lib/python3.4/site-packages/pipeline/packager.py", line 94, in pack_stylesheets
    variant=package.variant, **kwargs)
  File "/Users/achedeuzot/Documents/Projects/project/www.project.net/lib/python3.4/site-packages/pipeline/packager.py", line 105, in pack
    self.save_file(output_filename, content)
  File "/Users/achedeuzot/Documents/Projects/project/www.project.net/lib/python3.4/site-packages/pipeline/packager.py", line 117, in save_file
    return self.storage.save(path, ContentFile("\xf4"))
  File "/Users/achedeuzot/Documents/Projects/project/www.project.net/lib/python3.4/site-packages/django/core/files/storage.py", line 51, in save
    name = self._save(name, content)
  File "/Users/achedeuzot/Documents/Projects/project/www.project.net/lib/python3.4/site-packages/django/core/files/storage.py", line 228, in _save
    _file.write(chunk)
UnicodeEncodeError: 'ascii' codec can't encode character '\xf4' in position 0: ordinal not in range(128)

After some enquiries, the same error could be reproduced using the following lines:

    from django.contrib.staticfiles.storage import StaticFilesStorage
    from django.core.files.base import ContentFile

    storage = StaticFilesStorage()
    storage.save("test-output.txt", ContentFile("ô"))

I was able to "solve" the issue by using bytearray:

    storage.save("test-output.txt", ContentFile(bytearray("ô", 'utf-8')))

Also, another solution is/was to add LANG=en_US.utf-8 into the environment.

The line that trows the error is the following:

    mode = 'wb' if isinstance(chunk, bytes) else 'wt'
    _file = os.fdopen(fd, mode)
_file.write(chunk) # <= this one

We can see that if I send bytes, the write() happens without errors while if I send a string, the write() fails.

How come Django fails on a "standard" utf-8 string ? Is this a Django issue or from something else (OS write(), ...)?

[Areski Belaid]

I cannot reproduce this bug, I tested the following with Python2.7 and Python3.4:

import os, django
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "myproject.settings")
django.setup()

from django.contrib.staticfiles.storage import StaticFilesStorage
from django.core.files.base import ContentFile

storage = StaticFilesStorage()
storage.save("test-output.txt", ContentFile("ô"))

I work on Ubuntu14 and by default LANG is set, I tried to unset it but no luck.

[Klemen Sever]

Additional info:

• MacOSX 10.9.4
• Python 3.4.1
• Env: no LANG nor LC_ALL nor LC_CTYPE (set but empty)
• Django 1.7r2

Thanks for your update @areski !

[Tim Graham]

I think it's a problem with your configuration, see ​this FAQ entry.