id	summary	reporter	owner	description	type	status	component	version	severity	resolution	keywords	cc	stage	has_patch	needs_docs	needs_tests	needs_better_patch	easy	ui_ux
23317	StaticFileStorage Encoding error on '\xf4'	Klemen Sever	nobody	"While using the `collectstatic` command with `django-pipeline` and a i18n js library (containing lots of foreign characters), I stumbled on an encoding error:

{{{
Traceback (most recent call last):
  File ""./dev_manage.py"", line 10, in <module>
    execute_from_command_line(sys.argv)
  File ""/Users/achedeuzot/Documents/Projects/project/www.project.net/lib/python3.4/site-packages/django/core/management/__init__.py"", line 385, in execute_from_command_line
    utility.execute()
  File ""/Users/achedeuzot/Documents/Projects/project/www.project.net/lib/python3.4/site-packages/django/core/management/__init__.py"", line 377, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File ""/Users/achedeuzot/Documents/Projects/project/www.project.net/lib/python3.4/site-packages/django/core/management/base.py"", line 288, in run_from_argv
    self.execute(*args, **options.__dict__)
  File ""/Users/achedeuzot/Documents/Projects/project/www.project.net/lib/python3.4/site-packages/django/core/management/base.py"", line 337, in execute
    output = self.handle(*args, **options)
  File ""/Users/achedeuzot/Documents/Projects/project/www.project.net/lib/python3.4/site-packages/django/core/management/base.py"", line 532, in handle
    return self.handle_noargs(**options)
  File ""/Users/achedeuzot/Documents/Projects/project/www.project.net/lib/python3.4/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py"", line 168, in handle_noargs
    collected = self.collect()
  File ""/Users/achedeuzot/Documents/Projects/project/www.project.net/lib/python3.4/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py"", line 114, in collect
    for original_path, processed_path, processed in processor:
  File ""/Users/achedeuzot/Documents/Projects/project/www.project.net/lib/python3.4/site-packages/pipeline/storage.py"", line 32, in post_process
    packager.pack_stylesheets(package)
  File ""/Users/achedeuzot/Documents/Projects/project/www.project.net/lib/python3.4/site-packages/pipeline/packager.py"", line 94, in pack_stylesheets
    variant=package.variant, **kwargs)
  File ""/Users/achedeuzot/Documents/Projects/project/www.project.net/lib/python3.4/site-packages/pipeline/packager.py"", line 105, in pack
    self.save_file(output_filename, content)
  File ""/Users/achedeuzot/Documents/Projects/project/www.project.net/lib/python3.4/site-packages/pipeline/packager.py"", line 117, in save_file
    return self.storage.save(path, ContentFile(""\xf4""))
  File ""/Users/achedeuzot/Documents/Projects/project/www.project.net/lib/python3.4/site-packages/django/core/files/storage.py"", line 51, in save
    name = self._save(name, content)
  File ""/Users/achedeuzot/Documents/Projects/project/www.project.net/lib/python3.4/site-packages/django/core/files/storage.py"", line 228, in _save
    _file.write(chunk)
UnicodeEncodeError: 'ascii' codec can't encode character '\xf4' in position 0: ordinal not in range(128)
}}}

After some enquiries, the same error could be reproduced using the following lines:
{{{
    from django.contrib.staticfiles.storage import StaticFilesStorage
    from django.core.files.base import ContentFile

    storage = StaticFilesStorage()
    storage.save(""test-output.txt"", ContentFile(""ô""))
}}}

I was able to ""solve"" the issue by using `bytearray`:
{{{
    storage.save(""test-output.txt"", ContentFile(bytearray(""ô"", 'utf-8')))
}}}

Also, another solution is/was to add `LANG=en_US.utf-8` into the environment.

The line that trows the error is the following:
{{{
    mode = 'wb' if isinstance(chunk, bytes) else 'wt'
    _file = os.fdopen(fd, mode)
_file.write(chunk) # <= this one
}}}

We can see that if I send bytes, the write() happens without errors while if I send a string, the write() fails.

How come Django fails on a ""standard"" utf-8 string ? Is this a Django issue or from something else (OS write(), ...)?"	Bug	closed	File uploads/storage	1.7-rc-2	Normal	invalid	UnicodeEncodeError		Unreviewed	0	0	0	0	0	0