#23317 closed Bug (invalid)

StaticFileStorage Encoding error on '\xf4'

Reported by: achedeuzot Owned by: nobody
Component: File uploads/storage Version: 1.7-rc-2
Severity: Normal Keywords: UnicodeEncodeError
Cc: Triage Stage: Unreviewed
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

While using the collectstatic command with django-pipeline and a i18n js library (containing lots of foreign characters), I stumbled on an encoding error:

Traceback (most recent call last):
  File "./dev_manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/Users/achedeuzot/Documents/Projects/project/www.project.net/lib/python3.4/site-packages/django/core/management/__init__.py", line 385, in execute_from_command_line
    utility.execute()
  File "/Users/achedeuzot/Documents/Projects/project/www.project.net/lib/python3.4/site-packages/django/core/management/__init__.py", line 377, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/Users/achedeuzot/Documents/Projects/project/www.project.net/lib/python3.4/site-packages/django/core/management/base.py", line 288, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "/Users/achedeuzot/Documents/Projects/project/www.project.net/lib/python3.4/site-packages/django/core/management/base.py", line 337, in execute
    output = self.handle(*args, **options)
  File "/Users/achedeuzot/Documents/Projects/project/www.project.net/lib/python3.4/site-packages/django/core/management/base.py", line 532, in handle
    return self.handle_noargs(**options)
  File "/Users/achedeuzot/Documents/Projects/project/www.project.net/lib/python3.4/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 168, in handle_noargs
    collected = self.collect()
  File "/Users/achedeuzot/Documents/Projects/project/www.project.net/lib/python3.4/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 114, in collect
    for original_path, processed_path, processed in processor:
  File "/Users/achedeuzot/Documents/Projects/project/www.project.net/lib/python3.4/site-packages/pipeline/storage.py", line 32, in post_process
    packager.pack_stylesheets(package)
  File "/Users/achedeuzot/Documents/Projects/project/www.project.net/lib/python3.4/site-packages/pipeline/packager.py", line 94, in pack_stylesheets
    variant=package.variant, **kwargs)
  File "/Users/achedeuzot/Documents/Projects/project/www.project.net/lib/python3.4/site-packages/pipeline/packager.py", line 105, in pack
    self.save_file(output_filename, content)
  File "/Users/achedeuzot/Documents/Projects/project/www.project.net/lib/python3.4/site-packages/pipeline/packager.py", line 117, in save_file
    return self.storage.save(path, ContentFile("\xf4"))
  File "/Users/achedeuzot/Documents/Projects/project/www.project.net/lib/python3.4/site-packages/django/core/files/storage.py", line 51, in save
    name = self._save(name, content)
  File "/Users/achedeuzot/Documents/Projects/project/www.project.net/lib/python3.4/site-packages/django/core/files/storage.py", line 228, in _save
    _file.write(chunk)
UnicodeEncodeError: 'ascii' codec can't encode character '\xf4' in position 0: ordinal not in range(128)

After some enquiries, the same error could be reproduced using the following lines:

    from django.contrib.staticfiles.storage import StaticFilesStorage
    from django.core.files.base import ContentFile

    storage = StaticFilesStorage()
    storage.save("test-output.txt", ContentFile("ô"))

I was able to "solve" the issue by using bytearray:

    storage.save("test-output.txt", ContentFile(bytearray("ô", 'utf-8')))

Also, another solution is/was to add LANG=en_US.utf-8 into the environment.

The line that trows the error is the following:

    mode = 'wb' if isinstance(chunk, bytes) else 'wt'
    _file = os.fdopen(fd, mode)
_file.write(chunk) # <= this one

We can see that if I send bytes, the write() happens without errors while if I send a string, the write() fails.

How come Django fails on a "standard" utf-8 string ? Is this a Django issue or from something else (OS write(), ...)?

Change History (3)

comment:1 Changed 11 months ago by areski

  • Needs documentation unset
  • Needs tests unset
  • Patch needs improvement unset

I cannot reproduce this bug, I tested the following with Python2.7 and Python3.4:

import os, django
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "myproject.settings")
django.setup()

from django.contrib.staticfiles.storage import StaticFilesStorage
from django.core.files.base import ContentFile

storage = StaticFilesStorage()
storage.save("test-output.txt", ContentFile("ô"))

I work on Ubuntu14 and by default LANG is set, I tried to unset it but no luck.

comment:2 Changed 11 months ago by achedeuzot

Additional info:

  • MacOSX 10.9.4
  • Python 3.4.1
  • Env: no LANG nor LC_ALL nor LC_CTYPE (set but empty)
  • Django 1.7r2

Thanks for your update @areski !

comment:3 Changed 10 months ago by timgraham

  • Resolution set to invalid
  • Status changed from new to closed

I think it's a problem with your configuration, see this FAQ entry.

Note: See TracTickets for help on using tickets.
Back to Top