Opened 12 years ago

Closed 12 years ago

Last modified 10 years ago

#18091 closed Bug (fixed)

Non-ASCII templates break `django-admin.py startproject --template=TEMPLATE`

Reported by: Antti Kaihola Owned by: nobody
Component: Core (Management commands) Version: dev
Severity: Normal Keywords:
Cc: Clo74, Florian Apolloner, tomas.ehrlich@… Triage Stage: Accepted
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

If a file in a project template for manage.py startproject --template=TEMPLATE

  • has non-ASCII characters in its contents, and
  • is named with one of the extensions given in the --extension= argument,

a UnicodeDecodeError is raised:

$ django-admin.py startproject --template=mytemplate myproject
Traceback (most recent call last):
  File "/django/django/bin/django-admin.py", line 5, in <module>
    management.execute_from_command_line()
  File "django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File "django/core/management/__init__.py", line 381, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "django/core/management/base.py", line 195, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "django/core/management/base.py", line 231, in execute
    output = self.handle(*args, **options)
  File "django/core/management/commands/startproject.py", line 31, in handle
    super(Command, self).handle('project', project_name, target, **options)
  File "django/core/management/templates.py", line 161, in handle
    new_file.write(content.encode('UTF-8'))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 113: ordinal not in range(128)

The content of the template file is read from the disk as a bytestring (line 156) and run through the Django template engine, which outputs the rendered text as a Unicode object (line 159). The Unicode object is then attempted to be written in to a file without encoding it first (line 161).

Files whose extension isn't included in --extension= are read from the disk, not run through the template engine and written back to the destination without any decoding or encoding. For this reason, those files are handled without problems even if they have non-ASCII content.

Attachments (2)

18091-startproject-non-ascii-templates.diff (667 bytes ) - added by Antti Kaihola 12 years ago.
Avoid UnicodeDecodeError by encoding template engine output in UTF-8
18091-startproject-non-ascii-templates.2.diff (2.5 KB ) - added by Antti Kaihola 12 years ago.
Added a non-ASCII template to the startproject tests

Download all attachments as: .zip

Change History (12)

by Antti Kaihola, 12 years ago

Avoid UnicodeDecodeError by encoding template engine output in UTF-8

comment:1 by Antti Kaihola, 12 years ago

Summary: Non-ASCII templates break `manage.py startproject --template=TEMPLATE`Non-ASCII templates break `django-admin.py startproject --template=TEMPLATE`

I accidentally copy-pasted the traceback from a run with a patched Django source tree. This is the correct traceback:

$ django-admin.py startproject --template=project_template g2dt /tmp/g2dt
Traceback (most recent call last):
  File "/django/django/bin/django-admin.py", line 5, in <module>
    management.execute_from_command_line()
  File "django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File "django/core/management/__init__.py", line 381, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "django/core/management/base.py", line 195, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "django/core/management/base.py", line 231, in execute
    output = self.handle(*args, **options)
  File "django/core/management/commands/startproject.py", line 31, in handle
    super(Command, self).handle('project', project_name, target, **options)
  File "django/core/management/templates.py", line 161, in handle
    new_file.write(content)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe4' in position 75: ordinal not in range(128)

by Antti Kaihola, 12 years ago

Added a non-ASCII template to the startproject tests

comment:2 by Jannis Leidel, 12 years ago

Patch needs improvement: set
Triage Stage: UnreviewedAccepted

Using codecs.open would be better here, instead of encoding it by hand.

comment:3 by Clo74, 12 years ago

Cc: Clo74 added
Triage Stage: AcceptedReady for checkin

Another try with the help of the codecs library.
See https://github.com/django/django/pull/245/

It added the unit test case from akaihola.

comment:4 by Claude Paroz, 12 years ago

Triage Stage: Ready for checkinAccepted

Please do not set Ready for checkin for your own patches.

comment:5 by Florian Apolloner, 12 years ago

The patch in the current form causes the following test failures for me:

florian@apollo13:~/sources/django.git/tests$ PYTHONPATH=.. ./runtests.py --settings=test_sqlite admin_scripts
Creating test database for alias 'default'...
Creating test database for alias 'other'...
.....................................................................................................................................FF......F...F.
======================================================================
FAIL: test_custom_project_template (regressiontests.admin_scripts.tests.StartProject)
Make sure the startproject management command is able to use a different project template
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/florian/sources/django.git/tests/regressiontests/admin_scripts/tests.py", line 1444, in test_custom_project_template
    self.assertNoOutput(err)
  File "/home/florian/sources/django.git/tests/regressiontests/admin_scripts/tests.py", line 159, in assertNoOutput
    self.assertEqual(len(stream), 0, "Stream should be empty: actually contains '%s'" % stream)
AssertionError: Stream should be empty: actually contains 'UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 47: ordinal not in range(128)
'

======================================================================
FAIL: test_custom_project_template_context_variables (regressiontests.admin_scripts.tests.StartProject)
Make sure template context variables are rendered with proper values
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/florian/sources/django.git/tests/regressiontests/admin_scripts/tests.py", line 1537, in test_custom_project_template_context_variables
    self.assertNoOutput(err)
  File "/home/florian/sources/django.git/tests/regressiontests/admin_scripts/tests.py", line 159, in assertNoOutput
    self.assertEqual(len(stream), 0, "Stream should be empty: actually contains '%s'" % stream)
AssertionError: Stream should be empty: actually contains 'UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 47: ordinal not in range(128)
'

======================================================================
FAIL: test_no_escaping_of_project_variables (regressiontests.admin_scripts.tests.StartProject)
Make sure template context variables are not html escaped
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/florian/sources/django.git/tests/regressiontests/admin_scripts/tests.py", line 1554, in test_no_escaping_of_project_variables
    self.assertNoOutput(err)
  File "/home/florian/sources/django.git/tests/regressiontests/admin_scripts/tests.py", line 159, in assertNoOutput
    self.assertEqual(len(stream), 0, "Stream should be empty: actually contains '%s'" % stream)
AssertionError: Stream should be empty: actually contains 'UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 47: ordinal not in range(128)
'

======================================================================
FAIL: test_template_dir_with_trailing_slash (regressiontests.admin_scripts.tests.StartProject)
Ticket 17475: Template dir passed has a trailing path separator
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/florian/sources/django.git/tests/regressiontests/admin_scripts/tests.py", line 1456, in test_template_dir_with_trailing_slash
    self.assertNoOutput(err)
  File "/home/florian/sources/django.git/tests/regressiontests/admin_scripts/tests.py", line 159, in assertNoOutput
    self.assertEqual(len(stream), 0, "Stream should be empty: actually contains '%s'" % stream)
AssertionError: Stream should be empty: actually contains 'UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 47: ordinal not in range(128)
'

----------------------------------------------------------------------
Ran 147 tests in 22.055s

FAILED (failures=4)
Destroying test database for alias 'default'...
Destroying test database for alias 'other'...

Please reopen a pull request once those are fixed, Thx!

comment:6 by Florian Apolloner, 12 years ago

Cc: Florian Apolloner added

comment:7 by Tomáš Ehrlich, 12 years ago

Cc: tomas.ehrlich@… added
Patch needs improvement: unset

comment:8 by Florian Apolloner <florian@…>, 12 years ago

Resolution: fixed
Status: newclosed

In [3afb5916b215c79e36408b729c9516bc435f5cb7]:

Fixed #18091 -- Non-ASCII templates break django-admin.py startproject --template=TEMPLATE.

Thanks to Claude Huchet and Tomáš Ehrlich for the patch.

comment:9 by Ramiro Morales <cramm0@…>, 10 years ago

In 12ca312e1bc2cab614aed854b4c38dd86fefbdda:

Modified test added in 3afb5916b2 so it doesn't fail on Windows.

Refs #18091.

comment:10 by Ramiro Morales <cramm0@…>, 10 years ago

In c4468e0619ef45cae7914b2ebf8357951342dd72:

[1.6.x] Modified test added in 3afb5916b2 so it doesn't fail on Windows.

Refs #18091.

12ca312e1b from master.

Note: See TracTickets for help on using tickets.
Back to Top