Non-ASCII templates break `django-admin.py startproject --template=TEMPLATE`

[Antti Kaihola]

If a file in a project template for manage.py startproject --template=TEMPLATE

• has non-ASCII characters in its contents, and
• is named with one of the extensions given in the --extension= argument,

a UnicodeDecodeError is raised:

$ django-admin.py startproject --template=mytemplate myproject
Traceback (most recent call last):
  File "/django/django/bin/django-admin.py", line 5, in <module>
    management.execute_from_command_line()
  File "django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File "django/core/management/__init__.py", line 381, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "django/core/management/base.py", line 195, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "django/core/management/base.py", line 231, in execute
    output = self.handle(*args, **options)
  File "django/core/management/commands/startproject.py", line 31, in handle
    super(Command, self).handle('project', project_name, target, **options)
  File "django/core/management/templates.py", line 161, in handle
    new_file.write(content.encode('UTF-8'))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 113: ordinal not in range(128)

The content of the template file is read from the disk as a bytestring (line 156) and run through the Django template engine, which outputs the rendered text as a Unicode object (line 159). The Unicode object is then attempted to be written in to a file without encoding it first (line 161).

Files whose extension isn't included in --extension= are read from the disk, not run through the template engine and written back to the destination without any decoding or encoding. For this reason, those files are handled without problems even if they have non-ASCII content.

[Antti Kaihola]

Avoid UnicodeDecodeError by encoding template engine output in UTF-8

[Antti Kaihola]

I accidentally copy-pasted the traceback from a run with a patched Django source tree. This is the correct traceback:

$ django-admin.py startproject --template=project_template g2dt /tmp/g2dt
Traceback (most recent call last):
  File "/django/django/bin/django-admin.py", line 5, in <module>
    management.execute_from_command_line()
  File "django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File "django/core/management/__init__.py", line 381, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "django/core/management/base.py", line 195, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "django/core/management/base.py", line 231, in execute
    output = self.handle(*args, **options)
  File "django/core/management/commands/startproject.py", line 31, in handle
    super(Command, self).handle('project', project_name, target, **options)
  File "django/core/management/templates.py", line 161, in handle
    new_file.write(content)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe4' in position 75: ordinal not in range(128)

[Antti Kaihola]

Added a non-ASCII template to the startproject tests

[Jannis Leidel]

Using codecs.open would be better here, instead of encoding it by hand.

[Clo74]

Another try with the help of the codecs library.
See ​https://github.com/django/django/pull/245/

It added the unit test case from akaihola.

[Claude Paroz]

Please do not set Ready for checkin for your own patches.

[Florian Apolloner]

The patch in the current form causes the following test failures for me:

florian@apollo13:~/sources/django.git/tests$ PYTHONPATH=.. ./runtests.py --settings=test_sqlite admin_scripts
Creating test database for alias 'default'...
Creating test database for alias 'other'...
.....................................................................................................................................FF......F...F.
======================================================================
FAIL: test_custom_project_template (regressiontests.admin_scripts.tests.StartProject)
Make sure the startproject management command is able to use a different project template
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/florian/sources/django.git/tests/regressiontests/admin_scripts/tests.py", line 1444, in test_custom_project_template
    self.assertNoOutput(err)
  File "/home/florian/sources/django.git/tests/regressiontests/admin_scripts/tests.py", line 159, in assertNoOutput
    self.assertEqual(len(stream), 0, "Stream should be empty: actually contains '%s'" % stream)
AssertionError: Stream should be empty: actually contains 'UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 47: ordinal not in range(128)
'

======================================================================
FAIL: test_custom_project_template_context_variables (regressiontests.admin_scripts.tests.StartProject)
Make sure template context variables are rendered with proper values
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/florian/sources/django.git/tests/regressiontests/admin_scripts/tests.py", line 1537, in test_custom_project_template_context_variables
    self.assertNoOutput(err)
  File "/home/florian/sources/django.git/tests/regressiontests/admin_scripts/tests.py", line 159, in assertNoOutput
    self.assertEqual(len(stream), 0, "Stream should be empty: actually contains '%s'" % stream)
AssertionError: Stream should be empty: actually contains 'UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 47: ordinal not in range(128)
'

======================================================================
FAIL: test_no_escaping_of_project_variables (regressiontests.admin_scripts.tests.StartProject)
Make sure template context variables are not html escaped
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/florian/sources/django.git/tests/regressiontests/admin_scripts/tests.py", line 1554, in test_no_escaping_of_project_variables
    self.assertNoOutput(err)
  File "/home/florian/sources/django.git/tests/regressiontests/admin_scripts/tests.py", line 159, in assertNoOutput
    self.assertEqual(len(stream), 0, "Stream should be empty: actually contains '%s'" % stream)
AssertionError: Stream should be empty: actually contains 'UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 47: ordinal not in range(128)
'

======================================================================
FAIL: test_template_dir_with_trailing_slash (regressiontests.admin_scripts.tests.StartProject)
Ticket 17475: Template dir passed has a trailing path separator
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/florian/sources/django.git/tests/regressiontests/admin_scripts/tests.py", line 1456, in test_template_dir_with_trailing_slash
    self.assertNoOutput(err)
  File "/home/florian/sources/django.git/tests/regressiontests/admin_scripts/tests.py", line 159, in assertNoOutput
    self.assertEqual(len(stream), 0, "Stream should be empty: actually contains '%s'" % stream)
AssertionError: Stream should be empty: actually contains 'UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 47: ordinal not in range(128)
'

----------------------------------------------------------------------
Ran 147 tests in 22.055s

FAILED (failures=4)
Destroying test database for alias 'default'...
Destroying test database for alias 'other'...

Please reopen a pull request once those are fixed, Thx!

[Tomáš Ehrlich]

Tests fixed, ​https://github.com/django/django/pull/298

[Florian Apolloner <florian@…>]

In [3afb5916b215c79e36408b729c9516bc435f5cb7]:

Fixed #18091 -- Non-ASCII templates break django-admin.py startproject --template=TEMPLATE.

Thanks to Claude Huchet and Tomáš Ehrlich for the patch.

[Ramiro Morales <cramm0@…>]

In 12ca312e1bc2cab614aed854b4c38dd86fefbdda:

Modified test added in 3afb5916b2 so it doesn't fail on Windows.

Refs #18091.

[Ramiro Morales <cramm0@…>]

In c4468e0619ef45cae7914b2ebf8357951342dd72:

[1.6.x] Modified test added in 3afb5916b2 so it doesn't fail on Windows.

Refs #18091.

12ca312e1b from master.