Opened 3 years ago

Closed 3 years ago

Last modified 18 months ago

#18091 closed Bug (fixed)

Non-ASCII templates break `django-admin.py startproject --template=TEMPLATE`

Reported by: akaihola Owned by: nobody
Component: Core (Management commands) Version: master
Severity: Normal Keywords:
Cc: Clo74, apollo13, tomas.ehrlich@… Triage Stage: Accepted
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

If a file in a project template for manage.py startproject --template=TEMPLATE

  • has non-ASCII characters in its contents, and
  • is named with one of the extensions given in the --extension= argument,

a UnicodeDecodeError is raised:

$ django-admin.py startproject --template=mytemplate myproject
Traceback (most recent call last):
  File "/django/django/bin/django-admin.py", line 5, in <module>
    management.execute_from_command_line()
  File "django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File "django/core/management/__init__.py", line 381, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "django/core/management/base.py", line 195, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "django/core/management/base.py", line 231, in execute
    output = self.handle(*args, **options)
  File "django/core/management/commands/startproject.py", line 31, in handle
    super(Command, self).handle('project', project_name, target, **options)
  File "django/core/management/templates.py", line 161, in handle
    new_file.write(content.encode('UTF-8'))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 113: ordinal not in range(128)

The content of the template file is read from the disk as a bytestring (line 156) and run through the Django template engine, which outputs the rendered text as a Unicode object (line 159). The Unicode object is then attempted to be written in to a file without encoding it first (line 161).

Files whose extension isn't included in --extension= are read from the disk, not run through the template engine and written back to the destination without any decoding or encoding. For this reason, those files are handled without problems even if they have non-ASCII content.

Attachments (2)

18091-startproject-non-ascii-templates.diff (667 bytes) - added by akaihola 3 years ago.
Avoid UnicodeDecodeError by encoding template engine output in UTF-8
18091-startproject-non-ascii-templates.2.diff (2.5 KB) - added by akaihola 3 years ago.
Added a non-ASCII template to the startproject tests

Download all attachments as: .zip

Change History (12)

Changed 3 years ago by akaihola

Avoid UnicodeDecodeError by encoding template engine output in UTF-8

comment:1 Changed 3 years ago by akaihola

  • Needs documentation unset
  • Needs tests unset
  • Patch needs improvement unset
  • Summary changed from Non-ASCII templates break `manage.py startproject --template=TEMPLATE` to Non-ASCII templates break `django-admin.py startproject --template=TEMPLATE`

I accidentally copy-pasted the traceback from a run with a patched Django source tree. This is the correct traceback:

$ django-admin.py startproject --template=project_template g2dt /tmp/g2dt
Traceback (most recent call last):
  File "/django/django/bin/django-admin.py", line 5, in <module>
    management.execute_from_command_line()
  File "django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File "django/core/management/__init__.py", line 381, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "django/core/management/base.py", line 195, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "django/core/management/base.py", line 231, in execute
    output = self.handle(*args, **options)
  File "django/core/management/commands/startproject.py", line 31, in handle
    super(Command, self).handle('project', project_name, target, **options)
  File "django/core/management/templates.py", line 161, in handle
    new_file.write(content)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe4' in position 75: ordinal not in range(128)

Changed 3 years ago by akaihola

Added a non-ASCII template to the startproject tests

comment:2 Changed 3 years ago by jezdez

  • Patch needs improvement set
  • Triage Stage changed from Unreviewed to Accepted

Using codecs.open would be better here, instead of encoding it by hand.

comment:3 Changed 3 years ago by Clo74

  • Cc Clo74 added
  • Triage Stage changed from Accepted to Ready for checkin

Another try with the help of the codecs library.
See https://github.com/django/django/pull/245/

It added the unit test case from akaihola.

comment:4 Changed 3 years ago by claudep

  • Triage Stage changed from Ready for checkin to Accepted

Please do not set Ready for checkin for your own patches.

comment:5 Changed 3 years ago by apollo13

The patch in the current form causes the following test failures for me:

florian@apollo13:~/sources/django.git/tests$ PYTHONPATH=.. ./runtests.py --settings=test_sqlite admin_scripts
Creating test database for alias 'default'...
Creating test database for alias 'other'...
.....................................................................................................................................FF......F...F.
======================================================================
FAIL: test_custom_project_template (regressiontests.admin_scripts.tests.StartProject)
Make sure the startproject management command is able to use a different project template
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/florian/sources/django.git/tests/regressiontests/admin_scripts/tests.py", line 1444, in test_custom_project_template
    self.assertNoOutput(err)
  File "/home/florian/sources/django.git/tests/regressiontests/admin_scripts/tests.py", line 159, in assertNoOutput
    self.assertEqual(len(stream), 0, "Stream should be empty: actually contains '%s'" % stream)
AssertionError: Stream should be empty: actually contains 'UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 47: ordinal not in range(128)
'

======================================================================
FAIL: test_custom_project_template_context_variables (regressiontests.admin_scripts.tests.StartProject)
Make sure template context variables are rendered with proper values
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/florian/sources/django.git/tests/regressiontests/admin_scripts/tests.py", line 1537, in test_custom_project_template_context_variables
    self.assertNoOutput(err)
  File "/home/florian/sources/django.git/tests/regressiontests/admin_scripts/tests.py", line 159, in assertNoOutput
    self.assertEqual(len(stream), 0, "Stream should be empty: actually contains '%s'" % stream)
AssertionError: Stream should be empty: actually contains 'UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 47: ordinal not in range(128)
'

======================================================================
FAIL: test_no_escaping_of_project_variables (regressiontests.admin_scripts.tests.StartProject)
Make sure template context variables are not html escaped
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/florian/sources/django.git/tests/regressiontests/admin_scripts/tests.py", line 1554, in test_no_escaping_of_project_variables
    self.assertNoOutput(err)
  File "/home/florian/sources/django.git/tests/regressiontests/admin_scripts/tests.py", line 159, in assertNoOutput
    self.assertEqual(len(stream), 0, "Stream should be empty: actually contains '%s'" % stream)
AssertionError: Stream should be empty: actually contains 'UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 47: ordinal not in range(128)
'

======================================================================
FAIL: test_template_dir_with_trailing_slash (regressiontests.admin_scripts.tests.StartProject)
Ticket 17475: Template dir passed has a trailing path separator
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/florian/sources/django.git/tests/regressiontests/admin_scripts/tests.py", line 1456, in test_template_dir_with_trailing_slash
    self.assertNoOutput(err)
  File "/home/florian/sources/django.git/tests/regressiontests/admin_scripts/tests.py", line 159, in assertNoOutput
    self.assertEqual(len(stream), 0, "Stream should be empty: actually contains '%s'" % stream)
AssertionError: Stream should be empty: actually contains 'UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 47: ordinal not in range(128)
'

----------------------------------------------------------------------
Ran 147 tests in 22.055s

FAILED (failures=4)
Destroying test database for alias 'default'...
Destroying test database for alias 'other'...

Please reopen a pull request once those are fixed, Thx!

comment:6 Changed 3 years ago by apollo13

  • Cc apollo13 added

comment:7 Changed 3 years ago by Elvard

  • Cc tomas.ehrlich@… added
  • Patch needs improvement unset

comment:8 Changed 3 years ago by Florian Apolloner <florian@…>

  • Resolution set to fixed
  • Status changed from new to closed

In [3afb5916b215c79e36408b729c9516bc435f5cb7]:

Fixed #18091 -- Non-ASCII templates break django-admin.py startproject --template=TEMPLATE.

Thanks to Claude Huchet and Tomáš Ehrlich for the patch.

comment:9 Changed 18 months ago by Ramiro Morales <cramm0@…>

In 12ca312e1bc2cab614aed854b4c38dd86fefbdda:

Modified test added in 3afb5916b2 so it doesn't fail on Windows.

Refs #18091.

comment:10 Changed 18 months ago by Ramiro Morales <cramm0@…>

In c4468e0619ef45cae7914b2ebf8357951342dd72:

[1.6.x] Modified test added in 3afb5916b2 so it doesn't fail on Windows.

Refs #18091.

12ca312e1b from master.

Note: See TracTickets for help on using tickets.
Back to Top