Opened 5 years ago

Closed 5 years ago

Last modified 5 years ago

#30184 closed Bug (fixed)

Ellipses in output strings cause crashes on Windows

Reported by: Dan Davis Owned by: Dan Davis
Component: Core (Management commands) Version: 2.2
Severity: Normal Keywords:
Cc: jon.dufresne@… Triage Stage: Accepted
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description (last modified by Dan Davis)

By default, Windows terminals start with code page 437, which is roughly equivalent to ASCII. While working on #30178, I discovered that this causes the following traceback:

(mplusmon) λ manage.py runserver
Watchman unavailable: pywatchman not installed.
Watching for file changes with StatReloader
Exception in thread Thread-1:
Traceback (most recent call last):
  File "c:\tools\python\3.5\Lib\threading.py", line 914, in _bootstrap_inner
    self.run()
  File "c:\tools\python\3.5\Lib\threading.py", line 862, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\davisda4\PythonEnvs\mplusmon\lib\site-packages\django\utils\autoreload.py", line 54, in wrapper
    fn(*args, **kwargs)
  File "C:\Users\davisda4\PythonEnvs\mplusmon\lib\site-packages\django\core\management\commands\runserver.py", line 116, in inner_run
    self.stdout.write("Performing system checks\u2026\n\n")
  File "C:\Users\davisda4\PythonEnvs\mplusmon\lib\site-packages\django\core\management\base.py", line 145, in write
    self._out.write(style_func(msg))
  File "C:\Users\davisda4\PythonEnvs\mplusmon\lib\encodings\cp437.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_map)[0]
UnicodeEncodeError: 'charmap' codec can't encode character '\u2026' in position 24: character maps to <undefined>

A workaround is to change the code page to UTF-8, as follows:

chcp 65001

This appears to be a new issue relative to Django 2.1.7, because the problem did not occur there. I'll track down wherever someone entered the horizontal elipsis character rather than "..." into the code, because this ought to be easy enough to fix. Whomever did this was probably not even aware of it and it has worked well enough since almost all Django developers will use MacOS or Linux.

Change History (13)

comment:1 by Dan Davis, 5 years ago

Description: modified (diff)

comment:2 by Tim Graham, 5 years ago

Easy pickings: unset
Summary: Django 2.2b1 throws error on Windows in default code pageEllipses in output strings cause crashes on Windows
Triage Stage: UnreviewedAccepted
Type: UncategorizedBug

It was changed in #29654.

I also saw a report of the crash on django-users.

Since there's not much benefit to the ellipses characters compared to three dots, I think it's fine to revert the problematic parts.

comment:3 by Dan Davis, 5 years ago

Component: Core (Other)Core (Management commands)
Easy pickings: set
Summary: Ellipses in output strings cause crashes on WindowsDjango 2.2b1 throws error on Windows in default code page

Ellipsis is in runserver command.

comment:5 by Tim Graham, 5 years ago

Easy pickings: unset
Has patch: set
Patch needs improvement: set
Summary: Django 2.2b1 throws error on Windows in default code pageEllipses in output strings cause crashes on Windows

Please look at the linked ticket and uncheck "Patch needs improvement" after reverting all strings that might be output to the shell.

comment:6 by Dan Davis, 5 years ago

I will do that hopefully Thursday. Tomorrow is cub scout den meeting and I must prepare ;)
I maybe can write a unit test that captures output from management commands and tests them, but that maybe a little much.

comment:7 by Dan Davis, 5 years ago

Patch needs improvement: unset

Fun with ast module - https://gist.github.com/danizen/e8c53a36af6174f88194c5b1bd3e7a3f

That would probably be overmuch as a unit test. Still, all core management commands and contrib management commands are now cp437 safe.

Let me know if I rebased properly:

git rebase -i HEAD~2
git pull upstream master

in reply to:  2 comment:8 by Dan Davis, 5 years ago

Replying to Tim Graham:

It was changed in #29654.

Then I shall also check database creations. it is of course true that any "engineer" worth their salt should be unicode savvy, but DjangoCon 2016 convinced me that Django's tutorial should work just fine for a 15 year old programmer using their Mom's older Windows 7 Professional laptop.

Fortunate for me that matches up with the desktops my users are constrained to use at work :[

comment:9 by Jon Dufresne, 5 years ago

Cc: jon.dufresne@… added

Is there a larger problem that any Unicode output will fail on Windows? If so, is this just avoiding a single case rather than solving a deeper problem? Isn't it reasonable to expect some other Unicode output down the road?

comment:10 by Tim Graham, 5 years ago

Perhaps we could try to make Django not crash and have it instead output some instructions about how to correct a shell configuration to support Unicode. But that would be a separate low priority enhancement, and I still wouldn't put non-ASCII characters like ellipses in output strings when a suitable alternative exists. I haven't seen any reports of this sort of crash before.

comment:11 by Tim Graham <timograham@…>, 5 years ago

Resolution: fixed
Status: assignedclosed

In 2bd8df24:

Fixed #30184 -- Removed ellipsis characters from shell output strings.

Partially reverted 50b8493581fea3d7137dd8db33bac7008868d23a (refs #29654)
to avoid a crash when the user shell doesn't support non-ASCII characters.

comment:12 by Tim Graham <timograham@…>, 5 years ago

In 5e94c817:

[2.2.x] Fixed #30184 -- Removed ellipsis characters from shell output strings.

Partially reverted 50b8493581fea3d7137dd8db33bac7008868d23a (refs #29654)
to avoid a crash when the user shell doesn't support non-ASCII characters.
Backport of 2bd8df243ac6fc35e58c9fe90b20c9e42519a5ac from master.

comment:13 by Claude Paroz, 5 years ago

We are in 2019, and there still are terminals not supporting Unicode, sad :-(

Note: See TracTickets for help on using tickets.
Back to Top