Ellipses in output strings cause crashes on Windows

[Dan Davis]

By default, Windows terminals start with code page 437, which is roughly equivalent to ASCII. While working on #30178, I discovered that this causes the following traceback:

(mplusmon) λ manage.py runserver
Watchman unavailable: pywatchman not installed.
Watching for file changes with StatReloader
Exception in thread Thread-1:
Traceback (most recent call last):
  File "c:\tools\python\3.5\Lib\threading.py", line 914, in _bootstrap_inner
    self.run()
  File "c:\tools\python\3.5\Lib\threading.py", line 862, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\davisda4\PythonEnvs\mplusmon\lib\site-packages\django\utils\autoreload.py", line 54, in wrapper
    fn(*args, **kwargs)
  File "C:\Users\davisda4\PythonEnvs\mplusmon\lib\site-packages\django\core\management\commands\runserver.py", line 116, in inner_run
    self.stdout.write("Performing system checks\u2026\n\n")
  File "C:\Users\davisda4\PythonEnvs\mplusmon\lib\site-packages\django\core\management\base.py", line 145, in write
    self._out.write(style_func(msg))
  File "C:\Users\davisda4\PythonEnvs\mplusmon\lib\encodings\cp437.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_map)[0]
UnicodeEncodeError: 'charmap' codec can't encode character '\u2026' in position 24: character maps to <undefined>

A workaround is to change the code page to UTF-8, as follows:

chcp 65001

This appears to be a new issue relative to Django 2.1.7, because the problem did not occur there. I'll track down wherever someone entered the horizontal elipsis character rather than "..." into the code, because this ought to be easy enough to fix. Whomever did this was probably not even aware of it and it has worked well enough since almost all Django developers will use MacOS or Linux.

[Tim Graham]

It was changed in #29654.

I also saw a report of the crash on ​django-users.

Since there's not much benefit to the ellipses characters compared to three dots, I think it's fine to revert the problematic parts.

[Dan Davis]

Ellipsis is in runserver command.

[Dan Davis]

Pull request - ​https://github.com/django/django/pull/10989

[Tim Graham]

Please look at the linked ticket and uncheck "Patch needs improvement" after reverting all strings that might be output to the shell.

[Dan Davis]

I will do that hopefully Thursday. Tomorrow is cub scout den meeting and I must prepare ;)
I maybe can write a unit test that captures output from management commands and tests them, but that maybe a little much.

[Dan Davis]

Fun with ast module - ​https://gist.github.com/danizen/e8c53a36af6174f88194c5b1bd3e7a3f

That would probably be overmuch as a unit test. Still, all core management commands and contrib management commands are now cp437 safe.

Let me know if I rebased properly:

git rebase -i HEAD~2
git pull upstream master

[Dan Davis]

Replying to Tim Graham:

It was changed in #29654.

Then I shall also check database creations. it is of course true that any "engineer" worth their salt should be unicode savvy, but DjangoCon 2016 convinced me that Django's tutorial should work just fine for a 15 year old programmer using their Mom's older Windows 7 Professional laptop.

Fortunate for me that matches up with the desktops my users are constrained to use at work :[

[Jon Dufresne]

Is there a larger problem that any Unicode output will fail on Windows? If so, is this just avoiding a single case rather than solving a deeper problem? Isn't it reasonable to expect some other Unicode output down the road?

[Tim Graham]

Perhaps we could try to make Django not crash and have it instead output some instructions about how to correct a shell configuration to support Unicode. But that would be a separate low priority enhancement, and I still wouldn't put non-ASCII characters like ellipses in output strings when a suitable alternative exists. I haven't seen any reports of this sort of crash before.

[Tim Graham <timograham@…>]

In 2bd8df24:

Fixed #30184 -- Removed ellipsis characters from shell output strings.

Partially reverted 50b8493581fea3d7137dd8db33bac7008868d23a (refs #29654)
to avoid a crash when the user shell doesn't support non-ASCII characters.

[Tim Graham <timograham@…>]

In 5e94c817:

[2.2.x] Fixed #30184 -- Removed ellipsis characters from shell output strings.

Partially reverted 50b8493581fea3d7137dd8db33bac7008868d23a (refs #29654)
to avoid a crash when the user shell doesn't support non-ASCII characters.
Backport of 2bd8df243ac6fc35e58c9fe90b20c9e42519a5ac from master.

[Claude Paroz]

We are in 2019, and there still are terminals not supporting Unicode, sad :-(