Django runserver segfaults with autoreload enabled on Windows 10

[Aleksi Häkli]

Django runserver segfaults with autoreload enabled on the os.spawnve call on Python 3.5 and Python 3.6.

The problem is apparently on this line:

​https://github.com/django/django/blob/b79fc11d730b5beff92e9dd8853a61524cdeffe3/django/utils/autoreload.py#L290

Python documentation states that the os.spawnve function is NOT thread-safe on Windows and that the subprocess module should be used instead.

I hence suspect that a thread that is spawned on reload makes a memory violation and causes a segmentation fault which kills off the server.

I propose that autoreload.py be refactored to use a thread-safe implementation either by the subprocess module or by other alternative implementation.

Update: I added a preliminary patch implementation up for review that implements a fix for Python 2.4+ in ​https://github.com/django/djangopull/7748 by using the recommended subprocess.call() instead of the non-thread-safe os.spawnve() which was added in 2005.

---

Discussion

I have run Django 1.8, 1.9 and 1.10 on multiple Python versions on top of Windows 10 64-bit on different computers. Starting with Python 3.5 and Python 3.6 I have reproduced a consistent segmentation fault on the runserver command with both 32-bit and 64-bit Pythons. This has happened for several months. python manage.py migrate and other commands work well and consistently. python manage.py runserver is the ONLY command that segmentation faults and does NOT do this with --noreload option. Today I dug a bit deeper on the cause of the segmentation fault.

As a note, I have also run Django on top of Python 3.4 and at least the 32-bit distribution, for whatever reason, tends not to cause errors like these. This is, however, just my butt-feeling, as we say in Finland, and I have not measured the stuff for a small while.

---

Demonstration

This example is run with virtualenvwrapper installed virtual environment with Python 3.6.0 32-bit with the latest available Microsoft Visual C++ 2015 Redistributable (x86) - 14.0.23026 downloaded and patched today on Windows 10 64-bit, so I expect my computer is up to date. Program versions and tracebacks following.

---

Python installation

USER@COMPUTER MINGW32 ~
$ python
Python 3.6.0 (v3.6.0:41df79263a11, Dec 23 2016, 07:18:10) [MSC v.1900 32 bit (Intel)] on win32

USER@COMPUTER MINGW32 ~
$ pip freeze  # global dependencies
pbr==1.10.0
six==1.10.0
stevedore==1.19.1
virtualenv==15.1.0
virtualenv-clone==0.2.6
virtualenvwrapper==4.7.2

---

gdb traceback on the error which identifies ucrtbase.dll and wdupenv_s as the evil party:

(demo)
USER@COMPUTER MINGW32 ~/Documents/projects/demo (master)
$ gdb python
GNU gdb (GDB) 7.6.1
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "mingw32".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from C:\Users\USER\.virtualenvs\demo\Scripts\python.exe...(no debugging symbols found)...done.
(gdb) run manage.py runserver
Starting program: C:\Users\USER\.virtualenvs\demo\Scripts/python.exe manage.py runserver
[New Thread 6876.0x2a18]
[New Thread 6876.0x13f4]
[New Thread 6876.0x25d0]
[New Thread 6876.0x2a64]

Program received signal SIGSEGV, Segmentation fault.
0x77cebd9e in wdupenv_s () from C:\WINDOWS\SysWoW64\ucrtbase.dll
(gdb)

---

faulthandler traceback with a vanilla manage.py script with faulthandler enabled with faulthandler.enable() which traces the Django error back to autoreload.py:

(demo)
USER@COMPUTER MINGW32 ~/Documents/projects/demo (master)
$ python manage.py runserver

Windows fatal exception: access violation

Current thread 0x00001b6c (most recent call first):
  File "C:\Users\USER\.virtualenvs\demo\lib\site-packages\django\utils\autoreload.py", line 290 in restart_with_reloader
  File "C:\Users\USER\.virtualenvs\demo\lib\site-packages\django\utils\autoreload.py", line 304 in python_reloader
  File "C:\Users\USER\.virtualenvs\demo\lib\site-packages\django\utils\autoreload.py", line 333 in main
  File "C:\Users\USER\.virtualenvs\demo\lib\site-packages\django\core\management\commands\runserver.py", line 106 in run
  File "C:\Users\USER\.virtualenvs\demo\lib\site-packages\django\core\management\commands\runserver.py", line 97 in handle
  File "C:\Users\USER\.virtualenvs\demo\lib\site-packages\django\core\management\base.py", line 345 in execute
  File "C:\Users\USER\.virtualenvs\demo\lib\site-packages\django\core\management\commands\runserver.py", line 58 in execute
  File "C:\Users\USER\.virtualenvs\demo\lib\site-packages\django\core\management\base.py", line 294 in run_from_argv
  File "C:\Users\USER\.virtualenvs\demo\lib\site-packages\django\core\management\__init__.py", line 359 in execute
  File "C:\Users\USER\.virtualenvs\demo\lib\site-packages\django\core\management\__init__.py", line 367 in execute_from_command_line
  File "manage.py", line 26 in <module>
Segmentation fault

[Aleksi Häkli]

More detailed title

[Aleksi Häkli]

Add analysis on the os.spawnve() behaviour and its thread-safety.

[Aleksi Häkli]

Fix styling.

[Aleksi Häkli]

​PR