#31716 closed Bug (fixed)
django-admin runserver mostly does not work on Windows
Reported by: | Christian Ullrich | Owned by: | Tom Forbes |
---|---|---|---|
Component: | Core (Management commands) | Version: | 3.1 |
Severity: | Normal | Keywords: | Windows |
Cc: | Tom Forbes | Triage Stage: | Ready for checkin |
Has patch: | yes | Needs documentation: | no |
Needs tests: | no | Patch needs improvement: | no |
Easy pickings: | no | UI/UX: | no |
Description
I frequently encounter the problem that running "django-admin runserver" on Windows fails:
<some-venv>\scripts\python.exe: can't open file '<some-venv>\test\Scripts\django-admin': [Errno 2] No such file or directory
The command works if run with --noreload.
There are probably other conditions that must be met for it to fail, or work. I *think* it makes a difference, for example, if Django was installed with pip or setuptools. The .exe "console scripts " are different between the two; one type has a bit of readable Python at the end, the other does not.
What I do know is that the problem is either caused or enabled by the generated console scripts (the .exe wrappers in venv/Scripts) stripping the .exe extension from their own sys.argv[0] when passing that to Python to run. If I sabotage this by changing the regex in the file itself (where present), or by adding the extension again in utils/autoreload.py before running the command line, it works fine.
To be honest, I have no idea whether this is a bug in Django, or pip, or setuptools, or distutils, or something else entirely. I have spent the last two days trying to figure out what exactly goes wrong, and where, with little success. I'm reporting it here because Django is the only place I could find where it is possible to kludge the code into working. I'll be happy to take my business elsewhere if someone can point the way.
Change History (14)
comment:1 by , 4 years ago
Resolution: | → worksforme |
---|---|
Status: | new → closed |
comment:2 by , 4 years ago
Resolution: | worksforme |
---|---|
Status: | closed → new |
Here you go.
C:\Daten>py -m venv test C:\Daten>test\Scripts\activate (test) C:\Daten>python -c "import sys; print(sys.version)" 3.8.3 (tags/v3.8.3:6f8c832, May 13 2020, 22:37:02) [MSC v.1924 64 bit (AMD64)] (test) C:\Daten>python -m pip install -U pip setuptools Django Looking in indexes: https://nexus/repository/PyPI-mirror/simple Collecting pip Downloading https://nexus/repository/PyPI-mirror/packages/pip/20.1.1.post1/pip-20.1.1.post1-py2.py3-none-any.whl (1.6MB) |████████████████████████████████| 1.6MB 656kB/s Collecting setuptools Downloading https://nexus/repository/PyPI-mirror/packages/setuptools/47.3.1/setuptools-47.3.1-py3-none-any.whl (582kB) |████████████████████████████████| 583kB 726kB/s Collecting Django Downloading https://nexus/repository/PyPI-mirror/packages/django/3.0.7/Django-3.0.7-py3-none-any.whl (7.5MB) |████████████████████████████████| 7.5MB 726kB/s Collecting asgiref~=3.2 (from Django) Downloading https://nexus/repository/PyPI-mirror/packages/asgiref/3.2.9/asgiref-3.2.9-py3-none-any.whl Collecting pytz (from Django) Downloading https://nexus/repository/PyPI-mirror/packages/pytz/2020.1/pytz-2020.1-py2.py3-none-any.whl (510kB) |████████████████████████████████| 512kB 726kB/s Collecting sqlparse>=0.2.2 (from Django) Downloading https://nexus/repository/PyPI-mirror/packages/sqlparse/0.3.1/sqlparse-0.3.1-py2.py3-none-any.whl (40kB) |████████████████████████████████| 40kB 871kB/s Installing collected packages: pip, setuptools, asgiref, pytz, sqlparse, Django Found existing installation: pip 19.2.3 Uninstalling pip-19.2.3: Successfully uninstalled pip-19.2.3 Found existing installation: setuptools 41.2.0 Uninstalling setuptools-41.2.0: Successfully uninstalled setuptools-41.2.0 Successfully installed Django-3.0.7 asgiref-3.2.9 pip-20.1.1.post1 pytz-2020.1 setuptools-47.3.1 sqlparse-0.3.1 (test) C:\Daten\dtest>pip list Package Version ---------- ------------ asgiref 3.2.9 Django 3.0.7 pip 20.1.1.post1 pytz 2020.1 setuptools 47.3.1 sqlparse 0.3.1
pip 20.1.1.post1 is 20.1.1 minus the information disclosure. For our purposes it's the same.
(test) C:\Daten>openssl.exe sha256 test\Scripts\django-admin.exe SHA256(test\Scripts\django-admin.exe)= 24a24fefaaa27e069474985353cceacc9cb4128a35f50e99af487768d4255c90
No idea if they are identical between different installations.
(test) C:\Daten>django-admin startproject dtest (test) C:\Daten>set PYTHONPATH=C:\Daten\dtest (test) C:\Daten>set DJANGO_SETTINGS_MODULE=dtest.settings (test) C:\Daten>django-admin Type 'django-admin help <subcommand>' for help on a specific subcommand. Available subcommands: [django] check compilemessages createcachetable dbshell diffsettings dumpdata flush inspectdb loaddata makemessages makemigrations migrate runserver sendtestemail shell showmigrations sqlflush sqlmigrate sqlsequencereset squashmigrations startapp startproject test testserver Note that only Django core commands are listed as settings are not properly configured (error: No module named 'dtest'). (test) C:\Daten>cd dtest (test) C:\Daten\dtest>django-admin runserver C:\Program Files\Python38\python.exe: can't open file 'C:\Daten\test\Scripts\django-admin': [Errno 2] No such file or directory
Note the missing extension.
(test) C:\Daten\dtest>django-admin runserver --noreload Performing system checks... System check identified no issues (0 silenced). You have 17 unapplied migration(s). Your project may not work properly until you apply the migrations for app(s): admin, auth, contenttypes, sessions. Run 'python manage.py migrate' to apply them. June 17, 2020 - 08:58:46 Django version 3.0.7, using settings 'dtest.settings' Starting development server at http://127.0.0.1:8000/ Quit the server with CTRL-BREAK. ^C
Works with --noreload.
Now let's see what it is actually trying to execute.
(test) C:\Daten\dtest>vi ..\test\Lib\site-packages\django\utils\autoreload.py
Before line 230 (the subprocess.run call in restart_with_reloader()), insert "print(args)".
(test) C:\Daten\dtest>django-admin runserver ['c:\\daten\\test\\scripts\\python.exe', 'C:\\Daten\\test\\Scripts\\django-admin', 'runserver'] C:\Program Files\Python38\python.exe: can't open file 'C:\Daten\test\Scripts\django-admin': [Errno 2] No such file or directory
This cannot possibly work.
(test) C:\Daten\dtest>dir ..\test\scripts Volume in drive C is Windows Volume Serial Number is 8EC9-7F66 Directory of C:\Daten\test\scripts 2020-06-17 08:54 <DIR> . 2020-06-17 08:54 <DIR> .. 2020-06-17 08:53 2.273 activate 2020-06-17 08:53 950 activate.bat 2020-06-17 08:53 18.454 Activate.ps1 2020-06-17 08:53 368 deactivate.bat 2020-06-17 08:54 106.388 django-admin.exe 2020-06-17 08:54 142 django-admin.py 2020-06-17 08:54 106.355 easy_install-3.8.exe 2020-06-17 08:54 106.355 easy_install.exe 2020-06-17 08:54 106.346 pip.exe 2020-06-17 08:54 106.346 pip3.8.exe 2020-06-17 08:54 106.346 pip3.exe 2020-06-17 08:53 532.040 python.exe 2020-06-17 08:53 531.016 pythonw.exe 2020-06-17 08:54 106.341 sqlformat.exe 2020-06-17 08:54 <DIR> __pycache__ 14 File(s) 1.829.720 bytes 3 Dir(s) [...] bytes free (test) C:\Daten\dtest>vi -b ..\test\scripts\django-admin.exe
Edit the Python block at the end to not match and remove .exe anymore. I replaced "\.exe" with "\.rxe" in the regex.
(test) C:\Daten\dtest>django-admin runserver ['c:\\daten\\test\\scripts\\python.exe', 'C:\\Daten\\test\\Scripts\\django-admin.exe', 'runserver'] Watching for file changes with StatReloader Performing system checks... System check identified no issues (0 silenced). You have 17 unapplied migration(s). Your project may not work properly until you apply the migrations for app(s): admin, auth, contenttypes, sessions. Run 'python manage.py migrate' to apply them. June 17, 2020 - 09:07:47 Django version 3.0.7, using settings 'dtest.settings' Starting development server at http://127.0.0.1:8000/ Quit the server with CTRL-BREAK. ^C
With the extension left alone, this works now, too.
The .exe. generally isn't needed.
Not if it is the command name. Here it is an argument on python's command line, though, and python does not guess about missing extensions.
comment:3 by , 4 years ago
Component: | Uncategorized → Core (Management commands) |
---|---|
Keywords: | Windows added |
Triage Stage: | Unreviewed → Accepted |
Type: | Uncategorized → Bug |
OK, good yes, I see it. Thanks for the follow-up.
Looks like the auto-reloaded should indeed map back to an actually existing file.
Fancy making a PR for that?
(Wondering if Windows would let us drop the exe extensions totally as we did in POSIX-land...)
In the meantime, manage.py
works as expected.
comment:4 by , 4 years ago
The best solution that I can come up with is adding this in the else branch of get_child_arguments():
if sys.platform == 'win32': for i in range(1, len(args)): if args[i].lower().endswith("scripts\\django-admin"): # setuptools style; setup.py develop or pip install -e if os.path.exists(args[i] + "-script.py"): args[i] = args[i] + "-script.py" # pip style elif os.path.exists(args[i] + ".exe"): args[i] = args[i] + ".exe"
I think this has about as much chance of being accepted as would solving the problem properly by dropping all Windows support, but if you think I should make the attempt, please let me know.
The two branches exist because there are (at least) two types of these .exe "scripts". If generated by pip install, they have ~100 KiB, contain a small block of Python at the end, and can be run directly by the interpreter ("python.exe django-admin.exe" works in that Python finds the embedded script). However, if they are generated by setuptools, they do not have the Python block, are only ~70 KiB, and can not be run as scripts. Instead, there is the "...-script.py" companion script whose name they derive from their own and that they run via sys.executable .
Then there is the case of "python -m django runserver", about whose existence I just found out in get_child_arguments(), and whether that may need special treatment, too.
Not to mention that the processes end up stacked five deep. Yes, it is a dev situation so it doesn't matter much, but a better fix would still be to reinvent autoreloading entirely, again, and somehow get by without the monitoring process and $RUN_MAIN. (Getting rid of the bouncing around between the venv and system pythons would be nice, too, but that's another department.)
comment:5 by , 4 years ago
Cc: | added |
---|
follow-up: 7 comment:6 by , 4 years ago
I think this has about as much chance of being accepted as would solving the problem properly by dropping all Windows support...
😀 We can't do that I'm afraid. (I'm just warming to it as it happens...)
I see Tom has cc-d himself. Let's see if he has any thoughts... ;)
comment:7 by , 4 years ago
Replying to Carlton Gibson:
I think this has about as much chance of being accepted as would solving the problem properly by dropping all Windows support...
😀 We can't do that I'm afraid. (I'm just warming to it as it happens...)
I see Tom has cc-d himself. Let's see if he has any thoughts... ;)
My autoreloader-sense was tingling! I wonder if this has always been the case - the previous implementation was pretty much the same, but I see that Werkzeug has a partial workaround in their reloader: https://github.com/pallets/werkzeug/blob/fc999a6262c847ad185ea3ffe0cc6f2e915a867a/src/werkzeug/_reloader.py#L69-L93
I think we can take inspiration (read: shamelessly copy parts) their implementation as it seems to make sense, but I'd be interested to know why they don't handle -script.py
companions.
Lets be a bit generic here rather than hard-code anything to do with django-admin.py
. If sys.argv[0]
doesn't exist we check if -script.py
exists, falling back to looking for .exe
on Windows only. If we cannot find it we explode with a reasonable error message (what that is I cannot say!).
I'll work on a patch, and i would appreciate it if someone on Windows could test it? :D
comment:8 by , 4 years ago
Owner: | changed from | to
---|---|
Status: | new → assigned |
comment:9 by , 4 years ago
I think this might fix it: https://github.com/orf/django/commit/cb200453065ef0dec7ed97cb09eab02bc99342c1
Christian, any chance you could test with my branch? pip install https://github.com/orf/django/archive/31716-got-99-problems-but-exe-aint-one.zip
comment:12 by , 4 years ago
Triage Stage: | Accepted → Ready for checkin |
---|---|
Version: | 3.0 → 3.1 |
I can't reproduce an error here. venv (and virtualenv and ...) and pip (and ...) all seem to work correctly. Unless you can provide a reproduce I don't think there's anything we can say.
I would expect both
python
anddjango-admin
to be in<some-venv>\Scripts
.The
.exe.
generally isn't needed.If your venv is active you can run
django-admin
directly, e.g.> django-admin help
All I can think is to recommend checking that you're on up to date versions and such, but day to day this works fine normally... Sorry I can't say more.
You may need to see TicketClosingReasons/UseSupportChannels.