Exceptions from settings modules are hidden when running management commands

[Michael]

Maybe its regression, related to: https://code.djangoproject.com/ticket/27522:

> python manage.py runserver --nostatic
usage: manage.py runserver [-h] [--ipv6] [--nothreading] [--noreload] [--version] [-v {0,1,2,3}] [--settings SETTINGS]
                           [--pythonpath PYTHONPATH] [--traceback] [--no-color] [--force-color]
                           [addrport]
manage.py runserver: error: unrecognized arguments: --nostatic

> python manage.py runserver
python manage.py runserver
Traceback (most recent call last):
  File "/home/michael/venv/project/lib/python3.8/site-packages/django/core/management/base.py", line 354, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/home/michael/venv/project/lib/python3.8/site-packages/django/core/management/commands/runserver.py", line 61, in execute
    super().execute(*args, **options)
  File "/home/michael/venv/project/lib/python3.8/site-packages/django/core/management/base.py", line 398, in execute
    output = self.handle(*args, **options)
  File "/home/michael/venv/project/lib/python3.8/site-packages/django/core/management/commands/runserver.py", line 68, in handle
    if not settings.DEBUG and not settings.ALLOWED_HOSTS:
  File "/home/michael/venv/project/lib/python3.8/site-packages/django/conf/__init__.py", line 82, in __getattr__
    self._setup(name)
  File "/home/michael/venv/project/lib/python3.8/site-packages/django/conf/__init__.py", line 69, in _setup
    self._wrapped = Settings(settings_module)
  File "/home/michael/venv/project/lib/python3.8/site-packages/django/conf/__init__.py", line 170, in __init__
    mod = importlib.import_module(self.SETTINGS_MODULE)
  File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 973, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'dist.plug.settings'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "manage.py", line 16, in <module>
    execute_from_command_line(sys.argv)
  File "/home/michael/venv/project/lib/python3.8/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
    utility.execute()
  File "/home/michael/venv/project/lib/python3.8/site-packages/django/core/management/__init__.py", line 413, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/michael/venv/project/lib/python3.8/site-packages/django/core/management/base.py", line 367, in run_from_argv
    connections.close_all()
  File "/home/michael/venv/project/lib/python3.8/site-packages/django/db/utils.py", line 208, in close_all
    for alias in self:
  File "/home/michael/venv/project/lib/python3.8/site-packages/django/utils/connection.py", line 73, in __iter__
    return iter(self.settings)
  File "/home/michael/venv/project/lib/python3.8/site-packages/django/utils/functional.py", line 48, in __get__
    res = instance.__dict__[self.name] = self.func(instance)
  File "/home/michael/venv/project/lib/python3.8/site-packages/django/utils/connection.py", line 45, in settings
    self._settings = self.configure_settings(self._settings)
  File "/home/michael/venv/project/lib/python3.8/site-packages/django/db/utils.py", line 144, in configure_settings
    databases = super().configure_settings(databases)
  File "/home/michael/venv/project/lib/python3.8/site-packages/django/utils/connection.py", line 50, in configure_settings
    settings = getattr(django_settings, self.settings_name)
  File "/home/michael/venv/project/lib/python3.8/site-packages/django/conf/__init__.py", line 82, in __getattr__
    self._setup(name)
  File "/home/michael/venv/project/lib/python3.8/site-packages/django/conf/__init__.py", line 69, in _setup
    self._wrapped = Settings(settings_module)
  File "/home/michael/venv/project/lib/python3.8/site-packages/django/conf/__init__.py", line 170, in __init__
    mod = importlib.import_module(self.SETTINGS_MODULE)
  File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 973, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'dist.plug.settings'

Its a ligit error, just hidden by --nostatic

[Mariusz Felisiak]

Thanks for the report!

Bisected to c6864a01b25591d3a709da8071413b69c9e35341.

[Rohith P R]

​PR

[Mariusz Felisiak]

It's WIP

[bcail]

Opened a new PR from the previous one: ​https://github.com/django/django/pull/16337

[Paul Schilling]

The previous PR's focused on avoiding/working aroung the ImportError introduced by c6864a01b25591d3a709da8071413b69c9e35341. I believe this is treating the symptoms rather than the cause.

The runserver command is split in two: the "standard" command in core/management/commands/runserver.py and the special command in contrib/staticfiles/commands/runserver.py. The latter adds --nostatic and --insecure to the arguments. However, this module imports settings from django.conf. If settings doesn't exist or has errors, contrib/staticfiles/commands/runserver.py cannot be loaded either. As a result, the --nostatic argument is not recognized because it hasn't been added (the same is true of --insecure, although this hasn't been reported).

The problem can be solved by merging the two commands. The runserver command is simplified, and the code that introduced the ImportError can be left as is.

PR: ​https://github.com/django/django/pull/18260

[bcail]

I think ​this PR could be reviewed. Or, if the decision is to go with ​the other PR, I can close mine.

[Sarah Boyce]

Had a look at both PRs and, from what I can tell, they both need some work

[Jacob Walls]

The new attribute needs a release note and documentation.

[Natalia Bidart]

#36422 was a dupe, which is worth looking at to ensure the fix for this ticket covers that use case as well. That means including introspection of ImproperlyConfigured to determine if it emanated from a settings module before deciding to swallow it.

[Jacob Walls]

#36544 was another dupe.

[Brian Helba]

I believe that #36422 (closed as a duplicate of this one) is actually closer to the root cause, and this ticket is tracking a symptom.

The underlying problem is that when DJANGO_SETTINGS_MODULE fails to import, ManagementUtility.execute() ​stashes the exception on self.settings_exception and continues silently.

There is only one code path that surfaces it to the user: main_help_text(), reached by running ./manage.py with no arguments. Every other invocation exits without mentioning that settings failed to load, so end users have no way to diagnose the misconfiguration and remedy it.

The user-facing problem is more significant than just the missing traceback on runserver. Because settings.configured stays False, get_commands() returns only the django.core commands and every other command defined by an app in INSTALLED_APPS silently disappears. The user runs ./manage.py actual_custom_command, gets "Unknown command: 'actual_custom_command'", and is told nothing about why. Since we haven't shown the user the real error with their settings, they're getting mislead about the simplest way to fix the issue.

This ticket is focused on an instance of that same user problem: after a settings failure, django.contrib.staticfiles is silently not added to INSTALLED_APPS, so the --nostatic option disappears (since runserver is never overridden). The best solution isn't to patch --nostatic into the core command, it's to confront the user with the settings failure; once they fix that, INSTALLED_APPS and custom commands will work again.

[Jacob Walls]

Broadened issue title to reflect scope.

[Brian Helba]

I've made ​https://github.com/django/django/pull/21154 , which I hope resolves the root issue. I'm happy to respond to feedback over on the PR.

[Jacob Walls]

​bcail's PR with review feedback applied