Django urls patterns resolver shadowing exceptions thrown everywhere else

[Ariel Torti]

The new version of django somehow captures all the exceptions ocurred while parsing the urlpatterns.

How to reproduce:

• Create a django project
• Create a an app

app/views.py

from django.shortcuts import render

class MyCustomExceptionShadowed(Exception):
  pass

raise MyCustomExceptionShadowed('I have a description of your error')

# Create your views here.
def myview(request):
  pass

urls.py

from django.contrib import admin
from django.urls import path

from bugtest import views

urlpatterns = [
    path('', views.myview),
    path('admin/', admin.site.urls),
]

When running the server I get

django.core.exceptions.ImproperlyConfigured: The included URLconf 'bug.urls' does not appear to have any patterns in it. If you see valid patterns in the file then the issue is probably caused by a circular import.

When in fact I expect it to say

  File "/home/ariel-nt/Desktop/test/bug/bugtest/views.py", line 6, in <module>
    raise MyCustomExceptionShadowed('I have a description of your error')
bugtest.views.MyCustomExceptionShadowed: I have a description of your error

I tested this with django 2.1 and it works as expected, so I'll now try to find the root cause of the error and give an update here
The issue in introduced in this commit ​https://github.com/django/django/commit/097457afe47e50e76d53b1cd3312ba8364f866cb#diff-46e69f287173eef41fcbfeba05501954R274
This line ​https://github.com/django/django/blob/097457afe47e50e76d53b1cd3312ba8364f866cb/django/utils/autoreload.py#L274 absorbs all the exceptions not showing them to the developer.

[Tom Forbes]

I cannot seem to reproduce this with the latest master (bb9e82f2748ace292a584841ab9af8696df27f53), even with the same code as the description:

Exception in thread django-main-thread:
Traceback (most recent call last):
  ...
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/Users/tom/PycharmProjects/test_django_app/test_django_app/urls.py", line 6, in <module>
    from test_app import views
  File "/Users/tom/PycharmProjects/test_django_app/test_app/views.py", line 6, in <module>
    raise MyCustomExceptionShadowed('I have a description of your error')
test_app.views.MyCustomExceptionShadowed: I have a description of your error

There is this snippet of code in the autoreloader:

        try:
            get_resolver().urlconf_module
        except Exception:
            # Loading the urlconf can result in errors during development.
            # If this occurs then swallow the error and continue.
            pass

which catches the exception (added to fix #30323), but the exception should always be re-thrown by the system checks code in runserver(), or if that's somehow disabled/silenced then when the view is requested.

[Tom Forbes]

Also I cannot see how the given exception ("The included URLconf ...") could be caused by the autoreloader. It's triggered from this code:

        patterns = getattr(self.urlconf_module, "urlpatterns", self.urlconf_module)
        try:
            iter(patterns)
        except TypeError:
            msg = (
                "The included URLconf '{name}' does not appear to have any "
                "patterns in it. If you see valid patterns in the file then "
                "the issue is probably caused by a circular import."
            )
            raise ImproperlyConfigured(msg.format(name=self.urlconf_name))

If the urls.py code was evaluated at all then this code wouldn't be reached and the exception not thrown. So, if this is not something specific to the users environment then something very fishy is going on.

[Keryn Knight]

This looks like it could possibly be the same problem described in #30500 (and #30674). As with those, knowing the version of Python being used would be useful.

[Ariel Torti]

Replying to Tom Forbes:

Also I cannot see how the given exception ("The included URLconf ...") could be caused by the autoreloader. It's triggered from this code:

        patterns = getattr(self.urlconf_module, "urlpatterns", self.urlconf_module)
        try:
            iter(patterns)
        except TypeError:
            msg = (
                "The included URLconf '{name}' does not appear to have any "
                "patterns in it. If you see valid patterns in the file then "
                "the issue is probably caused by a circular import."
            )
            raise ImproperlyConfigured(msg.format(name=self.urlconf_name))

If the urls.py code was evaluated at all then this code wouldn't be reached and the exception not thrown. So, if this is not something specific to the users environment then something very fishy is going on.

Ok, so to make sure it was not my environment configuration getting on the way I tried to reproduce it on a docker container.
With Python 3.7.0 and Django 2.2.4 (the latest version from pip) it worked just right.

But then I tried running it with Python 3.5.2 (same version as my local machine where I have the issue) and I got the django.core.exceptions.ImproperlyConfigured instead of my exception. Apparently is a race condition (maybe?) because when I run it again it worked as expected, it failed only sometimes.

I will attach a file of the system configuration and the output showing it doesn't fail every time I run it. There's also the bundled project, can you try and check if you can reproduce it using Python 3.5.2

[Ariel Torti]

Bundled PoC project

[Ariel Torti]

System information and run output, you can see it appears the issue has something to do with race condition because it doesn't trigger everytime I run it.

[Ariel Torti]

Everything works just fine on Python 3.5.7, apparently it's a bug with python itself.
So the solution would be to update python, do I set the ticket as fixed ?

[Tom Forbes]

Let’s keep it open for now, thank you for your detailed and very helpful diagnostics

Likely the fix will be just “update Python”, but I want to do some digging around why exactly this happens and document it here.

Edit: and some people say they cannot update Python, so a workaround may be required

[Tom Forbes]

I'm going to close this in favour of #30500 which is the same issue, and has a lot more information there. I've posted my findings so far in https://code.djangoproject.com/ticket/30500#comment:14.