Opened 5 years ago

Closed 5 years ago

#30728 closed Bug (duplicate)

Django urls patterns resolver shadowing exceptions thrown everywhere else

Reported by: Ariel Torti Owned by: nobody
Component: Utilities Version: 2.2
Severity: Release blocker Keywords:
Cc: Tom Forbes Triage Stage: Accepted
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description (last modified by Ariel Torti)

The new version of django somehow captures all the exceptions ocurred while parsing the urlpatterns.

How to reproduce:

  • Create a django project
  • Create a an app

app/views.py

from django.shortcuts import render

class MyCustomExceptionShadowed(Exception):
  pass

raise MyCustomExceptionShadowed('I have a description of your error')

# Create your views here.
def myview(request):
  pass

urls.py

from django.contrib import admin
from django.urls import path

from bugtest import views

urlpatterns = [
    path('', views.myview),
    path('admin/', admin.site.urls),
]

When running the server I get

django.core.exceptions.ImproperlyConfigured: The included URLconf 'bug.urls' does not appear to have any patterns in it. If you see valid patterns in the file then the issue is probably caused by a circular import.

When in fact I expect it to say

  File "/home/ariel-nt/Desktop/test/bug/bugtest/views.py", line 6, in <module>
    raise MyCustomExceptionShadowed('I have a description of your error')
bugtest.views.MyCustomExceptionShadowed: I have a description of your error

I tested this with django 2.1 and it works as expected, so I'll now try to find the root cause of the error and give an update here
The issue in introduced in this commit https://github.com/django/django/commit/097457afe47e50e76d53b1cd3312ba8364f866cb#diff-46e69f287173eef41fcbfeba05501954R274
This line https://github.com/django/django/blob/097457afe47e50e76d53b1cd3312ba8364f866cb/django/utils/autoreload.py#L274 absorbs all the exceptions not showing them to the developer.

Attachments (2)

bug.zip (5.0 KB ) - added by Ariel Torti 5 years ago.
Bundled PoC project
output.log (14.7 KB ) - added by Ariel Torti 5 years ago.
System information and run output, you can see it appears the issue has something to do with race condition because it doesn't trigger everytime I run it.

Download all attachments as: .zip

Change History (13)

comment:1 by Ariel Torti, 5 years ago

Description: modified (diff)

comment:2 by Ariel Torti, 5 years ago

Component: Core (URLs)Utilities

comment:3 by Claude Paroz, 5 years ago

Severity: NormalRelease blocker
Triage Stage: UnreviewedAccepted

comment:4 by Mariusz Felisiak, 5 years ago

Cc: Tom Forbes added

comment:5 by Tom Forbes, 5 years ago

I cannot seem to reproduce this with the latest master (bb9e82f2748ace292a584841ab9af8696df27f53), even with the same code as the description:

Exception in thread django-main-thread:
Traceback (most recent call last):
  ...
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/Users/tom/PycharmProjects/test_django_app/test_django_app/urls.py", line 6, in <module>
    from test_app import views
  File "/Users/tom/PycharmProjects/test_django_app/test_app/views.py", line 6, in <module>
    raise MyCustomExceptionShadowed('I have a description of your error')
test_app.views.MyCustomExceptionShadowed: I have a description of your error

There is this snippet of code in the autoreloader:

        try:
            get_resolver().urlconf_module
        except Exception:
            # Loading the urlconf can result in errors during development.
            # If this occurs then swallow the error and continue.
            pass

which catches the exception (added to fix #30323), but the exception should always be re-thrown by the system checks code in runserver(), or if that's somehow disabled/silenced then when the view is requested.

comment:6 by Tom Forbes, 5 years ago

Also I cannot see how the given exception ("The included URLconf ...") could be caused by the autoreloader. It's triggered from this code:

        patterns = getattr(self.urlconf_module, "urlpatterns", self.urlconf_module)
        try:
            iter(patterns)
        except TypeError:
            msg = (
                "The included URLconf '{name}' does not appear to have any "
                "patterns in it. If you see valid patterns in the file then "
                "the issue is probably caused by a circular import."
            )
            raise ImproperlyConfigured(msg.format(name=self.urlconf_name))

If the urls.py code was evaluated at all then this code wouldn't be reached and the exception not thrown. So, if this is not something specific to the users environment then something very fishy is going on.

comment:7 by Keryn Knight, 5 years ago

This looks like it could possibly be the same problem described in #30500 (and #30674). As with those, knowing the version of Python being used would be useful.

in reply to:  6 comment:8 by Ariel Torti, 5 years ago

Replying to Tom Forbes:

Also I cannot see how the given exception ("The included URLconf ...") could be caused by the autoreloader. It's triggered from this code:

        patterns = getattr(self.urlconf_module, "urlpatterns", self.urlconf_module)
        try:
            iter(patterns)
        except TypeError:
            msg = (
                "The included URLconf '{name}' does not appear to have any "
                "patterns in it. If you see valid patterns in the file then "
                "the issue is probably caused by a circular import."
            )
            raise ImproperlyConfigured(msg.format(name=self.urlconf_name))

If the urls.py code was evaluated at all then this code wouldn't be reached and the exception not thrown. So, if this is not something specific to the users environment then something very fishy is going on.

Ok, so to make sure it was not my environment configuration getting on the way I tried to reproduce it on a docker container.
With Python 3.7.0 and Django 2.2.4 (the latest version from pip) it worked just right.

But then I tried running it with Python 3.5.2 (same version as my local machine where I have the issue) and I got the django.core.exceptions.ImproperlyConfigured instead of my exception. Apparently is a race condition (maybe?) because when I run it again it worked as expected, it failed only sometimes.

I will attach a file of the system configuration and the output showing it doesn't fail every time I run it. There's also the bundled project, can you try and check if you can reproduce it using Python 3.5.2

by Ariel Torti, 5 years ago

Attachment: bug.zip added

Bundled PoC project

by Ariel Torti, 5 years ago

Attachment: output.log added

System information and run output, you can see it appears the issue has something to do with race condition because it doesn't trigger everytime I run it.

comment:9 by Ariel Torti, 5 years ago

Everything works just fine on Python 3.5.7, apparently it's a bug with python itself.
So the solution would be to update python, do I set the ticket as fixed ?

comment:10 by Tom Forbes, 5 years ago

Let’s keep it open for now, thank you for your detailed and very helpful diagnostics

Likely the fix will be just “update Python”, but I want to do some digging around why exactly this happens and document it here.

Edit: and some people say they cannot update Python, so a workaround may be required

Last edited 5 years ago by Tom Forbes (previous) (diff)

comment:11 by Tom Forbes, 5 years ago

Resolution: duplicate
Status: newclosed

I'm going to close this in favour of #30500 which is the same issue, and has a lot more information there. I've posted my findings so far in https://code.djangoproject.com/ticket/30500#comment:14.

Note: See TracTickets for help on using tickets.
Back to Top