Opened 8 years ago

Closed 5 years ago

Last modified 5 years ago

#25624 closed Bug (fixed)

Autoreload fails if jinja2.ModuleLoader used

Reported by: SvartalF Owned by: Tom Forbes
Component: Utilities Version: 1.8
Severity: Normal Keywords: autoreload, jinja2, ModuleLoader
Cc: Triage Stage: Accepted
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

It happens because jinja2.ModuleLoader injects a weakref to a fake module into a sys.modules: https://github.com/mitsuhiko/jinja2/blob/f6b654de615de61a9ca79e7ccecf7b4a8bf90ec0/jinja2/loaders.py#L439-L449

So, because weakref is not a hashable object, autoreload fails.

Traceback (most recent call last):
  File "bug/manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/home/svartalf/projects/dj-module-loader-bug/.env/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 351, in execute_from_command_line
    utility.execute()
  File "/home/svartalf/projects/dj-module-loader-bug/.env/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 343, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/svartalf/projects/dj-module-loader-bug/.env/local/lib/python2.7/site-packages/django/core/management/base.py", line 394, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/home/svartalf/projects/dj-module-loader-bug/.env/local/lib/python2.7/site-packages/django/core/management/commands/runserver.py", line 49, in execute
    super(Command, self).execute(*args, **options)
  File "/home/svartalf/projects/dj-module-loader-bug/.env/local/lib/python2.7/site-packages/django/core/management/base.py", line 445, in execute
    output = self.handle(*args, **options)
  File "/home/svartalf/projects/dj-module-loader-bug/.env/local/lib/python2.7/site-packages/django/core/management/commands/runserver.py", line 88, in handle
    self.run(**options)
  File "/home/svartalf/projects/dj-module-loader-bug/.env/local/lib/python2.7/site-packages/django/core/management/commands/runserver.py", line 97, in run
    autoreload.main(self.inner_run, None, options)
  File "/home/svartalf/projects/dj-module-loader-bug/.env/local/lib/python2.7/site-packages/django/utils/autoreload.py", line 336, in main
    reloader(wrapped_main_func, args, kwargs)
  File "/home/svartalf/projects/dj-module-loader-bug/.env/local/lib/python2.7/site-packages/django/utils/autoreload.py", line 302, in python_reloader
    reloader_thread()
  File "/home/svartalf/projects/dj-module-loader-bug/.env/local/lib/python2.7/site-packages/django/utils/autoreload.py", line 278, in reloader_thread
    change = fn()
  File "/home/svartalf/projects/dj-module-loader-bug/.env/local/lib/python2.7/site-packages/django/utils/autoreload.py", line 207, in code_changed
    for filename in gen_filenames():
  File "/home/svartalf/projects/dj-module-loader-bug/.env/local/lib/python2.7/site-packages/django/utils/autoreload.py", line 94, in gen_filenames
    module_values = set(sys.modules.values())
TypeError: unhashable type: 'weakproxy'

I had reproduced this bug with a clean project and following environment:

Python 2.7.6

$ pip freeze
Django==1.8.5
Jinja2==2.7.3
MarkupSafe==0.23
argparse==1.2.1
wsgiref==0.1.2

Template settings:

template_loader = jinja2.ModuleLoader('compiled_templates')
TEMPLATES = [
    {
        'BACKEND': 'django.template.backends.jinja2.Jinja2',
        'DIRS': [os.path.join(BASE_DIR, 'templates')],
        'APP_DIRS': True,
        'OPTIONS': {
            'loader': template_loader,
        },
    },
]

P.S. Not sure if it is an "Utilities" or a "Template system" component.

Attachments (1)

25624.diff (1.2 KB ) - added by Paul 8 years ago.

Download all attachments as: .zip

Change History (11)

comment:1 by Tim Graham, 8 years ago

Triage Stage: UnreviewedAccepted

by Paul, 8 years ago

Attachment: 25624.diff added

comment:2 by Paul, 8 years ago

This patch fixes the issue by ignoring modules without a __file__ attribute altogether. There are probably cases where this is not quite desired behaviour, but I have no idea how to test this.

The other two options that I can think of to fix this, checking for weakref.ProxyType or the _jinja2_module_templates_ module name prefix, don't really seem like the right solution either...

comment:3 by Carl Meyer, 8 years ago

@hackerdd That fix looks fine to me. I don't think it should actually change the behavior, since modules without __file__ were excluded later on anyway. There's nothing we can do in the autoreloader with a module that doesn't have a __file__.

I think this should be testable without too much difficulty, just by inserting a problematic entry into sys.modules (in the test - and make sure it gets removed after the test, using a finally block) and then calling the gen_filenames function.

comment:4 by Aymeric Augustin, 8 years ago

Indeed there are a few tests for gen_filenames() in tests/utils_tests/test_autoreload.py. Could you try to add one as Carl says?

comment:5 by Matthew Scouten, 8 years ago

What version of django will the officially be fixed in? I ask because I have 1.8.9 and it still has this problem, and there is no 'target version' or 'fix version' on this site.

(I apologize if this is not an appropriate place to ask this question, but it is the only place I can find any information at all about this problem. )

comment:6 by Tim Graham, 8 years ago

There isn't a targeted version for the fix. It will be fixed when we have a patch (with a test) to review.

comment:7 by Tom Forbes, 5 years ago

Owner: changed from nobody to Tom Forbes
Status: newassigned

comment:8 by Tom Forbes, 5 years ago

Has patch: set
Version 0, edited 5 years ago by Tom Forbes (next)

comment:9 by Tim Graham <timograham@…>, 5 years ago

Resolution: fixed
Status: assignedclosed

In 1e92407:

Fixed #25624 -- Fixed autoreload crash with jinja2.ModuleLoader.

comment:10 by Tim Graham <timograham@…>, 5 years ago

In 728358c5:

[2.2.x] Fixed #25624 -- Fixed autoreload crash with jinja2.ModuleLoader.

Backport of 1e92407f83ed35be35f876777935b983ab9587be from master.

Note: See TracTickets for help on using tickets.
Back to Top