Autoreload fails if jinja2.ModuleLoader used

[SvartalF]

It happens because jinja2.ModuleLoader injects a weakref to a fake module into a sys.modules: ​https://github.com/mitsuhiko/jinja2/blob/f6b654de615de61a9ca79e7ccecf7b4a8bf90ec0/jinja2/loaders.py#L439-L449

So, because weakref is not a hashable object, autoreload fails.

Traceback (most recent call last):
  File "bug/manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/home/svartalf/projects/dj-module-loader-bug/.env/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 351, in execute_from_command_line
    utility.execute()
  File "/home/svartalf/projects/dj-module-loader-bug/.env/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 343, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/svartalf/projects/dj-module-loader-bug/.env/local/lib/python2.7/site-packages/django/core/management/base.py", line 394, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/home/svartalf/projects/dj-module-loader-bug/.env/local/lib/python2.7/site-packages/django/core/management/commands/runserver.py", line 49, in execute
    super(Command, self).execute(*args, **options)
  File "/home/svartalf/projects/dj-module-loader-bug/.env/local/lib/python2.7/site-packages/django/core/management/base.py", line 445, in execute
    output = self.handle(*args, **options)
  File "/home/svartalf/projects/dj-module-loader-bug/.env/local/lib/python2.7/site-packages/django/core/management/commands/runserver.py", line 88, in handle
    self.run(**options)
  File "/home/svartalf/projects/dj-module-loader-bug/.env/local/lib/python2.7/site-packages/django/core/management/commands/runserver.py", line 97, in run
    autoreload.main(self.inner_run, None, options)
  File "/home/svartalf/projects/dj-module-loader-bug/.env/local/lib/python2.7/site-packages/django/utils/autoreload.py", line 336, in main
    reloader(wrapped_main_func, args, kwargs)
  File "/home/svartalf/projects/dj-module-loader-bug/.env/local/lib/python2.7/site-packages/django/utils/autoreload.py", line 302, in python_reloader
    reloader_thread()
  File "/home/svartalf/projects/dj-module-loader-bug/.env/local/lib/python2.7/site-packages/django/utils/autoreload.py", line 278, in reloader_thread
    change = fn()
  File "/home/svartalf/projects/dj-module-loader-bug/.env/local/lib/python2.7/site-packages/django/utils/autoreload.py", line 207, in code_changed
    for filename in gen_filenames():
  File "/home/svartalf/projects/dj-module-loader-bug/.env/local/lib/python2.7/site-packages/django/utils/autoreload.py", line 94, in gen_filenames
    module_values = set(sys.modules.values())
TypeError: unhashable type: 'weakproxy'

I had reproduced this bug with a clean project and following environment:

Python 2.7.6

$ pip freeze
Django==1.8.5
Jinja2==2.7.3
MarkupSafe==0.23
argparse==1.2.1
wsgiref==0.1.2

Template settings:

template_loader = jinja2.ModuleLoader('compiled_templates')
TEMPLATES = [
    {
        'BACKEND': 'django.template.backends.jinja2.Jinja2',
        'DIRS': [os.path.join(BASE_DIR, 'templates')],
        'APP_DIRS': True,
        'OPTIONS': {
            'loader': template_loader,
        },
    },
]

P.S. Not sure if it is an "Utilities" or a "Template system" component.

[Paul]

This patch fixes the issue by ignoring modules without a __file__ attribute altogether. There are probably cases where this is not quite desired behaviour, but I have no idea how to test this.

The other two options that I can think of to fix this, checking for weakref.ProxyType or the _jinja2_module_templates_ module name prefix, don't really seem like the right solution either...

[Carl Meyer]

@hackerdd That fix looks fine to me. I don't think it should actually change the behavior, since modules without __file__ were excluded later on anyway. There's nothing we can do in the autoreloader with a module that doesn't have a __file__.

I think this should be testable without too much difficulty, just by inserting a problematic entry into sys.modules (in the test - and make sure it gets removed after the test, using a finally block) and then calling the gen_filenames function.

[Aymeric Augustin]

Indeed there are a few tests for gen_filenames() in tests/utils_tests/test_autoreload.py. Could you try to add one as Carl says?

[Matthew Scouten]

What version of django will the officially be fixed in? I ask because I have 1.8.9 and it still has this problem, and there is no 'target version' or 'fix version' on this site.

(I apologize if this is not an appropriate place to ask this question, but it is the only place I can find any information at all about this problem. )

[Tim Graham]

There isn't a targeted version for the fix. It will be fixed when we have a patch (with a test) to review.

[Tom Forbes]

Patch: ​https://github.com/django/django/pull/10870

[Tim Graham <timograham@…>]

In 1e92407:

Fixed #25624 -- Fixed autoreload crash with jinja2.ModuleLoader.

[Tim Graham <timograham@…>]

In 728358c5:

[2.2.x] Fixed #25624 -- Fixed autoreload crash with jinja2.ModuleLoader.

Backport of 1e92407f83ed35be35f876777935b983ab9587be from master.