Opened 2 years ago

Last modified 21 months ago

#25624 new Bug

Autoreload fails if jinja2.ModuleLoader used

Reported by: SvartalF Owned by: nobody
Component: Utilities Version: 1.8
Severity: Normal Keywords: autoreload, jinja2, ModuleLoader
Cc: Triage Stage: Accepted
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

It happens because jinja2.ModuleLoader injects a weakref to a fake module into a sys.modules: https://github.com/mitsuhiko/jinja2/blob/f6b654de615de61a9ca79e7ccecf7b4a8bf90ec0/jinja2/loaders.py#L439-L449

So, because weakref is not a hashable object, autoreload fails.

Traceback (most recent call last):
  File "bug/manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/home/svartalf/projects/dj-module-loader-bug/.env/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 351, in execute_from_command_line
    utility.execute()
  File "/home/svartalf/projects/dj-module-loader-bug/.env/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 343, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/svartalf/projects/dj-module-loader-bug/.env/local/lib/python2.7/site-packages/django/core/management/base.py", line 394, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/home/svartalf/projects/dj-module-loader-bug/.env/local/lib/python2.7/site-packages/django/core/management/commands/runserver.py", line 49, in execute
    super(Command, self).execute(*args, **options)
  File "/home/svartalf/projects/dj-module-loader-bug/.env/local/lib/python2.7/site-packages/django/core/management/base.py", line 445, in execute
    output = self.handle(*args, **options)
  File "/home/svartalf/projects/dj-module-loader-bug/.env/local/lib/python2.7/site-packages/django/core/management/commands/runserver.py", line 88, in handle
    self.run(**options)
  File "/home/svartalf/projects/dj-module-loader-bug/.env/local/lib/python2.7/site-packages/django/core/management/commands/runserver.py", line 97, in run
    autoreload.main(self.inner_run, None, options)
  File "/home/svartalf/projects/dj-module-loader-bug/.env/local/lib/python2.7/site-packages/django/utils/autoreload.py", line 336, in main
    reloader(wrapped_main_func, args, kwargs)
  File "/home/svartalf/projects/dj-module-loader-bug/.env/local/lib/python2.7/site-packages/django/utils/autoreload.py", line 302, in python_reloader
    reloader_thread()
  File "/home/svartalf/projects/dj-module-loader-bug/.env/local/lib/python2.7/site-packages/django/utils/autoreload.py", line 278, in reloader_thread
    change = fn()
  File "/home/svartalf/projects/dj-module-loader-bug/.env/local/lib/python2.7/site-packages/django/utils/autoreload.py", line 207, in code_changed
    for filename in gen_filenames():
  File "/home/svartalf/projects/dj-module-loader-bug/.env/local/lib/python2.7/site-packages/django/utils/autoreload.py", line 94, in gen_filenames
    module_values = set(sys.modules.values())
TypeError: unhashable type: 'weakproxy'

I had reproduced this bug with a clean project and following environment:

Python 2.7.6

$ pip freeze
Django==1.8.5
Jinja2==2.7.3
MarkupSafe==0.23
argparse==1.2.1
wsgiref==0.1.2

Template settings:

template_loader = jinja2.ModuleLoader('compiled_templates')
TEMPLATES = [
    {
        'BACKEND': 'django.template.backends.jinja2.Jinja2',
        'DIRS': [os.path.join(BASE_DIR, 'templates')],
        'APP_DIRS': True,
        'OPTIONS': {
            'loader': template_loader,
        },
    },
]

P.S. Not sure if it is an "Utilities" or a "Template system" component.

Attachments (1)

25624.diff (1.2 KB) - added by Paul 2 years ago.

Download all attachments as: .zip

Change History (7)

comment:1 Changed 2 years ago by Tim Graham

Triage Stage: UnreviewedAccepted

Changed 2 years ago by Paul

Attachment: 25624.diff added

comment:2 Changed 2 years ago by Paul

This patch fixes the issue by ignoring modules without a __file__ attribute altogether. There are probably cases where this is not quite desired behaviour, but I have no idea how to test this.

The other two options that I can think of to fix this, checking for weakref.ProxyType or the _jinja2_module_templates_ module name prefix, don't really seem like the right solution either...

comment:3 Changed 2 years ago by Carl Meyer

@hackerdd That fix looks fine to me. I don't think it should actually change the behavior, since modules without __file__ were excluded later on anyway. There's nothing we can do in the autoreloader with a module that doesn't have a __file__.

I think this should be testable without too much difficulty, just by inserting a problematic entry into sys.modules (in the test - and make sure it gets removed after the test, using a finally block) and then calling the gen_filenames function.

comment:4 Changed 2 years ago by Aymeric Augustin

Indeed there are a few tests for gen_filenames() in tests/utils_tests/test_autoreload.py. Could you try to add one as Carl says?

comment:5 Changed 21 months ago by mscout1

What version of django will the officially be fixed in? I ask because I have 1.8.9 and it still has this problem, and there is no 'target version' or 'fix version' on this site.

(I apologize if this is not an appropriate place to ask this question, but it is the only place I can find any information at all about this problem. )

comment:6 Changed 21 months ago by Tim Graham

There isn't a targeted version for the fix. It will be fixed when we have a patch (with a test) to review.

Note: See TracTickets for help on using tickets.
Back to Top