Opened 3 years ago

Closed 3 years ago

Last modified 3 years ago

#33107 closed Bug (fixed)

ImportError: partially initialized module '...' has no attribute '...'

Reported by: Collin Anderson Owned by: Mariusz Felisiak
Component: Utilities Version: 4.0
Severity: Release blocker Keywords:
Cc: Keryn Knight Triage Stage: Accepted
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

Today I've been getting some occasional "ImportError"s when reloading the runserver. I have a feeling it's a race condition when importing my custom auth backend when runserver is reloading, likely caused by #33099 cached imports, committed on Friday. It doesn't happen every time which makes it harder to track down.

Maybe the cached_import logic needs to somehow check for a "partially initialized module" and use the slow-path in that case?

I'm not 100% it's not just an issue on my end, but I at least wanted to raise this here in case other people have issues and want to diagnose further.

Here's an example (ImportError when importing my custom auth backend):

Request Method: GET
Request URL: /cart/dropdown/

Django Version: 4.0.dev20210913065016
Python Version: 3.8.10

Template error:
In template templates/order/cart_dropdown.html, error at line 3
   Module "authbackend" does not define a "Bknd" attribute/class
   3 :  {% if request.order.ops %}


Traceback (most recent call last):
  File "django/template/base.py", line 862, in _resolve_lookup
    current = current[bit]

During handling of the above exception ('WSGIRequest' object is not subscriptable), another exception occurred:
  File "django/utils/module_loading.py", line 26, in import_string
    return cached_import(module_path, class_name)
  File "django/utils/module_loading.py", line 12, in cached_import
    return getattr(modules[module_path], class_name)

The above exception (partially initialized module 'authbackend' has no attribute 'Bknd' (most likely due to a circular import)) was the direct cause of the following exception:
  File "django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File "django/core/handlers/base.py", line 181, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "django/shortcuts.py", line 19, in render
    content = loader.render_to_string(template_name, context, request, using=using)
  File "django/template/loader.py", line 62, in render_to_string
    return template.render(context, request)
  File "django/template/backends/django.py", line 61, in render
    return self.template.render(context)
  File "django/template/base.py", line 176, in render
    return self._render(context)
  File "django/template/base.py", line 168, in _render
    return self.nodelist.render(context)
  File "django/template/base.py", line 977, in render
    return SafeString(''.join([
  File "django/template/base.py", line 978, in <listcomp>
    node.render_annotated(context) for node in self
  File "django/template/base.py", line 938, in render_annotated
    return self.render(context)
  File "django/template/defaulttags.py", line 386, in render
    return strip_spaces_between_tags(self.nodelist.render(context).strip())
  File "django/template/base.py", line 977, in render
    return SafeString(''.join([
  File "django/template/base.py", line 978, in <listcomp>
    node.render_annotated(context) for node in self
  File "django/template/base.py", line 938, in render_annotated
    return self.render(context)
  File "django/template/defaulttags.py", line 288, in render
    match = condition.eval(context)
  File "django/template/defaulttags.py", line 829, in eval
    return self.value.resolve(context, ignore_failures=True)
  File "django/template/base.py", line 701, in resolve
    obj = self.var.resolve(context)
  File "django/template/base.py", line 829, in resolve
    value = self._resolve_lookup(context)
  File "django/template/base.py", line 870, in _resolve_lookup
    current = getattr(current, bit)
  File "order/middleware.py", line 20, in __get__
    if not request.user.pk:  # if not logged in
  File "django/utils/functional.py", line 248, in inner
    self._setup()
  File "django/utils/functional.py", line 384, in _setup
    self._wrapped = self._setupfunc()
  File "django/contrib/auth/middleware.py", line 25, in <lambda>
    request.user = SimpleLazyObject(lambda: get_user(request))
  File "django/contrib/auth/middleware.py", line 11, in get_user
    request._cached_user = auth.get_user(request)
  File "django/contrib/auth/__init__.py", line 183, in get_user
    backend = load_backend(backend_path)
  File "django/contrib/auth/__init__.py", line 21, in load_backend
    return import_string(path)()
  File "django/utils/module_loading.py", line 28, in import_string
    raise ImportError('Module "%s" does not define a "%s" attribute/class' % (

Exception Type: ImportError at /cart/dropdown/
Exception Value: Module "authbackend" does not define a "Bknd" attribute/class


Change History (10)

comment:1 by Collin Anderson, 3 years ago

Summary: ImportError: partially initialized module '...' has no attribute '...' (most likely due to a circular import)ImportError: partially initialized module '...' has no attribute '...'

comment:2 by Keryn Knight, 3 years ago

Cc: Keryn Knight added
Triage Stage: UnreviewedAccepted

Seems like a legitimate problem, that either requires working around (fastpath vs slowpath as mentioned) or reverting the change made in #33099.

It'd be interesting to see the middlewares & modules in play, even if they're reduced for privacy. My guess is circular imports are occurring possibly?

Version 0, edited 3 years ago by Keryn Knight (next)

comment:3 by Mariusz Felisiak, 3 years ago

Component: UncategorizedUtilities
Severity: NormalRelease blocker
Version: dev4.0

comment:4 by Mariusz Felisiak, 3 years ago

The easiest (but hacky) way to fix it is to check Python's internals and reload the module again:

diff --git a/django/utils/module_loading.py b/django/utils/module_loading.py
index 1df82b1c32..39cb784f72 100644
--- a/django/utils/module_loading.py
+++ b/django/utils/module_loading.py
@@ -7,7 +7,11 @@ from importlib.util import find_spec as importlib_find
 
 def cached_import(module_path, class_name):
     modules = sys.modules
-    if module_path not in modules:
+    if module_path not in modules or (
+        # Module it not fully initialize.
+        getattr(modules[module_path], '__spec__', None) is not None and
+        getattr(modules[module_path].__spec__, '_initializing', False) is True
+    ):
         import_module(module_path)
     return getattr(modules[module_path], class_name)
 

comment:6 by Keryn Knight, 3 years ago

Serendipitous find :) For historical reference, it looks like _initializing is set via importlib._bootstrap._load_unlocked:

# This must be done before putting the module in sys.modules
# (otherwise an optimization shortcut in import.c becomes
# wrong).
spec._initializing = True

Possibly worth noting however is importlib.util._module_to_load which has equivalent but different (at least in 3.9.5):

# This must be done before putting the module in sys.modules
# (otherwise an optimization shortcut in import.c becomes wrong)
module.__initializing__ = True

If we're going to try and keep the cached_import, perhaps we need to accommodate both, or at least investigate the __initializing__ variation?

FWIW, the investigation/patch proposed above does change the performance profile (it's 5.64 µs for me now) but is still world's better than repeatedly doing import_module (at least until Python 3.11 which might improve things via the linked bpo-43392)

comment:7 by Mariusz Felisiak, 3 years ago

Has patch: set
Owner: changed from nobody to Mariusz Felisiak
Status: newassigned

PR

Possibly worth noting however is importlib.util._module_to_load which has equivalent but different (at least in 3.9.5):

As far as I'm aware we shouldn't reach _module_to_load() using import_module(). Also Python uses the same strategy (checking __spec__._initialized) in few places, e.g. import_ensure_initialized().

comment:8 by Collin Anderson, 3 years ago

By the way, I figured out I can just put time.sleep(5) at the top-level of my custom auth backend to reproduce the issue the patch does fix the issue. Thanks!

comment:9 by GitHub <noreply@…>, 3 years ago

Resolution: fixed
Status: assignedclosed

In 6426c307:

Fixed #33107 -- Fixed import_string() crash on not fully initialized modules.

Regression in ecf87ad513fd8af6e4a6093ed918723a7d88d5ca.

Thanks Collin Anderson for the report.

comment:10 by GitHub <noreply@…>, 3 years ago

In a3185a6:

Refs #33107 -- Optimized cached_import() helper.

Note: See TracTickets for help on using tickets.
Back to Top