module_has_submodule() doesn't work correctly if the module_name argument is a dotted path

[tkhyn]

Hello,

Porting a django project from python 2.7 to 3.6, I noticed an issue with utils.module_loading.module_has_submodule. At some stage my project makes use of autodiscover_modules('module.submodule') to try and discover modules that are nested in my app.

I'm using django 1.11.1, but this bug probably also affects 1.8 as the code of module_has_submodule is the same.

So here we go, with python 2.7 we have the expected behavior (taking contenttypes as an example app):

python2.7
>>> from django.utils.module_loading import module_has_submodule
>>> from django.contrib import contenttypes
>>> module_has_submodule(contenttypes, 'invalid_module.submodule')
False
>>> module_has_submodule(contenttypes, 'checks')
True
>>> module_has_submodule(contenttypes, 'checks.submodule')
False

But, with python 3.6

python3.6
>>> from django.utils.module_loading import module_has_submodule
>>> from django.contrib import contenttypes
>>> module_has_submodule(contenttypes, 'invalid_module.submodule')
  File "<console>", line 1, in <module>
  File "d:\dev\.env\buildout\eggs\django-1.11.1-py3.6.egg\django\utils\module_loading.py", line 79, in module_has_submodule
    return importlib_find(full_module_name, package_path) is not None
  File "D:\dev\.env\venv\buildout\lib\importlib\util.py", line 88, in find_spec
    parent = __import__(parent_name, fromlist=['__path__'])
ModuleNotFoundError: No module named 'django.contrib.contenttypes.invalid_module'
>>> module_has_submodule(contenttypes, 'checks')
True
>>> module_has_submodule(contenttypes, 'checks.submodule')
  File "<console>", line 1, in <module>
  File "d:\dev\.env\buildout\eggs\django-1.11.1-py3.6.egg\django\utils\module_loading.py", line 79, in module_has_submodule
    return importlib_find(full_module_name, package_path) is not None
  File "D:\dev\.env\venv\buildout\lib\importlib\util.py", line 89, in find_spec
    return _find_spec(fullname, parent.__path__)
AttributeError: module 'django.contrib.contenttypes.checks' has no attribute '__path__'

From the replies I got on the python bug tracker (​http://bugs.python.org/issue30436) the AttributeError will be converted to ModuleNotFoundError only in Python 3.7, but that behavior is apparently not expected to be fixed in previous versions.

We'll definitely need to catch ModuleNotFoundError, and to have a proper fix for python < 3.7 we'll need to:

• either catch AttributeError as well. The problem I see is AttributeError is too broad and may result in 'false catches'
• or convert a dotted path to [package, name] (basically do what find_spec actually does) but detect if the module has a __path__ attribute before calling find_spec so that it can not raise exceptions

What are your thoughts ?

[Tim Graham]

I don't think the module_name argument of module_has_submodule() is expected to be a dotted path. For example, this test (incorrectly, as far as I see) passes on Python 2:

tests/utils_tests/test_module_loading.py

@@ -26 +26 @@
         test_no_submodule = import_module(
             'utils_tests.test_no_submodule')
 
+        self.assertTrue(module_has_submodule(test_module, 'invalid.good_module'))
+
         # An importable child
         self.assertTrue(module_has_submodule(test_module, 'good_module'))
         mod = import_module('utils_tests.test_module.good_module')

Is Django making any calls like that or is this function used like this outside of Django?