module_has_submodule() doesn't work correctly if the module_name argument is a dotted path

[tkhyn]

Hello,

Porting a django project from python 2.7 to 3.6, I noticed an issue with utils.module_loading.module_has_submodule. At some stage my project makes use of autodiscover_modules('module.submodule') to try and discover modules that are nested in my app.

I'm using django 1.11.1, but this bug probably also affects 1.8 as the code of module_has_submodule is the same.

So here we go, with python 2.7 we have the expected behavior (taking contenttypes as an example app):

python2.7
>>> from django.utils.module_loading import module_has_submodule
>>> from django.contrib import contenttypes
>>> module_has_submodule(contenttypes, 'invalid_module.submodule')
False
>>> module_has_submodule(contenttypes, 'checks')
True
>>> module_has_submodule(contenttypes, 'checks.submodule')
False

But, with python 3.6

python3.6
>>> from django.utils.module_loading import module_has_submodule
>>> from django.contrib import contenttypes
>>> module_has_submodule(contenttypes, 'invalid_module.submodule')
  File "<console>", line 1, in <module>
  File "d:\dev\.env\buildout\eggs\django-1.11.1-py3.6.egg\django\utils\module_loading.py", line 79, in module_has_submodule
    return importlib_find(full_module_name, package_path) is not None
  File "D:\dev\.env\venv\buildout\lib\importlib\util.py", line 88, in find_spec
    parent = __import__(parent_name, fromlist=['__path__'])
ModuleNotFoundError: No module named 'django.contrib.contenttypes.invalid_module'
>>> module_has_submodule(contenttypes, 'checks')
True
>>> module_has_submodule(contenttypes, 'checks.submodule')
  File "<console>", line 1, in <module>
  File "d:\dev\.env\buildout\eggs\django-1.11.1-py3.6.egg\django\utils\module_loading.py", line 79, in module_has_submodule
    return importlib_find(full_module_name, package_path) is not None
  File "D:\dev\.env\venv\buildout\lib\importlib\util.py", line 89, in find_spec
    return _find_spec(fullname, parent.__path__)
AttributeError: module 'django.contrib.contenttypes.checks' has no attribute '__path__'

From the replies I got on the python bug tracker (​http://bugs.python.org/issue30436) the AttributeError will be converted to ModuleNotFoundError only in Python 3.7, but that behavior is apparently not expected to be fixed in previous versions.

We'll definitely need to catch ModuleNotFoundError, and to have a proper fix for python < 3.7 we'll need to:

• either catch AttributeError as well. The problem I see is AttributeError is too broad and may result in 'false catches'
• or convert a dotted path to [package, name] (basically do what find_spec actually does) but detect if the module has a __path__ attribute before calling find_spec so that it can not raise exceptions

What are your thoughts ?

[Tim Graham]

I don't think the module_name argument of module_has_submodule() is expected to be a dotted path. For example, this test (incorrectly, as far as I see) passes on Python 2:

tests/utils_tests/test_module_loading.py

@@ -26 +26 @@
         test_no_submodule = import_module(
             'utils_tests.test_no_submodule')
 
+        self.assertTrue(module_has_submodule(test_module, 'invalid.good_module'))
+
         # An importable child
         self.assertTrue(module_has_submodule(test_module, 'good_module'))
         mod = import_module('utils_tests.test_module.good_module')

Is Django making any calls like that or is this function used like this outside of Django?

[tkhyn]

That's what I said, it works on Python 2, not on Python 3. (sorry, I misread, replied too quickly)

So you're saying that module_has_submodule should not accept dotted paths anyway. As a consequence, autodiscover_modules should not either ... However the dotted path feature I used and that actually worked ok for me until then with python 2.7 would not be hard to support.

The real-life problem I have with that is I need to use autodiscover_modules('path_to.my_module'), which calls module_has_submodule in every app. In Python 2 it attempts to discover this module in all apps (and consistently returns True or False). In Python 3 it raises an exception the first time find_spec fails, preventing loading any module in subsequent apps.

I know one solution is to re-organize my apps structure to put my_module at the apps' root level, but:

• that does mess up with the organization of my code
• this used to work in Python 2 and could easily be made to work in Python 3, and I would like to avoid having to monkey-patch django just to add a try / catch block ...

[tkhyn]

In addition, trying to reproduce your example with python 2.7.13 I get:

>>> from importlib import import_module
>>> from django.utils.module_loading import module_has_submodule
>>> test_module = import_module('utils_tests.test_module')
>>> module_has_submodule(test_module, 'invalid.good_module')
False

[Tim Graham]

I don't know if module_has_submodule() should not accept dotted paths, however, as far as I can tell, the behavior is undocumented and untested. We could either fix it or raise a more helpful error message.

[tkhyn]

Great, I'll submit a PR with testcases + solution for all currently supported versions.

[tkhyn]

Here is a patch: ​PR 8611

[Tim Graham <timograham@…>]

In f6bd0013:

Fixed #28241 -- Allowed module_has_submodule()'s module_name arg to be a dotted path.