Opened 2 years ago

Closed 2 years ago

Last modified 2 years ago

#32302 closed Cleanup/optimization (fixed)

Permit migrations in non-namespace packages that don't have __file__

Reported by: William Schwartz Owned by: William Schwartz
Component: Migrations Version: dev
Severity: Normal Keywords: migrations freezers
Cc: Triage Stage: Ready for checkin
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no



This feature request, for which I will post a PR shortly, aims to improve the specificity of the migration loader's check for and rejection of PEP-420 namespace packages. I am NOT asking to allow namespace packages for apps' migrations. I merely want to make the existing check more compliant with Python's documented import API. This would remove one impediment to using Django in so-called frozen Python environments (such as those mentioned in #30950) that do not set __file__ on regular packages by default.

This narrow proposal does not change Django's behavior at all for normal Python environments. The only change for frozen environments is that Django will learn how to find existing migrations. In particular, at this time I am not proposing to enable any other Django feature that does not already work in frozen environments.

I would love for this feature to land in Django 3.2.


I initially broached this idea on the django-developers mailing list. This is my second ticket related to frozen Python environments, the first being #32177.

The current implementation of the migration loader's no-namespace-package check in django.db.migrations.loader.MigrationLoader.load_disk skips searching for migrations in a module m if getattr(m, '__file__', None) is false.

The trouble with this implementation is that namespace packages are not the only modules with no __file__. Indeed, the Python documentation states that

__file__ is optional. If set, this attribute's value must be a string. The import system may opt to leave __file__ unset if it has no semantic meaning (e.g. a module loaded from a database).

However, Python's documentation also states

Namespace packages do not use an ordinary list for their __path__ attribute. They instead use a custom iterable type....

The class of namespace packages' __path__ in CPython is _NamespacePath, but that is a CPython implementation detail. Instead, I propose to augment getattr(m, '__file__', None) with and isinstance(m.__path__, list).

Change History (5)

comment:1 Changed 2 years ago by William Schwartz

Owner: changed from nobody to William Schwartz
Status: newassigned

comment:2 Changed 2 years ago by Mariusz Felisiak

Keywords: freezers added
Triage Stage: UnreviewedAccepted
Type: New featureCleanup/optimization

Thanks for the rationale. Sounds reasonable.

comment:3 Changed 2 years ago by Mariusz Felisiak

Triage Stage: AcceptedReady for checkin

comment:4 Changed 2 years ago by Mariusz Felisiak <felisiak.mariusz@…>

Resolution: fixed
Status: assignedclosed

In e64c1d80:

Fixed #32302 -- Allowed migrations to be loaded from regular packages with no file attribute.

The migrations loader prevents the use of PEP-420 namespace packages
for holding apps' migrations modules. Previously the loader tested for
this only by checking that app.migrations.file is present. This
prevented migrations' being found in frozen Python environments that
don't set file on any modules. Now the loader *additionally* checks
whether app.migrations.path is a list because namespace packages
use a different type for path. Namespace packages continue to be
forbidden, and, in fact, users of normal Python environments should
experience no change whatsoever.

comment:5 Changed 2 years ago by William Schwartz

Appendix: Research Notes

This ticket is closed. You need read no further unless you're researching a different issue related to the Python import system or Django's migration loading.

While doing my research this ticket, I took a lot of notes that didn't make it into the actual report. Here they are, in case it helps someone else down the line.

More background on __file__

__file__ is set by the import machinery from importlib.machinery.ModuleSpec.origin, which may be None:

Name of the place from which the module is loaded, e.g. “builtin” for built-in modules and the filename for modules loaded from source. Normally “origin” should be set, but it may be None (the default) which indicates it is unspecified (e.g. for namespace packages).

One frozen Python environment, PyOxidizer, relies on this API rule.

Proving a module is a namespace package

Since Python 3.7 fixed bpo-32303, it is no longer possible to prove definitively that a module is a namespace package. In Python 3.6 and earlier, a module's __loader__ was None if and only if the module was a namespace package. This was because PEP 451 required that a finder's find_spec set loader to None:

Currently a path entry finder may return (None, portions) from find_loader() to indicate it found part of a possible namespace package. To achieve the same effect, find_spec() must return a spec with "loader" set to None (a.k.a. not set) and with submodule_search_locations set....

In Python 3.9, calling importlib.util.find_spec on the name of a not-yet-imported namespace package returned a spec whose loader is None, but once the module is imported, the returned spec's loader is an instance of _frozen_importlib_external._NamespaceLoader. Note that _NamespaceLoader is an implementation detail of CPython.

We must resist the temptation to rely on Python's documented code for detecting namespace packages:

if spec.origin is None and spec.submodule_search_locations is not None:
    # namespace package
    sys.modules[] = module

This documentation is patently falsified by a cursory review of Python's source code, which clearly relies on loader being None.

See also PEP-420's summary of differences between namespace and regular packages, which is now somewhat outdated.

History of the migration loader's no-namespace check

The reason, given both in the source code comments and in discussions among maintainers, for the migration loader's check that an app's migrations package is not a namespace package is to discourage the latter's use.

The current namespace-check implementation was originally introduced in GH-1541, fixing #21015, to skip empty directories. GH-9665 allowed for the possibility that __file__ be None rather than missing on namespace packages because of bpo-32305, which was fixed in Python 3.7. The implementation was briefly removed in GH-11141 to fix #30300, but revived in GH-13218 because

Supporting namespace packages without a real use case seems contrary to the consensus in this thread (which I see as not promoting implicit namespace packages). Based on that consensus, my inclination wouldn't be to try to make Django work with as few files as possible.

Note: See TracTickets for help on using tickets.
Back to Top