This is an attempt to bring together the current state and long term plans of how application loading is done in Django, what it could do, and why.

Author: Marc Tamlyn. Hence references to "me" or "I"

Application loading refactor

At present application loading is handled in different ways through Django. Because of some of the limitations of Python, and some peculiarities of how models work, modules should not be reimported. Consequently Django maintains a global static, called the application cache. Models are imported from the application cache transparently when interacting with the database. But applications listed in INSTALLED_APPS contain more than just models. Some contain template tags, admin.py files, search_indexes.py files from haystack, template folders and many other features which applications use to inspect each other, or to contribute objects to a site-wide collection.

For the majority of applications, even reusable applications, anything more than the current system is unnecessary. Models are automatically registered, template tags can be loaded, and templates are available. (Tests are also available but this is a separate issue.)

However, not every application has all of these components - there are many useful pluggable applications which do not contain models, but must still be registered in INSTALLED_APPS. A simple example is django-pagination - which consists of a single template tag. This module must ship with an empty models.py module to allow it to be registered as an installed app, for that template tag to be available.

The problem is even more difficult for "meta-applications" - that is to say applications which exist solely to interact with other applications. The canonical example of this is the django admin, but there are many third party applications which do a similar thing. Haystack (the search engine framework) is a good example. Also there are applications which have a plugin archicture, whereby adding a second application to installed apps changes the behaviour of the first application. All of these third party systems have implemented their own ways to do this, with varying degrees of success.

This is an opportunity for Django to do something great for the third party application community. It will be a very powerful tool, and opens up a range of magical things which could be done. App loading is magical, but Magic is not always a bad thing - most Django developers are very pleased they don't have to individually register every model with Django as they do to register a model with the admin. You just put it in the models.py file and it works! This will allow the magic to be more explicit, and ultimately more consistent between third party applications.

This is a big change, and almost redefines what an "App" is within Django. To my mind, an app was originally intended to be a fairly self contained part of the system, normally in fact a self contained part of the Web site.

Specification

Here is a specification of what an application is, and what it can do. Perhaps not all of these points need to be implemented to be able to integrate this refactor into core, but they should be considered to avoid future backwards incompatibilities.

• An application is a self contained piece of code which contributes something to the site. It may contibute any number of the following:
  • models
  • templates
  • template tags
  • static files
  • admin configuration
  and these are just (some of?) the things which are picked up on by other parts of the Django core. Other things (seach indexes, migrations etc) may also be contributed by the application, and processed by other applications.

Note Some things are explicitly registered in settings, such as middleware, database routers and cache backends. These things may be contained within an application's code base, but they will not be registered automatically by the presence of that application in the INSTALLED_APPS.

• An application may contain urls and views. These will generally be explicitly registered, but some applications may choose to attach dynamic urls offer a way of getting the urls from the application rather than directly (similar to how the admin does to it's site class).

• An application may modify another application. That is to say, an application may provide an interface on it's application class for other applications to modify it. This will facilitate a plugin architecture. A classic example of where this will be useful is a Content Management System.

• An application may depend on the presence of another application. It is preferable for the application to depend on a defined contract with another application, rather than explicity depend on that application. This should allow applications to be more loosely coupled, leading potentially to pluggable auth/messages applications. It also allows such applications to be tested perhaps without the presence of the dependant app - a mocked object may suffice.

• An application may inspect other applications to gather information about them. The system should provide tools to easily and safely access this information, whether it is accessing a folder or a python module. Where possible, we should modify parts of Django core & contrib to use this system. Examples would include the app directories template loader, admin site and template tag libraries.

• An exact copy of an application can be deployed within the same project by changing its label. This will have some subtle issues regarding imports.

Implementation plan (incomplete)

Application class. Talk about what APIs would bee needed on that class to achieve the above. Separate by the above points. Ideally, does not need to be contained within the python module it relates to (thus allowing me to subclass an app's Application Class in my own project to tweak how it works, this may or may not be better than passing configuration options)

Application cache. Discuss what hooks this would need to provide, including signals, for the above to work.

Examine any backwards incompatibilities which may result. The app refactor itself shouldn't introduce any, but the new functionality which it offers and the resulting changes in other applications such as contrib.auth may need deprecation paths.

Current status

This concept has been around in the Django ecosystem for about 4 years now. The original impetus was to allow custom app_label and verbose_name attributes on an app (see #3591). The sticking point was often the difficulties of installing the same app multiple times - there are a bunch of problems with this, not least including dealing with from grail.models import Spam. Well, which version of Spam?

An original discussion of the ideas can be found at InstalledAppsRevision.

It was taken on as a Google Summer of Code project in summer 2010 by Arthur Koziel. Good progress was made, with a refactor of the AppCache and a new App class. Some of the design decisions made are discussed here: ​https://groups.google.com/forum/?fromgroups#!topic/django-developers/TMoght1IWHk

The last action on this branch was in summer 2011 by Jannis. Travis Swicegood and Marc Tamlyn have both expressed interest in working further with the project.

Preston has merged master into the existing branch with work done by Jannis and Arthur. (​https://github.com/ptone/django/commit/8722bdcc17293baa8b527ae7169b9a465e728d41)

The app cache branch has a good number of tests, but little documentation. We need to start by working out what has been achieved so far and how it fits in with the long term plan. The tests for the new classes work, but there are a few other tests breaking as a result of them expecting certain behaviour about the AppCache.

These tests will need fixing before any merge to master can be made.