Removing the magic

The "magic-removal" branch aims to make several sweeping changes to the Django codebase, removing warts that Django has accumulated over the years. Most changes involve the database API and removing some of its unneeded magic, and other changes involve improving the framework's simplicity and usability.

These changes will be integrated into the next Django release, 0.92.

This document explains the changes in the branch.

TOC(inline, RemovingTheMagic)

How to get the branch

Play with it! The branch is available via Subversion at ​http://code.djangoproject.com/svn/django/branches/magic-removal .

Using two versions of Django side-by-side

Here's one way to use Django's trunk and magic-removal branch on the same machine. This assumes Django's trunk (or a release such as 0.90 or 0.91) is installed:

# Get the magic-removal code somewhere on your filesystem. In this example, we use /home/python/django.
$ cd /home/python/django
$ svn co http://code.djangoproject.com/svn/django/branches/magic-removal

# This will have created a "magic-removal" directory.

# Whenever you want to use magic-removal, set the environment variable {{{PYTHONPATH}}} to the directory containing magic-removal.
export PYTHONPATH=/home/python/django/magic-removal

Overview

The biggest changes in magic-removal are:

• The magic package django.models no longer exists. To use models, just import the model class from wherever it lives on the Python path. Similarly, the magic modules (such as django.models.polls in the tutorial) no longer exist; now, you interact directly with the model class.
• All automatic pluralization is gone.
• The database API has changed in several ways.
• Various packages, such as the Django template system (previously in django.core.template), have been moved around to make importing less verbose and easier to remember.

Status

Not for the faint of heart at this point. Still being heavily developed.

Database changes you'll need to make

To upgrade from a previous Django installation, you'll need to make some database changes. Obviously, this doesn't apply if you're starting from scratch.

Rename core database tables

We've renamed a bunch of the core Django tables. To upgrade in MySQL and SQLite, execute this SQL in your database:

ALTER TABLE auth_groups RENAME TO auth_group;
ALTER TABLE auth_groups_permissions RENAME TO auth_group_permissions;
ALTER TABLE auth_messages RENAME TO auth_message;
ALTER TABLE auth_permissions RENAME TO auth_permission;
ALTER TABLE auth_users RENAME TO auth_user;
ALTER TABLE auth_users_groups RENAME TO auth_user_groups;
ALTER TABLE auth_users_user_permissions RENAME TO auth_user_user_permissions;
ALTER TABLE content_types RENAME TO django_content_type;
ALTER TABLE core_sessions RENAME TO django_session;
ALTER TABLE django_flatpages RENAME TO django_flatpage;
ALTER TABLE django_flatpages_sites RENAME TO django_flatpage_sites;
ALTER TABLE django_redirects RENAME TO django_redirect;
ALTER TABLE sites RENAME TO django_site;
DROP TABLE packages;

PostgreSQL is different, because you have to rename sequences, too. To upgrade in PostgreSQL, execute this SQL in your database:

BEGIN;

ALTER TABLE auth_groups RENAME TO auth_group;
ALTER TABLE auth_groups_id_seq RENAME TO auth_group_id_seq;
ALTER TABLE auth_group ALTER COLUMN id DROP DEFAULT;
ALTER TABLE auth_group ALTER COLUMN id SET DEFAULT nextval('public.auth_group_id_seq'::text);

ALTER TABLE auth_groups_permissions RENAME TO auth_group_permissions;
ALTER TABLE auth_groups_permissions_id_seq RENAME TO auth_group_permissions_id_seq;
ALTER TABLE auth_group_permissions ALTER COLUMN id DROP DEFAULT;
ALTER TABLE auth_group_permissions ALTER COLUMN id SET DEFAULT nextval('public.auth_group_permissions_id_seq'::text);

ALTER TABLE auth_messages RENAME TO auth_message;
ALTER TABLE auth_messages_id_seq RENAME TO auth_message_id_seq;
ALTER TABLE auth_message ALTER COLUMN id DROP DEFAULT;
ALTER TABLE auth_message ALTER COLUMN id SET DEFAULT nextval('public.auth_message_id_seq'::text);

ALTER TABLE auth_permissions RENAME TO auth_permission;
ALTER TABLE auth_permissions_id_seq RENAME TO auth_permission_id_seq;
ALTER TABLE auth_permission ALTER COLUMN id DROP DEFAULT;
ALTER TABLE auth_permission ALTER COLUMN id SET DEFAULT nextval('public.auth_permission_id_seq'::text);

ALTER TABLE auth_users RENAME TO auth_user;
ALTER TABLE auth_users_id_seq RENAME TO auth_user_id_seq;
ALTER TABLE auth_user ALTER COLUMN id DROP DEFAULT;
ALTER TABLE auth_user ALTER COLUMN id SET DEFAULT nextval('public.auth_user_id_seq'::text);

ALTER TABLE auth_users_groups RENAME TO auth_user_groups;
ALTER TABLE auth_users_groups_id_seq RENAME TO auth_user_groups_id_seq;
ALTER TABLE auth_user_groups ALTER COLUMN id DROP DEFAULT;
ALTER TABLE auth_user_groups ALTER COLUMN id SET DEFAULT nextval('public.auth_user_groups_id_seq'::text);

ALTER TABLE auth_users_user_permissions RENAME TO auth_user_user_permissions;
ALTER TABLE auth_users_user_permissions_id_seq RENAME TO auth_user_user_permissions_id_seq;
ALTER TABLE auth_user_user_permissions ALTER COLUMN id DROP DEFAULT;
ALTER TABLE auth_user_user_permissions ALTER COLUMN id SET DEFAULT nextval('public.auth_user_user_permissions_id_seq'::text);

ALTER TABLE content_types RENAME TO django_content_type;
ALTER TABLE content_types_id_seq RENAME TO django_content_type_id_seq;
ALTER TABLE django_content_type ALTER COLUMN id DROP DEFAULT;
ALTER TABLE django_content_type ALTER COLUMN id SET DEFAULT nextval('public.django_content_type_id_seq'::text);

ALTER TABLE core_sessions RENAME TO django_session;

ALTER TABLE django_flatpages RENAME TO django_flatpage;
ALTER TABLE django_flatpages_id_seq RENAME TO django_flatpage_id_seq;
ALTER TABLE django_flatpage ALTER COLUMN id DROP DEFAULT;
ALTER TABLE django_flatpage ALTER COLUMN id SET DEFAULT nextval('public.django_flatpage_id_seq'::text);

ALTER TABLE django_flatpages_sites RENAME TO django_flatpage_sites;
ALTER TABLE django_flatpages_sites_id_seq RENAME TO django_flatpage_sites_id_seq;
ALTER TABLE django_flatpage_sites ALTER COLUMN id DROP DEFAULT;
ALTER TABLE django_flatpage_sites ALTER COLUMN id SET DEFAULT nextval('public.django_flatpage_sites_id_seq'::text);

ALTER TABLE django_redirects RENAME TO django_redirect;
ALTER TABLE django_redirects_id_seq RENAME TO django_redirect_id_seq;
ALTER TABLE django_redirect ALTER COLUMN id DROP DEFAULT;
ALTER TABLE django_redirect ALTER COLUMN id SET DEFAULT nextval('public.django_redirect_id_seq'::text);

DROP TABLE packages;

ALTER TABLE sites RENAME TO django_site;
ALTER TABLE sites_id_seq RENAME TO django_site_id_seq;
ALTER TABLE django_site ALTER COLUMN id DROP DEFAULT;
ALTER TABLE django_site ALTER COLUMN id SET DEFAULT nextval('public.django_site_id_seq'::text);

COMMIT;

Database table-naming scheme has been changed

Database table names formerly were created by joining the app_label and module_name. Example: polls_polls.

Because there's no longer any concept of module_name, database table names are now formed by joining the app_label and model class name (lower case). Example: polls_poll.

As always, this behavior can be overridden on a per-model basis by specifying the db_table attribute in class Meta in your model.

To upgrade, you'll either have to explicitly set db_table in your models or rename your database tables to fit the new naming scheme Django expects. We'd recommend setting db_table, because it's easier.

Code changes you'll need to make

Model class and Field classes renamed/relocated

Change your models to import from django.db.models instead of django.core.meta.

from django.db import models

class Person(models.Model):
    first_name = models.CharField(maxlength=30)
    last_name = models.CharField(maxlength=30)

Interact directly with model classes, not with magic modules

Import the model class directly from the module in which it was defined. No more django.models.* magic.

from myproject.people.models import Person
p = Person(first_name='John', last_name='Smith')
p.save()

Namespace simplification

django.utils.httpwrappers has moved to django.http.

django.core.exceptions.Http404 has moved to django.http.Http404.

django.core.template has moved to django.template.

django.core.formfields has moved to django.forms.

django.core.extensions has moved to django.shortcuts.

django.core.extensions.DjangoContext has been renamed to RequestContext and moved to django.template.RequestContext.

You'll need to remove ".core." from your TEMPLATE_LOADERS settings values. Old:

TEMPLATE_LOADERS = (
    'django.core.template.loaders.filesystem.load_template_source',
    'django.core.template.loaders.app_directories.load_template_source',
#    'django.core.template.loaders.eggs.load_template_source',
)

New:

TEMPLATE_LOADERS = (
    'django.template.loaders.filesystem.load_template_source',
    'django.template.loaders.app_directories.load_template_source',
#    'django.template.loaders.eggs.load_template_source',
)

Custom template tag definitions need a similar change. Old:

from django.core import template
register = template.Library()

New:

from django.template import Library
register = Library()

The "auth" and "core" models have been split and moved to django.contrib as follows:

• django.models.auth has moved to django.contrib.auth.models.
• django.models.core.sites has moved to django.contrib.sites.models.
• django.models.core.contenttypes has moved to django.contrib.contenttypes.models.
• django.models.core.packages has moved to django.contrib.contenttypes.models. (Note that "packages" are going away before magic-removal is done.)

Session middleware has moved from django.middleware.sessions.SessionMiddleware to django.contrib.sessions.middleware.SessionMiddleware. Make sure to update your MIDDLEWARE_CLASSES setting, if you're using sessions.

Also, the Session model has moved from django/models/core.py to django/contrib/sessions/models.py. If you're accessing the Session model for some reason, note that location change.

The 'packages' module is no more

Packages no longer exist (they were redundant). If you've done lookups against content-types or permissions you'll need to modify your code slightly:

Old	New
contenttypes.get_list(package__label__exact='foo')	ContentType.objects.filter(package__exact='foo')
permissions.get_list(package__label__exact='foo')	Permission.objects.filter(package__exact='foo')

Changes to model syntax

• class META should now be class Meta. The latter is easier on the eyes.

• The following are no longer valid parameters to class Meta and should be removed:
  • module_name
  • admin (See "Moved admin options to 'class Admin'" below.)
  • exceptions (Just put your exceptions in the module that contains the models and access them normally.)
  • module_constants (Just put your constants in the module that contains the models and access them normally.)
  • where_constraints (Just use a custom manager. See "Custom managers, and multiple managers" below.)

Moved admin options to 'class Admin'

Instead of admin=meta.Admin in the class META, all admin options are in an inner class Admin.

Old:

class Person(meta.Model):
    first_name = meta.CharField(maxlength=30)
    last_name = meta.CharField(maxlength=30)
    class META:
        admin = meta.Admin(
            list_display = ('first_name', 'last_name')
        )

New:

class Person(models.Model):
    first_name = models.CharField(maxlength=30)
    last_name = models.CharField(maxlength=30)
    class Admin:
        list_display = ('first_name', 'last_name')

Database connection relocated/renamed

For any code that uses the raw database connection, use django.db.connection instead of django.core.db.db.

Old:

from django.core.db import db
cursor = db.cursor()

New:

from django.db import connection
cursor = connection.cursor()

Backend-specific functions, if you should need them, are available at django.db.backend.

Old:

from django.core import db
db.quote_name('foo')

New:

from django.db import backend
backend.quote_name('foo')

Also, the various backend functionality has been split into three separate modules for each backend -- base.py, creation.py and introspection.py. This is purely for performance and memory savings, so that basic, everyday Django usage doesn't have to load the introspective functionality into memory.

Model methods no longer automatically have access to datetime and db modules

Formerly, each model method magically had access to the datetime module and to the variable db, which represents the current database connection. Now, those have to be imported explicitly.

Old:

    def some_method(self):
        print datetime.datetime.now()
        cursor = db.cursor()
        cursor.execute("UPDATE something;")

New:

import datetime
from django.db import connection

# ...

    def some_method(self):
        print datetime.datetime.now()
        cursor = connection.cursor()
        cursor.execute("UPDATE something;")

Descriptor fields

All "table-level" functions -- ways of retrieving records tablewide rather than performing instance-specific tasks -- are now accessed via a model class's objects attribute. They aren't direct methods of a model instance object because we want to keep the "table-wide" and "row-specific" namespaces separate.

A model class's objects attribute is an instance of django.db.models.manager.Manager. A manager has the following methods, all of which return a QuerySet instance.

• all() -- Returns a QuerySet of all objects in the database. This is like the old get_list(). Takes no arguments.
• filter(**kwargs) -- Returns a QuerySet, filtered by the given keyword arguments. Lookup arguments are in the same style as previously, e.g. pubdate__year=2005, except you can leave off __exact as a convenience. For example, name='John' and name__exact='John' are equivalent.
• exclude(**kwargs) is the same as filter(), but returns objects where the given arguments are not true.
• order_by(*fieldnames) -- Returns a QuerySet
• count() -- Returns the count of all objects in the database.
• dates(field_name, kind) -- Like the old get_FIELD_list() for date fields. For example, old-school get_pubdate_list('year') is now dates('pubdate', 'year').
• delete() -- Deletes all objects.
• distinct() -- Returns a QuerySet with DISTINCT set.
• extra(select=None, where=None, params=None, tables=None) -- Sets the select, where, params and tables arguments, which are in the same format as before.
• get(**kwargs) -- Like the old get_object(). Returns an object or raises DoesNotExist on error.
• in_bulk(id_list) -- Like the old get_in_bulk().
• iterator() -- Returns a generator that iterators over results.
• select_related() -- Returns a QuerySet with the "select related" option (which acts the same as before) set.
• values(*fieldnames) -- Like the old get_values().

Each QuerySet has the following methods, which return a clone of the query set with the appropriate changes made:

• filter(**kwargs)
• order_by(*fieldnames)
• iterator()
• count()
• get(**kwargs)
• delete()
• filter(**kwargs)
• select_related()
• order_by(*fieldnames)
• distinct()
• extra(select=None, where=None, params=None, tables=None)

Here are some examples, which use the following models:

class Reporter(models.Model):
    fname = models.CharField(maxlength=30)
    lname = models.CharField(maxlength=30)

class Site(models.Model):
    name = models.CharField(maxlength=20)

class Article(models.Model):
    headline = models.CharField(maxlength=50)
    reporter = models.ForeignKey(Reporter)
    pub_date = models.DateField()
    sites = models.ManyToManyField(Site)

Old syntax	New syntax
reporters.get_list()	Reporter.objects.all()
reporters.get_list(fname__exact='John')	Reporter.objects.filter(fname='John')
reporters.get_list(order_by=('-lname', 'fname'))	Reporter.objects.order_by('-lname', 'fname')
reporters.get_list(fname__exact='John', order_by=('lname',))	Reporter.objects.filter(fname='John').order_by('lname')
reporters.get_object(pk=3)	Reporter.objects.get(pk=3)
reporters.get_object(fname__contains='John')	Reporter.objects.get(fname__contains='John')
reporters.get_list(fname__ne='John')	Reporter.objects.exclude(fname='John') (note that ne is no longer a valid lookup type)
(not previously possible)	Reporter.objects.exclude(fname__contains='n')
reporters.get_list(distinct=True)	Reporter.objects.distinct()
reporters.get_values()	Reporter.objects.values()
reporters.get_in_bulk([1, 2])	Reporter.objects.in_bulk([1, 2])
reporters.get_in_bulk([1, 2], fname__exact='John')	Reporter.objects.filter(fname='John').in_bulk([1, 2])
Date lookup
articles.get_pub_date_list('year')	Article.objects.dates('pub_date', 'year')
Latest-object lookup
articles.get_latest() (required get_latest_by in model)	Article.objects.latest() (with get_latest_by in model)
(Not previously possible)	Article.objects.latest('pub_date') # Latest by pub_date (overrides get_latest_by field in model)
Many-to-one related lookup
article_obj.reporter_id	article_obj.reporter.id
article_obj.get_reporter()	article_obj.reporter
reporter_obj.get_article_list()	reporter_obj.article_set.all()
reporter_obj.get_article_list(headline__exact='Hello')	reporter_obj.article_set.filter(headline='Hello')
reporter_obj.get_article_count()	reporter_obj.article_set.count()
reporter_obj.add_article(headline='Foo')	reporter_obj.article_set.add(headline='Foo')
(Alternate syntax)	reporter_obj.article_set.add(article_obj)
("values" lookup, etc., not previously possible)	reporter_obj.article_set.values()
Many-to-many related lookup
article_obj.get_site_list()	article_obj.sites.all()
article_obj.set_sites([s1.id, s2.id])	article_obj.sites.clear(); article_obj.sites.add(s1); article_obj.sites.add(s2)
article_obj.set_sites([s1.id]) # deletion	article_obj.sites.remove(s2)
site_obj.get_reporter_list()	site_obj.reporter_set.all()

Note that related-object lookup uses the default manager of the related object, which means the API for accessing related objects is completely consistent with the API for accessing objects via a manager.

Also note that managers can't be accessed from instances:

p = Person.objects.get(pk=1)
p.objects.all() # Raises AttributeError

Override default manager name ('objects')

If a model already has an objects attribute, you'll need to specify an alternate name for the objects manager.

class Person(models.Model):
    first_name = models.CharField(maxlength=30)
    last_name = models.CharField(maxlength=30)
    objects = models.TextField()
    people = models.Manager()

p = Person(first_name='Mary', last_name='Jones', objects='Hello there.')
p.save()
p.objects == 'Hello there.'
Person.people.all()

Custom managers, and multiple managers

You can create as many managers as you want. When necessary (such as on the admin), Django will use the first one defined, in order.

If you define at least one custom manager, it will not get the default "objects" manager.

class Person(models.Model):
    first_name = models.CharField(maxlength=30)
    last_name = models.CharField(maxlength=30)
    people = models.Manager()
    fun_people = SomeOtherManager()

If a manager needs to access its associated model class, it should use self.model. Example:

class PersonManager(models.Manager):
    def get_fun_person(self):
        try:
            return self.get(fun=True)
        except self.model.DoesNotExist:
            print "Doesn't exist."

Using a custom manager for the admin

Sometimes you'll want to use a different manager for the admin displays (e.g. to display only objects matching some criteria in the admin). You can do this by defining the manager option in your Admin declaration:

class LivingPeopleManager(models.Manager):
    def get_query_set(self):
        return super(LivingPeopleManager, self).get_query_set().filter(is_alive=True) 

class Person(models.Model):
    name = models.CharField(maxlength=50)
    is_alive = models.BooleanField()
    
    class Admin:
        manager = LivingPeopleManager()

(see "You can override default QuerySets" for more on QuerySets)

Added a more powerful way of overriding model methods, removed hard-coded _pre_save(), _post_save(), etc.

Proper subclassing of methods now works, so you can subclass the automatic save() and delete() methods. This removes the need for the _pre_save(), _post_save(), _pre_delete() and _post_delete() hooks -- all of which have been removed. Example:

class Person(models.Model):
    first_name = models.CharField(maxlength=30)
    last_name = models.CharField(maxlength=30)

    def save(self):
        self.do_something()
        super(Person, self).save() # Call the "real" save() method.
        self.do_something_else()

You can even skip saving (as requested in #1014).

class Person(models.Model):
    first_name = models.CharField(maxlength=30)
    last_name = models.CharField(maxlength=30)

    def save(self):
        if datetime.date.today() > datetime.date(2005, 1, 1):
            super(Person, self).save() # Call the "real" save() method.
        else:
            # Don't save.
            pass

Renamed DoesNotExist exception

Instead of people.PersonDoesNotExist, it's Person.DoesNotExist.

Old:

from django.models.myapp import people
try:
    people.get_object(pk=1)
except people.PersonDoesNotExist:
    print "Not there"

New:

from path.to.myapp.models import Person
try:
    Person.objects.get(pk=1)
except Person.DoesNotExist:
    print "Not there"

Moved admin URLconf to shorten its path

You'll need to change your URLconf to include the new location.

• Old: django.contrib.admin.urls.admin
• New: django.contrib.admin.urls

get_object_or_404 and get_list_or_404 now take model classes, not modules

Old:

get_object_or_404(polls, pk=1)

New:

get_object_or_404(Poll, pk=1)

Changed the parameters you pass to generic views

Because there's no longer a concept of module_name, the "info_dicts" passed to ​generic views no longer accept "app_label" and "module_name". Instead, pass the parameter "queryset", which should be a QuerySet instance.

Old:

info_dict = {
    'app_label': 'blog',
    'module_name': 'entries'
}

New:

from myproject.blog.models import Entry
info_dict = {'queryset': Entry.objects.all()}

Changed template names in generic views

Because there's no longer a concept of module_name, ​generic views no longer create templates based on the module_name. Wherever they used module_name, they now use model_name, a lowercase version of the model name.

Note that app_label remains the same.

These examples assume models live in myproject/blog/models.py.

• Old: blog/entries_archive.html
• New: blog/entry_archive.html

Moved settings into an instance

Settings have moved out of a dedicated module django.conf.settings into an instance in the django.conf module. So now you need to import the settings object and reference settings as attributes of that instance.

• Old: from django.conf.settings import LANGUAGE_CODE
• New: from django.conf import settings

Wrappers around the Django machinery can make use of this by exchanging the settings instance with a proxy instance that delegates attribute access to a per-thread or per-location global.

Removed SilentVariableFailure exception

Old behavior: Any exception that subclasses django.core.template.SilentVariableFailure fails silently in the template system.

New behavior: Any exception that has a silent_variable_failure attribute fails silently in the template system. django.core.template.SilentVariableFailure no longer exists.

Request.user is now set via middleware

It used to be set in the modepython and wsgi handlers. You will need to add "django.contrib.auth.middleware.RequestUserMiddleware" somewhere after "django.contrib.sessions.middleware.SessionMiddleware" in your settings.py file. Otherwise authentication won't work.

New functionality you can start using

Models support properties

Unlike before, properties are supported on models.

from django.db import models

class Person(models.Model):
    first_name = models.CharField(maxlength=30)
    last_name = models.CharField(maxlength=30)

    def _get_full_name(self):
        return "%s %s" % (self.first_name, self.last_name)
    full_name = property(_get_full_name)

You can override default QuerySets

You can specify the default QuerySet (see "Descriptor fields" above) that a manager uses. For example:

class PublishedBookManager(models.Manager):
    def get_query_set(self):
        return super(PublishedBookManager, self).get_query_set().filter(is_published=True)

class Book(models.Model):
    title = models.CharField(maxlength=50)
    author = models.CharField(maxlength=30)
    is_published = models.BooleanField()
    published_objects = PublishedBookManager()

Stuff that still needs to be done

Remove automatic manipulators, in favor of validation-aware models

Status: Not done yet

Change subclassing syntax

Status: Not done yet

See ModelInheritance