Splitting up the settings file

If you use a source control system (CVS, SVN, ...), or want to publish your application on the web, it may be a good idea to move sensitive or machine/user specific settings like database passwords and such out of the main settings.py file.

As discussions on the ​django-developers mailing list have shown everybody has different requirements and ideas how to do this. This page is meant to collect some of these ideas for future reference.

One thing to keep in mind is that Django's config files are pure Python. This gives you the ultimate flexibility to handle configurations the way you think is best. Or to quote Adrian Holovaty:

We don't need a default solution for this. It's not within the scope of this project to tell people how they should organize their settings files. Take that opportunity to showcase your individualism.

Some of the examples listed below need to be modified for compatibility with Django 1.4 and later. They are marked with "Django <1.4" in the title.

Links to discussions

• ​how to modularize django settings.py? on stackoverflow.com

Different settings in different files

I believe the first user who came up with this was Hugo, who used this method in the projects he published on ​his site.

SECRET_KEY = open(os.path.expanduser('~/.gallery-secret')).read().strip()

ini-style file for deployment

This is a solution that Michael Radziej posted to django-developers. His motivation was to be able to store the settings for an app he published under /etc with all the other system config files.

For a similar but more transparent solution see fox's method

/etc/whatever/settings.ini (Django <1.4)

[database]
DATABASE_USER: bla
DATABASE_PASSWORD: XXXXXXXX
DATABASE_HOST: dev
DATABASE_PORT:
DATABASE_ENGINE: mysql
DATABASE_NAME: blo
TESTSUITE_DATABASE_NAME: test_blo

[secrets]
SECRET_KEY: random-string-of-ascii
CSRF_MIDDLEWARE_SECRET: random-string-of-ascii

[cookies]
SESSION_COOKIE_DOMAIN:

# all settings in debug section should be false in productive
environment
# INTERNAL_IPS should be empty in productive environment
[debug]
DEBUG: true
TEMPLATE_DEBUG: true
VIEW_TEST: true
INTERNAL_IPS: 127.0.0.1
SKIP_CSRF_MIDDLEWARE: true

[email]
SERVER_EMAIL: django@localhost
EMAIL_HOST: localhost

# the [error mail] and [404 mail] sections are special. Just add
lines with
#  full name: email_address@domain.xx
# each section must be present but may be empty.
[error mail]
Adam Smith: adam@localhost

[404 mail]
John Wayne: john@localhost

/path/to/whatever/settings.py (Django <1.4)

from ConfigParser import RawConfigParser

config = RawConfigParser()
config.read('/etc/whatever/settings.ini')

DATABASE_USER = config.get('database', 'DATABASE_USER')
DATABASE_PASSWORD = config.get('database', 'DATABASE_PASSWORD')
DATABASE_HOST = config.get('database', 'DATABASE_HOST')
DATABASE_PORT = config.get('database', 'DATABASE_PORT')
DATABASE_ENGINE = config.get('database', 'DATABASE_ENGINE')
DATABASE_NAME = config.get('database', 'DATABASE_NAME')
TEST_DATABASE_NAME = config.get('database', 'TESTSUITE_DATABASE_NAME')

SECRET_KEY = config.get('secrets','SECRET_KEY')
CSRF_MIDDLEWARE_SECRET = config.get('secrets', 'CSRF_MIDDLEWARE_SECRET')

SESSION_COOKIE_DOMAIN = config.get('cookies','SESSION_COOKIE_DOMAIN')

DEBUG = config.getboolean('debug','DEBUG')
TEMPLATE_DEBUG = config.getboolean('debug','TEMPLATE_DEBUG')
VIEW_TEST = config.getboolean('debug', 'VIEW_TEST')
INTERNAL_IPS = tuple(config.get('debug', 'INTERNAL_IPS').split())
if config.getboolean('debug', 'SKIP_CSRF_MIDDLEWARE'):
    MIDDLEWARE_CLASSES = tuple([x for x in list(MIDDLEWARE_CLASSES)
                                  if not x.endswith('CsrfMiddleware')])

SERVER_EMAIL = config.get('email', 'SERVER_EMAIL')
EMAIL_HOST = config.get('email', 'EMAIL_HOST')
ADMINS = tuple(config.items('error mail'))
MANAGERS = tuple(config.items('404 mail'))

Multiple setting files importing from each other

This is my (Steven Armstrong) preferred solution.

Keep application wide, unsensitive settings and sane defaults in your normal settings.py file.

/path/to/whatever/settings.py (Django <1.4)

import os

BASE_DIR = os.path.dirname(os.path.abspath(__file__))

# why is this here?
#PROJECT_DIR = os.path.abspath(os.path.join(BASE_DIR, '..'))


PROJECT_DIR = BASE_DIR

DEBUG = False
TEMPLATE_DEBUG = DEBUG

ADMINS = (
    ('Mr Sysadmin', 'sysadmin@domain.tld'),
)

MANAGERS = ADMINS

DATABASE_ENGINE = 'sqlite3'
DATABASE_NAME = os.path.join(PROJECT_DIR, 'project.db')
DATABASE_USER = ''
DATABASE_PASSWORD = ''
DATABASE_HOST = ''
DATABASE_PORT = ''

TIME_ZONE = 'Europe/Zurich'
LANGUAGE_CODE = 'en-us'
SECRET_KEY = 'secret'

#[more default and app wide settings]

from settings_local import *

At the end of your normal settings.py include * from an other, machine specific config file which could look something like this.

/path/to/whatever/settings_local.py (Django <1.4)

DEBUG = True
TEMPLATE_DEBUG = DEBUG

# don't want emails while developing
ADMINS = ()
MANAGERS = ADMINS

DATABASE_ENGINE = 'mysql'
DATABASE_NAME = 'mydbname'
DATABASE_USER = 'mydbuser'
DATABASE_PASSWORD = 'mydbpassword'
DATABASE_HOST = 'localhost'
DATABASE_PORT = ''

SECRET_KEY = 'random-string-of-ascii'

#[more user/machine specific settings]

urls.py

import settings
     (r'^static/(?P<path>.*)$', 'django.views.static.serve', {'document_root': settings.BASE_DIR+'/core/static/', 'show_indexes': True}),                      

settings.py goes into CVS, SVN, (put your favorite RCS here)
settings_local.py does _not_ go under RCS

If wanted a settings_local.template file can be put under version control with instructions to copy it over to settings_local.py, change it to suite the environment, and to never ever commit it to the RCS system.

Development/Machine Dependant Settings Configuration

I personally use this solution.

/path/to/project/settings.py

# Django project settings loader
import os
ROOT_PATH = os.path.dirname(os.path.abspath(__file__))

# You can key the configurations off of anything - I use project path.
configs = {
    '/path/to/user1/dev/project': 'user1',
    '/path/to/user2/dev/project': 'user2',
    '/path/to/qa/project': 'qa',
    '/path/to/prod/project': 'prod',
}

# Import the configuration settings file - REPLACE projectname with your project
config_module = __import__('config.%s' % configs[ROOT_PATH], globals(), locals(), 'projectname')

# Load the config settings properties into the local scope.
for setting in dir(config_module):
    if setting == setting.upper():
        locals()[setting] = getattr(config_module, setting)

This settings loader will import the appropriate settings module. I store my settings files in a config directory in the root of the project.

/path/to/project/config/prod.py (Django <1.4)

DEBUG = False
TEMPLATE_DEBUG = DEBUG

# don't want emails while developing
ADMINS = ()
MANAGERS = ADMINS

DATABASE_ENGINE = 'mysql'
DATABASE_NAME = 'mydbname'
DATABASE_USER = 'mydbuser'
DATABASE_PASSWORD = 'mydbpassword'
DATABASE_HOST = 'localhost'
DATABASE_PORT = ''

SECRET_KEY = 'random-string-of-ascii'

# More production settings

/path/to/project/config/user1.py (Django <1.4)

# Load the production settings that we will overwrite as needed in our user1 settings file.
from projectname.config.prod import *

DEBUG = True

DATABASE_NAME = 'devdbname'

# More development/maching specific settings

Using exec to incorporate local settings

I've been using code like this to incorporate local settings:

settings.py

# Non-machine-specific settings live directly in settings.py
# Machine-specific settings live in another file, perhaps local_settings.conf

exec open('local_settings.conf') in globals()

This allows the local settings to both read and modify the main settings' values (and it's simple).

As an enhancement, a small amount of privilege separation can be had by making the local_settings.conf file unreadable by the uid that runs the web app, opening local_settings.conf in a setuid wrapper, passing its file descriptor to the Django process, and using exec os.fdopen(..., 'r') in globals(). This isn't perfect, since any secrets read from that file are still stored in the Django process, but it does make it harder for someone who has gained access to the user id running the Django app to discover your secrets. (Obviously all of the Python files used by the app must be owned by a user other than the one the app runs as, as well.)

Using a list of conf files (Transifex)

​Transifex has adopted a solution using an ordered list of multiple .conf files inside a settings directory. This helps separating the config options, packaging in Linux distributions, and putting the settings files under /etc.

$ ls settings/

10-base.conf
20-engines.conf
30-site.conf
...

$ cat settings.py

import os.path
import glob
conffiles = glob.glob(os.path.join(os.path.dirname(__file__), 'settings', '*.conf'))
conffiles.sort()
for f in conffiles:
    execfile(os.path.abspath(f))

To apply local-only changes, create a XX-local.conf file, which can be ignored with your VCS of choice:

$ cat settings/51-local.conf

INSTALLED_APPS += ['debug_toolbar']

$ cat .hgignore

syntax: glob
settings/*-local.conf

Accessing and modifying existing settings

I (​Antti Kaihola) use the following solution for defining local settings. It provides access to the original settings while defining local settings. Note: this only works if you import from the base settings.py or settings/__init__.py.

settings/__init__.py:

INSTALLED_APPS = ('whatever',)
try:
    from settings.local import *
except ImportError:
    pass

settings/local.py:

import sys
globals().update(vars(sys.modules['settings']))

INSTALLED_APPS += ('another_app',)

Older, clumsier methods I've used:

settings.py:

INSTALLED_APPS = ('whatever',)
import more_settings
more_settings.modify(globals())

more_settings.py:

def modify(settings):
    settings['INSTALLED_APPS'] += ('another_app',)

alternate more_settings.py with more magic:

def config(INSTALLED_APPS=(), **other_settings):
    INSTALLED_APPS += ('another_app',)
    del other_settings
    return locals()

def modify(settings):
    settings.update(config(**settings))

Settings inheritance

I (​Sergey Yushckeyev) have extended proposed solution:

env

export APP_SETTINGS = 'development.cordis'

settings/__init__.py

CONFIGURATION = 'common'

def deep_update(from_dict, to_dict):
    for (key, value) in from_dict.iteritems():
        if key in to_dict.keys() and isinstance(to_dict[key], dict) and isinstance(value, dict):
            deep_update(value, to_dict[key])
        else:
            to_dict[key] = value

try:
    modules = os.environ.get('APP_SETTINGS', '').split('.')
    current = __name__
    for name in modules:
        module  = getattr(__import__(current, globals(), locals(), [name]), name)
        current += '.' + name
        current_settings = {}
        for setting in dir(module):
            if setting == setting.upper():
                current_settings[setting] = getattr(module, setting)
        deep_update(current_settings, locals())
except ImportError, e:
    print 'Unable to import configuration: %s' % e

settings/development/cordis.py

CONFIGURATION = 'cordis'

app/models.py

from settings import CONFIGURATION
print CONFIGURATION # prints 'cordis'

Setting Inheritance with Hierarchy

I (​Karim A. Nassar) need to be able to inherit from a base settings file, then for each environment provide overrides. In dev env, we need developer specific settings. The following extends ​Sergey Yushckeyev's solution above.

This script merges settings/common.py, settings/$APP_ENV.py and settings/<developer overrides>.py, in that order.

settings/__init__.py

import os, pwd

# certain keys we want to merge instead of copy
merge_keys = ('INSTALLED_APPS', 'MIDDLEWARE_CLASSES')

def deep_update(from_dict, to_dict):
    for (key, value) in from_dict.iteritems():
        if key in to_dict.keys() and isinstance(to_dict[key], dict) and isinstance(value, dict):
            deep_update(value, to_dict[key])
        elif key in merge_keys:
            if not key in to_dict:
                to_dict[key] = ()
            to_dict[key] = to_dict[key] + from_dict[key]
        else:
            to_dict[key] = value

# this should be one of prod, qa, staging, dev. Default to dev for safety.
env = os.environ.get('APP_ENV', 'dev')

# try to load user specific settings
uid = pwd.getpwuid(os.getuid())[0]

modules = ('common', env, uid)
current = __name__
for module_name in modules:
    try:
        module = getattr(__import__(current, globals(), locals(), [module_name]), module_name)
    except ImportError, e:
        print 'ERROR: Unable to import %s configuration: %s' % (module_name, e)
        raise
    except AttributeError, e:
        if env == 'dev' and module_name == uid:
            print 'WARNING: Unable to import %s dev configuration: does %s.py exist?' % (module_name, module_name)
        else:
            raise

    # create a local copy of this module's settings
    module_settings = {}
    for setting in dir(module):
        # all django settings are uppercase, so this ensures we
        # are only processing settings from the dir() call
        if setting == setting.upper():
            module_settings[setting] = getattr(module, setting)
    deep_update(module_settings, locals())

#print locals() # for debugging

Then, move settings.py to settings/common.py.

The above defaults to dev env. You could pass in anything to manage.py via the APP_ENV environment variable. Common ones would be prod, staging, qa, etc. For each APP_ENV, you create a file $APP_ENV.py. Finally, create $USER.py file with developer specific overrides. Example files (excepting settings/common.py) below..

settings/dev.py

import os

# cwd is settings. determine project path
cwd = os.path.dirname(os.path.abspath(__file__)) 
project_path = cwd[:-9] # chop off "settings/"

DEBUG = True
TEMPLATE_DEBUG = DEBUG

TEMPLATE_DIRS = (
    # Put strings here, like "/home/html/django_templates" or "C:/www/django/templates".
    # Always use forward slashes, even on Windows.
    # Don't forget to use absolute paths, not relative paths.

    # automatically inject project templates directory into the dev env
    '%s/templates' % ( project_path )
)

settings/$USER.py

DATABASES = {
    'default': {
        'USER': 'me',           
        'PASSWORD': 'i<3django',
    }
}

INSTALLED_APPS = (
    'haystack',
)

HAYSTACK_CONNECTIONS = {
    'default': {
        'ENGINE': 'haystack.backends.solr_backend.SolrEngine',
        'URL': 'http://127.0.0.1:8983/solr'
    },
}

Of course, you may find that you actually want some of the above settings in common.py. Configure however you need for your situation.

Simple Package Organization for Environments

​I have used a Python package organization similar to those above for per-environment settings, but much simplified: leaving inheritance to be handled by plain Python modules, without deep-merging dictionary magic. Just be explicit! It's easier to know what production.py configures at a glance without having to go dig into data structures in other files. Having per-developer settings is asking for issues with inconsistent configurations to crop up in deployment environments.

[myapp]$ tree settings
settings
├── __init__.py
├── defaults.py
├── dev.py
└── production.py

The initial settings.py was moved to settings/defaults.py. By default, the settings package will use development configuration, so that local dev Just Works as normal:

settings/__init__.py

from dev import *

settings/dev.py

# Load defaults in order to then add/override with dev-only settings
from defaults import *

DEBUG = True
# ... etc.

For production (or staging, etc.), just add an appropriate module and set Django's conventional ​DJANGO_SETTINGS_MODULE environment variable in your deployment setup to load it:

settings/production.py

# Load defaults in order to then add/override with production-only settings
from defaults import *

DEBUG = False
# ... etc.

Wherever appropriate for your deployment (such as an Apache SetEnv directive, include the equivalent of:

$ export DJANGO_SETTINGS_MODULE=myapp.settings.production

Using django-admin.py in deployment rather than manage.py should automatically put your app's package on sys.path so that an import path qualified this way will work.

If you must use a settings/local.py for some reason, virtualenvwrapper's postactivate/postdeactivate hooks are a good place to set DJANGO_SETTINGS_MODULE to point to it for local development per-project. It's advisable to eliminate as many potential discrepancies between dev and production configuration as possible, though.

Yipit's method

​Adam Nelson from ​Yipit blogged about their method of managing Django settings. See ​Extending Django Settings for the Real World. TODO: summarize the blog post here.

Rob Golding's method

Rob Golding posted on his blog how he solved this issue. It's a bit of a hack really but it works nicely and allows you to override or extend anything in settings.py.

Add this to the bottom of settings.py:

try:
    LOCAL_SETTINGS
except NameError:
    try:
        from local_settings import *
    except ImportError:
        pass

Now create a local_settings.py with the following:

LOCAL_SETTINGS = True
from settings import *

You can now add things to local_settings.py like this:

MIDDLEWARE_CLASSES +=
('debug_toolbar.middleware.DebugToolbarMiddleware',)
INTERNAL_IPS = ('127.0.0.1',)
INSTALLED_APPS += ('debug_toolbar',)

This does however have the drawback of settings.py being parsed twice but that shouldn't cause any sort of mention worthy performance loss.

fox's method

First write a config file with passwords ad secret keys. Save it somewhere. This is the external_config.py

from ConfigParser import ConfigParser, NoOptionError

class ExternalConfig(object):
    """
    Wrapper for ConfigParser
    Reads a section from a config file and provides an object with the
    properties and the values defined in the config.
    """
    def __init__(self, conf_file, section):
        """
        Reads the section "section" from the config "conf_file"
        """
        self._conf_file = conf_file
        self._section = section

        self._config = ConfigParser()
        self._config.read(self._conf_file)

        self._index = len(self._config.options(self._section))

    def __getattr__(self, name):
        """
        Read the property from the config
        """
        if name.startswith('_'):
            return self.__dict__[name]
        try:
            value = self._config.get(self._section, name)
            if value.isdigit():
                return int(value)
            elif value in ('True', 'False'):
                return value == 'True'
            else:
                return value
        except NoOptionError:
            raise AttributeError

    def __getitem__(self, name):
        """
        Read the property from the config
        """
        return self.__getattr__(name)

    def __iter__(self):
        """
        __iter__ method
        """
        return self

    def next(self):
        """
        next method
        """
        if self._index == 0:
            raise StopIteration
        self._index -= 1
        return self._config.options(self._section)[self._index]

This is production_settings.py

from external_config import ExternalConfig
import sys

_filename = ""  # add your config file
_section = "mysection"  # add your config section
_this_module = sys.modules[__name__]
_config = ExternalConfig(_filename, _section)

__all__ = []

for key in _config:
    __all__.append(key)
    value = _config[key]
    setattr(_this_module, key, value)

Now importing * from production_settings.py will import data from the config file

readthedocs.org's method

The settings directory used at ​Read The Docs is available ​in their repository.

modular method

Move settings into a package, split everything into modules, then import in __init__.py.

See this ​post and the ​example on github.

django-flexisettings

Another option we use at ledapei.com is ​django-flexisettings which breaks down configuration files in a very easy to use way. The stress is put on flexibility and powerful configuration possibilities, the project is then easy to share between multiple environments/developers. For instructions read the ​installation document.

django-split-settings

See the ​blog post and ​GitHub project page for details. Main features:

• Settings can be split into files and directories.
• Files later in the list can modify configurations, for example add or remove apps from INSTALLED_APPS.
• The main settings file lists the files that make up the project’s settings.
• Files can be marked optional. Optional files can be used to override settings per instance.
• Wildcards can be used in file paths.
• Maintains support for Django’s runserver auto-reloading.

django-configglue

Use ini-style configuration files with support for overriding options from the commandline or via environment variables. See the ​quickstart guide for more details and checkout the ​repository if your curious about it.

Main features are:

• ini-style configuration files
• schema-based configuration
• command-line integration
• configuration validation
• multiple configuration files support with layering