django.conf.settings is hard to override

I definitely +1 this. It won't just make using Paste easier (as I suppose that's why Ian is looking at the code), but will make other things possible (I especially like the idea of configuration switching). And it might even - with a bit of monkey patching - make it possible to run multiple Django apps within one server context, which could be really helpful if you want to use Django apps within a larger WSGI setup.

Good call. Let's make this happen.

I'm not sure whether it'd be possible to write it in a backwards-compatible way, because some code does this:

from django.conf.settings import SOME_SETTING

That said, we shouldn't let backwards-compatibility stand in the way, as this would be an important improvement.

54 occurences of that in 46 files. Sounds like something that could be cleaned up in a few hours in magic-removal. After that cleanup it shouldn't be too problematic to switch over to a settings object instead of a settings module. Adrian: feel free to assign the ticket to me (or other tickets, to keep you free for dev stuff) if you think I should take up on it.

Hugo -- it's all yours. :) BTW, you have commit access on magic-removal, so have at it!

Before you do any code, though, we should spend a tiny bit more time on design. What were you thinking the interface would be?

The first thing I would do would only be going through all the code to change any "from django.conf.settings import SETTING" into "from django.conf import settings" and then qualifying all those settings in the file itself with "settings." - that's not much of an interface, just stupid editing ;-)

The second step would be the settings object, of course - and I agree that we should throw some thinking into that. My preference would be to have a threadlocal that's automatically populated by loading the config. That would allow to do SIGHUP stuff by just running the settings-loading-function in every thread or process of a server (for example it can be done in the FLUP framework by signalling all childs when the master receives a SIGHUP). That way you could have nice "life transitions" of settings, because you don't need to kill the server. Or do it like 'apachectl graceful' - stopping threads/processes when they are done handling the current request and restarting them fresh (so they reload the config). The exact way to do the reload is up to the server, but the interface would have to provide the needed functionality of config reloading.

One way I did something like this in another project (TooFpy?): the config still was in a standard module file. The difference was just the settings loader, which pulled the globals out of that settings module into the (in that project global) configuration object. That way the settings themselves won't change in their syntax, it's just the internal semantics that would switch to a settings object. This would especially keep the nice "from othersettings import *" mechanism intact.

One thing we would need to address with that mechanism is module caching, of course - so either we would need to use compile/exec instead of importing, or remove modules from sys.modules after doing the import, or using reload on modules to make sure we get the most current version. This does get complicated with projects where settings are spread over multiple modules, though - we need to know what modules to reload (or what modules to remove, of course).

Removing after import is much simpler, as we just could keep old sys.modules values and just remove those that are new. The whole config loading would have to be secured by a lock, though, to make sure that no two threads be in the same code segment.

Sounds good. Go for it!

Please remember to follow the new "Committing code" guidelines -- keep commits as granular as possible, etc. I just added that bit of docs/policy the other day.

Another general issue for the swappability of configuration is places where the configuration is used on module import. I believe that django.core.db in particular loads up DATABASE_ENGINE and sets things based on that.

Anyway, as far as actually swapping settings, there's some code in djangopaste.wsgi that does this, if you want to look at it (it actually effectively swaps the module itself). But that doesn't really help when it comes to configuration that is used at import time -- to handle that each place where that happens would have to have a threadlocal value put in, and swapped at runtime.

A first step is done now by rewriting all settings accessing parts to allways going through "settings.". Next step would be to change settings from a module to an object. The last step would be to collect places where settings are captured at import time.

(In [2031]) Refs #1212 - moved settings from a dedicated module into a dedicated global instance

I close this for now, if somebody stumbles over settings that are captured on load time and not reloaded later on, please open up new tickets for the specific places where this happens.