Opened 5 years ago

Closed 3 years ago

#13053 closed New feature (wontfix)

Support for non-static DB connection info

Reported by: Lynge Owned by: nobody
Component: Database layer (models, ORM) Version: master
Severity: Normal Keywords:
Cc: Triage Stage: Someday/Maybe
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

It would be great if we could specify a class or function as the source of the DB-connection information.

I am thinking something like:

DATABASES = {
    'default': {
        'INFO_FINDER': 'myproject.myDBFinderApp.MyDBFinder',
    },
    'users': {
        'NAME': 'user_data',
        'ENGINE': 'django.db.backends.mysql',
        'USER': 'mysql_user',
        'PASSWORD': 'priv4te'
    }
}

This would then get executed every time something was needed from the 'default' DB.

Where would this be useful?
In my current project I am building a webapp where each client gets their own subdomain and is free to create users inside it. It would be very useful to shard data by this subdomain since my clients never have to communicate via the webapp. This means that I have to be able to create a new DB and connect to it whenever a new client signs up. This is not possible with the current setup using settings.py.

This method of dividing clients is not new at all so I guess there is many more than myself that will benefit from this, but more importantly (I think) is that it will most likely cover any other corner-case that Django is not capable of, since it will be so easy to just write your own DBFinder and link it. Any DBFinder will, of course, have to always return the connection info in the correct format just like to one normally used in settings.

I have tried to do it myself, but I just cant make any sense of it and have no idea where to do the check to see if settings give connection-info or just a place to get it.
But I cannot imagine it would be any huge amount of work for the people who wrote the thing or just about anyone with greater skills in Django-development than my own.

Attachments (1)

a_s-draft.diff (1.7 KB) - added by Lynge 5 years ago.
Initial draft of the idea. Most likely not commit-ready.

Download all attachments as: .zip

Change History (9)

comment:1 Changed 5 years ago by russellm

  • Needs documentation unset
  • Needs tests unset
  • Patch needs improvement unset
  • Triage Stage changed from Unreviewed to Someday/Maybe

Accepting the use case; not necessarily sold on the proposed implementation approach.

It might be possible to do right now by replacing the entire settings dictionary object with an object that implements the dict interface.

The bigger issue will be getting access to useful context information (e.g., the domain where an request originates) to be able to shard on that basis. I'm not sure I see an easy way to get access to this data where it will be required.

comment:2 Changed 5 years ago by Lynge

I will look into your suggestion on the settings dictionary object replacement.

On the subject of useful context information:
I have found something called django_globals
It is very simple. It just uses threading.local() to store values, but if it will work everywhere I have no idea.

I have been testing it on uWSGI with Cherokee in front and it seems to be behaving as one would want, but I have no idea if it is a viable method for any kind of setup. E.g. if it will share the values between requests in certain kinds of setups.

Changed 5 years ago by Lynge

Initial draft of the idea. Most likely not commit-ready.

comment:3 Changed 5 years ago by Lynge

And now, 8 months later, I finally get around to looking at it.
I am submitting a draft for, what I have called, ActiveSettings

This is just meant to be a hook that will let external projects get a chance to change the settings as they are being requested.
If this, or something like it, could be implemented, then I can just construct an external project that will let people solve this problem of sharding costumers and without getting in the way of anyone that will not be needing it, which I suspect is the largest part of the Django user-base.

The diff is just a draft since this is my first contribution to Django, I thought I would post it for scrutiny before doing the docs and tests. I am sure there is a lot of errors in it, but the idea should be clear.

Also, should I be taking this up on the mailing list? I read something about that in the contribution guide.

comment:4 Changed 4 years ago by lukeplant

  • Type set to New feature

comment:5 Changed 4 years ago by lukeplant

  • Severity set to Normal

comment:6 Changed 4 years ago by aaugustin

  • UI/UX unset

Change UI/UX from NULL to False.

comment:7 Changed 4 years ago by aaugustin

  • Easy pickings unset

Change Easy pickings from NULL to False.

comment:8 Changed 3 years ago by aaugustin

  • Resolution set to wontfix
  • Status changed from new to closed

This is a subset of the general problem of multi-tenancy (#15089).

The approach taken by the patch doesn't work because Django maintains internal caches that depend on the values of settings. If you change the active settings between two requests handled by the same Python process, these caches may hold stale values, with undefined results.

The settings_changed signal might help, but it's only designed for tests at this time, and I'm not sure it covers all settings at this time.

I'm going to close this ticket because I don't think the solution to multi-tenancy will come from hacks in the settings.

Note: See TracTickets for help on using tickets.
Back to Top