Opened 9 years ago

Closed 22 months ago

Last modified 12 days ago

#26029 closed New feature (fixed)

Provide an API to configure arbitrary file storage backends

Reported by: Aymeric Augustin Owned by: Jarosław Wygoda
Component: File uploads/storage Version: dev
Severity: Normal Keywords:
Cc: django@… Triage Stage: Accepted
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

Currently Django has two hardcoded file storage backends, "default" (for media files) and "static" (for static files). Idioms for configuring file storage backends in pluggable apps are cumbersome.

It would be nice to be able to configure arbitrary file storage backends, like caches and databases e.g.:

FILE_STORAGES = {
   'media': {
       'BACKEND': settings.DEFAULT_FILE_STORAGE,
       'OPTIONS': {
            'location': settings.MEDIA_ROOT,
            'base_url': settings.MEDIA_URL,
            # possible override of settings.FILE_CHARSET
       },
   },
   'static': {
       'BACKEND': settings.STATICFILES_STORAGE,
       'OPTIONS': {
            'location': settings.STATIC_ROOT,
            'base_url': settings.STATIC_URL,
            # replacement for STATICFILES_FINDERS and STATICFILES_DIRS that would look a lot like template loaders
            # possible override of settings.FILE_CHARSET
       },
   }

This was discussed on django-developers: https://groups.google.com/d/msg/django-developers/gEbFApLLuzg/IW1LDUwmEgAJ

There were some concerns about introducing another large dict in settings. The general ideas was uncontroversial.

Change History (24)

comment:1 by Keryn Knight, 9 years ago

For what it's worth, bmispelon and I discussed something akin to this recently, out of which I started implementing something similar entirely as an experiment. One thing I liked about where I went with that was that a named storage key in the dictionary was a dotted path, and behaved somewhat like a logging config in that the nearest 'parent' match such that each model field which uses a storage could be addressed separately, like so:

class Test(Model):
    a = FileField(storage='myapp.test.a')
    b = ImageField(storage='myapp.test.b')

and then if myapp.test.a or myapp.test.b were in the configuration dictionary, they would be used, but if not, and myapp.test was, that would be used, and so on walking backwards up the dotted path (falling back to ostensibly default [''] in the end)

The notion there was third-party apps could namespacing storages per-field, but at a project level one only has to opt-in as much as one cares (ie: myapp, or myapp.test, or default etc.) to have finegrained control.

comment:2 by Keryn Knight, 9 years ago

Cc: django@… added

comment:3 by Sasha Gaevsky, 9 years ago

Configurable file storage backends was already proposed to introduce in terms of task #23251.

comment:4 by Sasha Gaevsky, 9 years ago

Owner: changed from nobody to Sasha Gaevsky
Status: newassigned

in reply to:  description comment:5 by Sasha Gaevsky, 9 years ago

Replying to aaugustin:

FILE_STORAGES = {
   'media': {
       'BACKEND': settings.DEFAULT_FILE_STORAGE,
       'OPTIONS': {
            'location': settings.MEDIA_ROOT,
            'base_url': settings.MEDIA_URL,
            # possible override of settings.FILE_CHARSET
       },
   ...
}

Media backend will have media key in settings, not default, to keep it consistent with the current settings?

comment:6 by Sasha Gaevsky, 9 years ago

As proposed Aymeric Augustin in the original PR, I started to compose DEP, so now it's in progress and I hope to finish it sometime soon.

comment:7 by Sasha Gaevsky, 7 years ago

Owner: Sasha Gaevsky removed
Status: assignednew

comment:8 by Jarosław Wygoda, 3 years ago

Owner: set to Jarosław Wygoda
Status: newassigned

comment:9 by Jarosław Wygoda, 3 years ago

I'd like to introduce a file storage registry similar to BaseConnectionHandler (django/utils/connection.py) and EngineHandler (django/template/utils.py).

Example settings.py snippet:

STORAGES = {  # rename to FILE_STORAGES to make it more explictit?
    'example': {
        'BACKEND': 'django.core.files.storage.FileSystemStorage',
        'OPTIONS': {
            'location': '/example',
            'base_url': '/example/',
        },
    },
}

Changes introduced by this pr are backward compatible. Users can still use existing settings to configure static and media storages.

Currently storages can be retrieved from the following objects:

django/core/files/storage.py:

  • get_storage_class
  • DefaultStorage
  • default_storage

django/contrib/staticfiles/storage.py:

  • ConfiguredStorage
  • staticfiles_storage

What do you think about deprecating them?

I'll write tests and docs if this approach is acceptable.

https://github.com/django/django/pull/15610

comment:10 by Claude Paroz, 3 years ago

Has patch: set
Needs tests: set

comment:11 by Mariusz Felisiak, 2 years ago

Patch needs improvement: set

comment:12 by Mariusz Felisiak <felisiak.mariusz@…>, 2 years ago

In 9e7cb27:

Refs #26029 -- Doc'd django.core.files.storage.default_storage.

comment:13 by Mariusz Felisiak <felisiak.mariusz@…>, 2 years ago

In 04ec8bf9:

[4.1.x] Refs #26029 -- Doc'd django.core.files.storage.default_storage.

Backport of 9e7cb27a5b7363239d1db02d29fe12efdf25b899 from main

comment:14 by Carlton Gibson, 2 years ago

Needs tests: unset
Patch needs improvement: unset

Unchecked flags, for a new review.

Last edited 2 years ago by Carlton Gibson (previous) (diff)

comment:15 by Carlton Gibson, 23 months ago

Patch needs improvement: set

PR looks promising, but have comments outstanding ref adding the new STORAGES defaults.

comment:16 by Carlton Gibson, 23 months ago

Patch needs improvement: unset

I think the PR looks close. (I suggested a few docs tweaks)

Main remaining point (for me) is being sure about the signal handling with override_settings, and the usual Settings/UserSettingsHolder complexities.

comment:17 by Mariusz Felisiak <felisiak.mariusz@…>, 22 months ago

In d16079d:

Refs #26029 -- Added LazySettings._show_deprecation_warning() hook.

comment:18 by Mariusz Felisiak <felisiak.mariusz@…>, 22 months ago

Resolution: fixed
Status: assignedclosed

In 1ec3f09:

Fixed #26029 -- Allowed configuring custom file storage backends.

comment:19 by Mariusz Felisiak <felisiak.mariusz@…>, 22 months ago

In 32940d3:

Refs #26029 -- Deprecated DEFAULT_FILE_STORAGE and STATICFILES_STORAGE settings.

comment:20 by Mariusz Felisiak <felisiak.mariusz@…>, 14 months ago

In f72f420:

Refs #26029 -- Removed DEFAULT_FILE_STORAGE and STATICFILES_STORAGE settings.

This also removes django.core.files.storage.get_storage_class().

Per deprecation timeline.

comment:21 by GitHub <noreply@…>, 13 months ago

In fa6e6f31:

[5.0.x] Refs #26029 -- Improved get_storage_class() deprecation warning with stacklevel=2.

Addition of the stacklevel argument shows the source of the
deprecated call, making updating the client code simpler.

comment:22 by Mariusz Felisiak <felisiak.mariusz@…>, 13 months ago

In 0cbc92bc:

[4.2.x] Refs #26029 -- Improved get_storage_class() deprecation warning with stacklevel=2.

Addition of the stacklevel argument shows the source of the
deprecated call, making updating the client code simpler.

Backport of fa6e6f31137ee4025adfb27f88c07eebd488d446 from stable/5.0.x.

comment:23 by GitHub <noreply@…>, 12 days ago

In 6dcab75d:

Refs #26029 -- Extended docs for the StorageHandler default instance.

Third-party packages that provide storages need to rely on the
StorageHandler API in order to allow users to use the storages module
instance to override defaults. Minimally documenting these methods
allows package authors to rely on them.

Co-authored-by: Sarah Boyce <42296566+sarahboyce@…>

comment:24 by Natalia <124304+nessita@…>, 12 days ago

In 56ad530a:

[5.1.x] Refs #26029 -- Extended docs for the StorageHandler default instance.

Third-party packages that provide storages need to rely on the
StorageHandler API in order to allow users to use the storages module
instance to override defaults. Minimally documenting these methods
allows package authors to rely on them.

Co-authored-by: Sarah Boyce <42296566+sarahboyce@…>

Backport of 6dcab75d5d8c2ef18de15323930057e6fa9ec00f from main.

Note: See TracTickets for help on using tickets.
Back to Top