With #5361 ready for review, this page provides a basic rundown of what changed, in case the code doesn't easily expose all the improvements.
- All file handling is now done through storage classes, which handle basic interactions with an underlying storage system. Django will ship with just one, which deals with the filesystem just like everything works now, but users can create whatever other backends they like. Examples could be storing files on S3 (a common request), customizing file naming behavior, encrypting/decrypting files transparently, etc. The docs on the ticket do a good job of explaining how to use them, so I won't bore you with that here. The default storage system is
FileSystemStorage, and is specified by the new
FileFieldnow provides a
FieldFileobject instead just a filename, so that file operations can take place on it directly. This was needed on one hand to move people away from the
open(instance.get_avatar_filename()), since that won't work once other backends enter the picture. Instead,
instance.avatarcan be used as a file-like object directly. Also, it has
urlproperties instead of
open()method of storage objects also accepts a
mixinargument, which allows the returned
Fileto have overrides and extra methods for specific file types.
FileSystemBackendwon't allow access to any file that's not beneath the path it was instantiated with. This is primarily useful for security, but also as a deterrent against accidentally putting a leading slash in
- Speaking of
upload_to, it now accepts a callable as well as a string. If a callable is provided, it's called with the model instance and the filename, so user code has much greater control over how files are named.
- The current default storage system will always be available, both as a class and as an object, from
django.core.files.storage. The class is
DefaultStorageand the instance of it is
default_storage. This way, subclasses can override just some behavior, such as file naming, without worrying about how the files are really being stored, or views can save/retrieve files manually, with the same flexibility.
FileFieldalso accepts a
storageargument, where a custom storage object can be passed in, to be used as an override of
Differences from previous patches
django.core.filestoragefrom older patches is gone, and everything has been moved into
django.core.files, in keeping with the trend set by . The basic File object is at django.core.files.base, while all storage-related classes and functions are at
django.core.files.storage. This means that, by default, all new storage systems would start as third-party apps, since there's no longer a "dedicated" place for them.
- There's now a single base
django.core.files.base.Fileclass for all file types, regardless of whether they come from a storage system, an upload, or whatever. This means, for instance, that all storage-related operations are now capable of chunking, while all uploaded files also have
__nonzero__based on file.name. Both
FieldFilehave customizations on top of it though, so it's not like
Fileis built to be all things to all people.
- Other image-related functionality has also been moved to
django.utils.images, in the form of
ImageFile, a mixin that provides width and height options.
- The API for getting meta-information about a file (such as its size, filesystem path, URL, width and height) has changed from methods to read-only properties. Those that would have to access the content (size, width and height) cache the results so they don't have to do so more than necessary.
I've tried very hard to maintain backwards-compatibility wherever reasonable, but there are still a few places where API improvements merit some changes. In addition, there's one (hopefully) rare case where backwards-compatbility is impossible to retain.
Most of the
Model._get_FIELD_* methods have been deprecated, pointing to the appropriate attributes on the new
FieldFile instance. Given a model like this:
class FileContent(models.Model): content = models.FileField(upload_to='content')
Here's how the changes map out:
|Old way||New way|
| || |
| || |
| || |
| || |
| || |
| || |
ImageField have moved
Because of all the new code in an already-crowded
ImageField have been moved into their own module:
django.db.models.fields.files. They're still imported directly into
django.db.models, so the vast majority of existing code will continue to work without modification, but if anyone's importing them directly, they'll need to update the import path.
django.utils.images has moved
The new location is
django.core.files.images, where it's far more appropriate. An
import and a
DeprecationWarning have been left at the old location.
File to save raw content
Passing raw file content as strings to
save() has been deprecated, in favor of using a
File subclass. In addition to a
DeprecationWarning and an automatic conversion, there's now a
django.core.files.base.ContentFile, which is a simpler class than
SimpleUploadedFile, as it doesn't deal with content-type, charset or even a filename. It's basically just a light wrapper around
StringIO that adds chunking behavior, since most of the internals expect to be able to use that.
FileField values are no longer
FileField will always provide an object to model instances, regardless of whether there's actually a file associated with it, which is necessary for the
instance.content.save() behavior. Previously, if there was no file attached,
instance.content would be
None, which is no longer true, so the following will no longer work:
if instance.content is not None: # Process the file's content here.
File objects evaluate to
False on their own, so the following is functionally identical:
if instance.content: # Process the file's content here.
FileField can't be
The exact behavior of these with a
FileField was undefined, and was causing problems in many cases, so they now raise a
TypeError if supplied.
There are a number of tickets marked fs-rf, indicating that they're impacted by this patch. First, the issues that are truly fixed, and can be marked as
fixed in the commit:
- #5361 - The main file storage ticket, where the patch itself resides.
- #3621 - If
upload_tostarts with a slash,
FileSystemStorage's increased security will now raise a
SuspiciousOperationwhen saving a file, long before it hits the database.
- #5655 - The supplied patch was adapted and included, resolving the issue. Tests have been included to verify this.
- #7415 - Saved files are now always stored in the database using forward slashes, and retrieving using
And there are also a few that will be made possible, but not provided in core directly (probably mark as
- #2983 - Since saving and deleting behavior has been moved into
Model, a subclass can provide this behavior. If not that way, a custom storage object can do the rest, passing it into the
- #4339 - By providing a custom storage class, it's easy to change this type of file naming behavior. The patch's tests include a Trac-style example, using numbers instead of underscores.
- #4948 - If this is even still an issue (see this comment), more fine-grained locking can be provided by a custom storage class.
- #5485 - Like #4339 above, custom file naming across the board is easy with a custom storage class.
- #5966 - Custom storage can create or delete directories however is necessary for a given environment.
- #6390 - Custom backends will be quite possible, but are best suited as third-party apps.
And a couple where the problem was resolved by removing the feature that was causing problems (I'm not sure if these should be
- #3567 - Since
coreis no longer allowed on
FileField, this situation is no longer valid.
- #4345 - Since
uniqueis no longer allowed on
FileField, this situation is no longer valid.
Finding alternative storages
django-storages is a project which plans to provide storages for different systems. For the moment, there is only a storage for the S3 backend, this is a work in progress. Please let me know if you want to participate and/or to add your own storage. Basic documentation is provided as-is (only tests for now). Much more to come this week-end.