Opened 16 years ago

Closed 16 years ago

Last modified 12 years ago

#6390 closed (wontfix)

File storage backend for Amazon S3

Reported by: Marty Alchin Owned by: David Larlet
Component: Uncategorized Version: dev
Severity: Keywords: fs-rf-docs
Cc: cmawebsite@…, larlet@…, prufrocks@…, sebastian.serrano@… Triage Stage: Design decision needed
Has patch: yes Needs documentation: yes
Needs tests: yes Patch needs improvement: yes
Easy pickings: UI/UX:

Description

Based on the work in #5361, this ticket offers a storage backend for Amazon's Simple Storage Service.

Attachments (3)

s3.py (2.8 KB ) - added by Marty Alchin 16 years ago.
Broken first draft
S3_6390.20080225.py (3.2 KB ) - added by David Larlet 16 years ago.
Functional S3 backend, use only Amazon's S3 lib (warning it overwrites filename)
S3_6390.20080321.py (3.2 KB ) - added by David Larlet 16 years ago.
New S3 storage which works with the latest storage patch (14) with new functions' names, I'm not totally satisfied with the way arguments are passed, I'll try to do it better

Download all attachments as: .zip

Change History (30)

by Marty Alchin, 16 years ago

Attachment: s3.py added

Broken first draft

comment:1 by Marty Alchin, 16 years ago

Needs documentation: set
Needs tests: set
Patch needs improvement: set

This first draft was started using Amazon's own library, but during the last sprint, I started to migrate it to boto, per Jacob's recommendation. As such, it's a bit between the two, and doesn't provide everything it needs anyway. I'm putting it up now to allow other people to flesh it out and make it work.

comment:2 by Collin Anderson, 16 years ago

Cc: cmawebsite@… added

comment:3 by Chris Beaven, 16 years ago

Triage Stage: UnreviewedDesign decision needed

Do we want this in trunk?

comment:4 by empty, 16 years ago

I feel that something like an s3 backend should not be in trunk. I think it stifles competition. I would rather the framework provide the hooks and let the plugable backends be provide by the community.

comment:5 by David Larlet, 16 years ago

Cc: larlet@… added

comment:6 by Waldemar Kornewald, 16 years ago

After having compared Amazon's S3 library and boto's S3 module I have to say that neither of them is really great. Amazon can't stream, but unlike boto it supports EU buckets and virtual hosts. Also, boto has a lot of redundant code and adding EU bucket support (in a clean way) would require quite a few refactorings.

Do you want the backend to make the browser upload via PUSH, so files don't have to be routed through the server (which would consume a lot of resources)? In that case, Amazon's library wins because boto's streaming support wouldn't be used. The best solution would of course be to merge boto with Amazon's library.

comment:7 by David Larlet, 16 years ago

After trying both libs too, I give up with boto because it looks really messy. The uploaded version requires only Amazon's S3 library. This is just a functional example from the broken first draft of Gulopine.

I think it could be interesting to have it in trunk (batteries included?).

comment:8 by Waldemar Kornewald, 16 years ago

I've implemented two patches for boto: EU support (and vhosts, but haven't tested that) and an S3 code cleanup (though still not *perfect* :), but I don't know when they'll be committed. You can get them here:
http://freenet-homepage.de/wkornewald/eu-bucket.diff
http://freenet-homepage.de/wkornewald/simplified.diff

Or did you mean the API is messy? There are a few hacky places (e.g., set_key_class), but it doesn't look so bad to me.

BTW, why did you use CallingFormat.REGULAR instead of SUBDOMAIN?

comment:9 by David Larlet, 16 years ago

Really interesting patches. I'll try to find the time to try at least the simplified one, yes I mean code cleanup by messy :).

I use REGULAR instead of SUBDOMAIN in order to be consistent with default arguments in S3.py but I always instantiate with CallingFormat.SUBDOMAIN for my own needs.

comment:10 by Waldemar Kornewald, 16 years ago

If you take a look at EU bucket support please reload my patch. It now passes all tests and has a nasty connection cache.

comment:11 by Waldemar Kornewald, 16 years ago

EU bucket support is now in boto trunk. Next, I'll send my simplification patch for review.

by David Larlet, 16 years ago

Attachment: S3_6390.20080225.py added

Functional S3 backend, use only Amazon's S3 lib (warning it overwrites filename)

comment:12 by Marty Alchin, 16 years ago

Keywords: fs-rf-docs added; fs-rf removed

comment:13 by Marty Alchin, 16 years ago

Just a quick note to anybody working on this, or even just watching it. The most recent patch to #5361, which is likely to be applied to trunk soon, renames most of the things that this patch relies on, as well as how it must be instantiated. The docs in #5361 have been updated to reflect these new requirements, but the best reference will be looking at the django.core.filestorage.filesystem.FileSystemStorage class.

by David Larlet, 16 years ago

Attachment: S3_6390.20080321.py added

New S3 storage which works with the latest storage patch (14) with new functions' names, I'm not totally satisfied with the way arguments are passed, I'll try to do it better

comment:14 by Brian Rosner, 16 years ago

Resolution: fixed
Status: newclosed

(In [7393]) newforms-admin: Fixed #6390 -- Added a missing import in django/newforms/formsets.py. Thanks, Jurian Botha.

comment:15 by Brian Rosner, 16 years ago

Resolution: fixed
Status: closedreopened

Opps, my bad. A bit of dyslexia ;) I meant #6930.

comment:16 by anonymous, 16 years ago

Do you plan to switch to boto with your code?

comment:17 by David Larlet, 16 years ago

Owner: changed from nobody to David Larlet
Status: reopenednew

That's a possible option. I already use boto for SQS so I plan to switch to a full boto Amazon access in a "near" future, let me know if you need it quickly.

comment:18 by anonymous, 16 years ago

Cc: prufrocks@… added

comment:19 by Marty Alchin, 16 years ago

milestone: 1.0 beta

comment:20 by David Larlet, 16 years ago

Latest version of this patch is now hosted at http://code.larlet.fr/django-storages/

Do not hesitate to add your own storage if you want or contact me.

comment:21 by sserrano <sebastian.serrano@…>, 16 years ago

Cc: sebastian.serrano@… added

comment:22 by Marty Alchin, 16 years ago

I've put up a new patch on #5361 that pretty much destroys whatever patches are left on this ticket. Also, that patch makes it a bit less straightforward to include other backends in Django core, so this is more likely now to be closed as a wontfix. That's still awaiting official word, but that's how it looks at the moment.

comment:23 by David Larlet, 16 years ago

As I previously said, I have no problem with that and that's why I set up an external repository (I know you know, my comment is just here for the "official" decision).

comment:24 by Michael Radziej, 16 years ago

not a bug => not milestone 1.0 beta.

comment:25 by Marty Alchin, 16 years ago

Bugs aren't the only things planned for 1.0 beta. This is being partially addressed as part of #5361, which is planned to make it into 1.0 beta. Even though it won't be included directly, these types of backends will be quite possible once that goes in. Per Jacob, all such related tickets should be placed in the 1.0 beta milestone.

comment:26 by Gary Wilson, 16 years ago

Resolution: wontfix
Status: newclosed

I am of the opinion that this should be implemented outside of django core in some sort of Amazon library for Django. A quick search turns up django-aws and boto. Might be others. Once such a library matured, then we could maybe think about inclusion into django.contrib.

comment:27 by Jacob, 12 years ago

milestone: 1.0 beta

Milestone 1.0 beta deleted

Note: See TracTickets for help on using tickets.
Back to Top