Django

Code

Ticket #6390 (new)

Opened 6 months ago

Last modified 6 days ago

File storage backend for Amazon S3

Reported by: Gulopine Assigned to: david
Milestone: 1.0 beta Component: Uncategorized
Version: SVN Keywords: fs-rf-docs
Cc: cmawebsite@gmail.com, larlet@gmail.com, prufrocks@gmail.com, sebastian.serrano@gmail.com Triage Stage: Design decision needed
Has patch: 1 Needs documentation: 1
Needs tests: 1 Patch needs improvement: 1

Description

Based on the work in #5361, this ticket offers a storage backend for Amazon's Simple Storage Service.

Attachments

s3.py (2.8 kB) - added by Gulopine on 01/16/08 06:09:49.
Broken first draft
S3_6390.20080225.py (3.2 kB) - added by david on 03/04/08 07:33:05.
Functional S3 backend, use only Amazon's S3 lib (warning it overwrites filename)
S3_6390.20080321.py (3.2 kB) - added by david on 03/21/08 05:21:57.
New S3 storage which works with the latest storage patch (14) with new functions' names, I'm not totally satisfied with the way arguments are passed, I'll try to do it better

Change History

01/16/08 06:09:49 changed by Gulopine

  • attachment s3.py added.

Broken first draft

01/16/08 06:12:31 changed by Gulopine

  • needs_better_patch set to 1.
  • needs_tests set to 1.
  • needs_docs set to 1.

This first draft was started using Amazon's own library, but during the last sprint, I started to migrate it to boto, per Jacob's recommendation. As such, it's a bit between the two, and doesn't provide everything it needs anyway. I'm putting it up now to allow other people to flesh it out and make it work.

01/18/08 10:30:51 changed by CollinAnderson

  • cc set to cmawebsite@gmail.com.

01/20/08 19:01:00 changed by SmileyChris

  • stage changed from Unreviewed to Design decision needed.

Do we want this in trunk?

01/24/08 21:46:57 changed by empty

I feel that something like an s3 backend should not be in trunk. I think it stifles competition. I would rather the framework provide the hooks and let the plugable backends be provide by the community.

02/09/08 14:23:07 changed by david

  • cc changed from cmawebsite@gmail.com to cmawebsite@gmail.com, larlet@gmail.com.

02/25/08 02:46:33 changed by wkornewald

After having compared Amazon's S3 library and boto's S3 module I have to say that neither of them is really great. Amazon can't stream, but unlike boto it supports EU buckets and virtual hosts. Also, boto has a lot of redundant code and adding EU bucket support (in a clean way) would require quite a few refactorings.

Do you want the backend to make the browser upload via PUSH, so files don't have to be routed through the server (which would consume a lot of resources)? In that case, Amazon's library wins because boto's streaming support wouldn't be used. The best solution would of course be to merge boto with Amazon's library.

02/25/08 03:25:20 changed by david

After trying both libs too, I give up with boto because it looks really messy. The uploaded version requires only Amazon's S3 library. This is just a functional example from the broken first draft of Gulopine.

I think it could be interesting to have it in trunk (batteries included?).

02/28/08 04:56:38 changed by wkornewald

I've implemented two patches for boto: EU support (and vhosts, but haven't tested that) and an S3 code cleanup (though still not *perfect* :), but I don't know when they'll be committed. You can get them here: http://freenet-homepage.de/wkornewald/eu-bucket.diff http://freenet-homepage.de/wkornewald/simplified.diff

Or did you mean the API is messy? There are a few hacky places (e.g., set_key_class), but it doesn't look so bad to me.

BTW, why did you use CallingFormat?.REGULAR instead of SUBDOMAIN?

02/28/08 05:27:22 changed by david

Really interesting patches. I'll try to find the time to try at least the simplified one, yes I mean code cleanup by messy :).

I use REGULAR instead of SUBDOMAIN in order to be consistent with default arguments in S3.py but I always instantiate with CallingFormat.SUBDOMAIN for my own needs.

02/28/08 08:34:48 changed by wkornewald

If you take a look at EU bucket support please reload my patch. It now passes all tests and has a nasty connection cache.

03/02/08 05:03:46 changed by wkornewald

EU bucket support is now in boto trunk. Next, I'll send my simplification patch for review.

03/04/08 07:33:05 changed by david

  • attachment S3_6390.20080225.py added.

Functional S3 backend, use only Amazon's S3 lib (warning it overwrites filename)

03/18/08 11:32:34 changed by Gulopine

  • keywords changed from fs-rf to fs-rf-docs.

03/20/08 21:57:03 changed by Gulopine

Just a quick note to anybody working on this, or even just watching it. The most recent patch to #5361, which is likely to be applied to trunk soon, renames most of the things that this patch relies on, as well as how it must be instantiated. The docs in #5361 have been updated to reflect these new requirements, but the best reference will be looking at the django.core.filestorage.filesystem.FileSystemStorage class.

03/21/08 05:21:57 changed by david

  • attachment S3_6390.20080321.py added.

New S3 storage which works with the latest storage patch (14) with new functions' names, I'm not totally satisfied with the way arguments are passed, I'll try to do it better

03/31/08 10:37:10 changed by brosner

  • status changed from new to closed.
  • resolution set to fixed.

(In [7393]) newforms-admin: Fixed #6390 -- Added a missing import in django/newforms/formsets.py. Thanks, Jurian Botha.

03/31/08 10:38:13 changed by brosner

  • status changed from closed to reopened.
  • resolution deleted.

Opps, my bad. A bit of dyslexia ;) I meant #6930.

04/03/08 16:23:28 changed by anonymous

Do you plan to switch to boto with your code?

04/07/08 06:23:07 changed by david

  • owner changed from nobody to david.
  • status changed from reopened to new.

That's a possible option. I already use boto for SQS so I plan to switch to a full boto Amazon access in a "near" future, let me know if you need it quickly.

04/16/08 01:23:22 changed by anonymous

  • cc changed from cmawebsite@gmail.com, larlet@gmail.com to cmawebsite@gmail.com, larlet@gmail.com, prufrocks@gmail.com.

06/16/08 13:30:46 changed by Gulopine

  • milestone set to 1.0 beta.

06/23/08 08:41:02 changed by david

Latest version of this patch is now hosted at http://code.larlet.fr/django-storages/

Do not hesitate to add your own storage if you want or contact me.

06/29/08 23:10:48 changed by sserrano <sebastian.serrano@gmail.com>

  • cc changed from cmawebsite@gmail.com, larlet@gmail.com, prufrocks@gmail.com to cmawebsite@gmail.com, larlet@gmail.com, prufrocks@gmail.com, sebastian.serrano@gmail.com.

Add/Change #6390 (File storage backend for Amazon S3)




Change Properties
Action