Upload Progress Middleware and Integration into the Admin

The Middleware and JavaScript

Start of new improved version

Øyvind Saltvik, that last patch doesn't belong here (at least I think). If you read the description in #2070, it clearly states a problem, and we should fix that problem in that ticket. I meant for this ticket to be for the stuff *beyond* #2070. (i.e. middleware and javascript)

The point is that we should be careful in how we design #2070, but instead of doing both at the same time, make #2070 powerful enough that we can design this ticket with a good API (provided in #2070) in mind.

The above comment was remarking on attachment '5099-streaming_file_upload_with_safe_file_move_progress.diff'. Also, I didn't mean to clear needs_better_patch or any of the others.

Ok, I see and agree :)

Pretty Admin Upload Survives!

Alright, I've cleaned up ideas from #2070 and reformulated to this patch (the JavaScript is left unchanged). Here's how you can use it:

1. Add the patch from #2070 (e.g. 5100_file_upload_core.diff:

patch -p0 < 5100_file_upload_core.diff

2. Add this patch (same as above)

3. Create the model:

from django.db import models

# Create your models here.
class FileList(models.Model):
    name = models.CharField(maxlength=255)
    email = models.EmailField()

    class Admin:
        js = ('/admin/media/js/UploadProgress.js',)

class AFile(models.Model):
    descr = models.CharField(maxlength=255,core=True)
    file = models.FileField(upload_to='files')
    inlist = models.ForeignKey(FileList,edit_inline=models.STACKED)

4. Add django.middleware.upload.UploadStateMiddleware to your MIDDLEWARE_CLASSES setting (at the beginning, possibly)

5. Set FILE_UPLOAD_DIR in your settings file to wherever you want to store temporary files.

6. Enjoy :)

TODO:

* Test

* Write tests/docs

* Write widget for newforms (that generates uuid)

Follow above step, not work with

KeyError? at /file/upload/ 'UPLOAD_PROGRESS_ID' Request Method: POST Exception Type: KeyError? Exception Value: 'UPLOAD_PROGRESS_ID' Exception Location: C:\Python24\lib\site-packages\django\http\init.py in init, line 193

For your info, -100_file_upload_core.diff -5100_admin_progress_bar.diff -Fast cgi mode Cheer

Replying to anonymous:

Follow above step, not work with ... Cheer

Thank you! It was a line I forgot to add to wsgi.py, should be set now. Please use 5113_file_upload_core_with_middleware_hooks.diff in #2070.

New and improved version!

With the latest patch, there's a new process for using this:

1. Add the patch from #2070 (e.g. 5113_file_upload_core_with_middleware_hooks.diff)

patch -p0 < 5113_file_upload_core_with_middleware_hooks.diff

2. Add this patch:

patch -p0 < 5116_new_admin_progress_bar.diff

3. Create the model:

from django.db import models

# Create your models here.
class FileList(models.Model):
    name = models.CharField(maxlength=255)
    email = models.EmailField()

    class Admin:
        pass

class AFile(models.Model):
    descr = models.CharField(maxlength=255,core=True)
    file = models.FileField(upload_to='files')
    inlist = models.ForeignKey(FileList,edit_inline=models.STACKED)

5. Set FILE_UPLOAD_DIR in your settings file to wherever you want to store temporary files.

The changes between this one and the last version:

• No MiddleWare required. There's no reason to have middleware, the request has to deal with file uploads anyway, it should deal with it in a way that will allow views to use it. (If anyone has any differing opinions, let me know.)

• No js = () in the class Admin required. It's now an integral part of the oldforms Admin. What's next? newforms-admin.

• The JavaScript now deals with stopping and restarting uploads in a useful way.

Newforms-Admin branch patch

Reworked a lot of JS, this is for newforms-admin.

Reworked a lot of JS, this is for oldforms-admin.

Wrote some example MiddleWare and such. Using caching makes a lot of sense for upload progress.

I reworked the JavaScript from its original carnation. The main goal that I think is reasonably tackled is the case where the server doesn't handle file upload progress well. If it fails, it won't display a broken progress bar, it will just fail mostly silently. The JavaScript? is also more a little bit cleaner and more careful.

FYI

Not work for IE6,javascript error

Cheer

On IE6,

An ajax request is not work, I try this;

function createRequestObject(){

var req = false;

if(window.XMLHttpRequest){

req = new XMLHttpRequest();

if(req.overrideMimeType){ req.overrideMimeType('text/xml'); }

} else if(window.ActiveXObject){

try{ req = new ActiveXObject("Msxml2.XMLHTTP"); } catch(e){

try{ req = new ActiveXObject("Microsoft.XMLHTTP"); } catch(e){}

}

}

if(!req){

return false;

} else{ return req; }

}

It work for first request and stop after that. The view function never reach by an ajax call.

Any comment?

I am currently using the xmlhttp function given in admin/js/core.js . I don't think that's the problem. I'll look around in IE to see where the problem is and how I can fix it.

Hi,

Hoola, I search a google and found that IE not allow calling same uri repeatively, So, I append a random number after uri, it work.

http://www.robertnyman.com/2007/04/04/weird-xmlhttprequest-error-in-ie-just-one-call-allowed/

Thank

Works in IE etc

Thank you previous poster (anonymous), adding an incremental (why random?) number after the URI was one of the several changes I've made in order to get this to work in IE (5.5,6, and 7).

Thanks to all who have been testing, this new patch should be a great improvement -- especially for those using IE. However, there are a few things that need to be changed.

Since #2070 has now been stripped of its default FileProgressDescriptor, you need to add some middleware/add models to get this to work.

INSTALLATION

1. Add #2070 to your source tree

2. Add the latest patch to your source tree.

3. Setup your middleware/models to track file progress.

4. Set FILE_UPLOAD_DIR to a writable folder in your settings.py file.

For (3), there are two options that come in the patch (as uploadprogress contrib):

• FileProgressStore -- uses the cache framework for quick ajax (will NOT work if your cache framework doesn't)
• FileProgressDB -- uses Database models

Depending on what you choose, continue below.

FileProgressStore

To enable this, add 'django.contrib.uploadprogress.middleware.FileProgressStore' to your MIDDLEWARE_CLASSES.

FileProgressDB

To enable this:

• Add 'django.contrib.uploadprogress.middleware.FileProgressStore' to your MIDDLEWARE_CLASSES.
• Add 'django.contrib.uploadprogress' to your INSTALLED_APPS.
• run a ./manage.py syncdb

removed duplicates

New batch of JavaScript? and Admin interaction

A new version has been uploaded, making a few changes over the last version:

• The JavaScript will now infer the path of Admin for you. (You can even make admin the root of the domain...)

• Removed upload_progress}} portions of middleware...now using {{{process_request

• Raises error whenever client asks for data that's not asked for. This signals the JavaScript to silently fail (and Stop!) after 10 tries.

• Tested / tweaked for different edge cases regarding what users will have set.

• Included a missing __init__.py file.

INSTALLATION

The above docs don't really work. Here's a revised version.

1. Add #2070 to your source tree

2. Add the latest patch (of this ticket) to your source tree.

3. Setup your middleware/models to track file progress.

4. Set FILE_UPLOAD_DIR to a writable folder in your settings.py file.

For (3), there are two options that come in the patch (as uploadprogress contrib):

• FileProgressCached -- uses the cache framework for quicker ajax (will NOT work if your cache framework doesn't)
• FileProgressDB -- uses Database models

Depending on what you choose, continue below.

FileProgressCached

To enable this, add 'django.contrib.uploadprogress.middleware.FileProgressCached' to your MIDDLEWARE_CLASSES.

FileProgressDB

To enable this:

• Add 'django.contrib.uploadprogress.middleware.FileProgressDB' to your MIDDLEWARE_CLASSES.
• Add 'django.contrib.uploadprogress' to your INSTALLED_APPS.
• Issue a ./manage.py syncdb

I have problem with upload progress with Apache 1.3 + FastCGI:

with request: http://test-fcgi/admin/upload_progress/?0

i have answer: Exception Type: OperationalError? Exception Value: (1048, "Column 'progress_id' cannot be null") Exception Location: c:\web\Python24\lib\site-packages\MySQLdb\connections.py in defaulterrorhandler, line 35

exception in: django\contrib\uploadprogress\middleware\uploaddb.py in line: self._db_row, created = FileProgress?.objects.get_or_create(remote_addr = request.METAREMOTE_ADDR?, progress_id = request.METAUPLOAD_PROGRESS_ID?)

may be request.METAUPLOAD_PROGRESS_ID? is empty ?

http://test-fcgi/admin/upload_progress/?0 --> Is this a target location of upload form? It should be a random 32 character upload id. Check you javascript to see what exactly created.

Ok, my fault. Check your javascript that set http request header (req.setRequestHeader("X-Upload-Id", uuid); )

Replying to anonymous:

Ok, my fault. Check your javascript that set http request header (req.setRequestHeader("X-Upload-Id", uuid); )

Yes. The GET querystring is an incremental number. This changes the URI for each request which supposedly is needed because IE (6?) will not do another async request for the same URI repeatedly if one of them haven't answered yet or IE has "given up" on one of them.

As of the newest release of #2070, the only two ways to specify the upload ID is through the header X-Upload-Id and the GET variable progress_id.

By following the instructions in Michael Axiak <axiak@mit.edu> as exactly as possible, using the 5343_javascript_and_admin.diff patch I found that file uploads cause lots of broken pipes with the development server. File uploads DO however seem to work, allthough not with edit_inline. Nevertheless I get no file progress reports.

Is this patch meant to work with the development server at all, should I migrate to Apache and mod_python for my development instead - or am I missing something?

By the way; in django.contrib.uploadprogress there is still an init file missing. ;)

FYI; using Firefox 2, the FileField? is in my case a subclassed version of the ImageField? (source available on request)

Replying to Michael Axiak <axiak@mit.edu>:

A new version has been uploaded, making a few changes over the last version: * The JavaScript will now infer the path of Admin for you. (You can even make admin the root of the domain...) * Removed upload_progress}} portions of middleware...now using {{{process_request * Raises error whenever client asks for data that's not asked for. This signals the JavaScript to silently fail (and Stop!) after 10 tries. * Tested / tweaked for different edge cases regarding what users will have set. * Included a missing __init__.py file.

Hi,

I've been unable to get the most recent patch from #4165 (http://code.djangoproject.com/ticket/4165) to work. If anyone suggest what I might be doing wrong, or provide a minimal working example, that would be most helpful.

My understanding is that #4165 should work more or less out of the box with the default admin view, once a few configurations have been made. Specifically, I should see an upload progress bar in a model which contains eg a FileField? when I try to upload something. Note also that JS is enabled in the browsers I'm using -- I tried a couple of different ones.

So, I have followed the most recent instructions on the wiki as far as I was able. See below.

The model I am using is

extract from models.py

class FileUpload(models.Model):
    upload_date = models.DateTimeField(default=datetime.now(), blank=True, editable=False)
    upload = models.FileField(upload_to="bixfile")
    name = models.CharField(core=True, maxlength=100)
    description = models.CharField(blank=True, maxlength=200)
    folder = models.ForeignKey(FolderUpload, null=True, blank=True, related_name='parentfolder',
edit_inline=models.TABULAR)

    class Admin:
        pass

I checked that a table called uploadprogress_fileprogress is created in the db, as see below

sqlite output

sqlite> .dump uploadprogress_fileprogress
BEGIN TRANSACTION;
CREATE TABLE "uploadprogress_fileprogress" (
    "id" integer NOT NULL PRIMARY KEY,
    "remote_addr" varchar(64) NOT NULL,
    "progress_id" varchar(32) NOT NULL,
    "progress_text" text NOT NULL,
    "last_ts" datetime NOT NULL,
    UNIQUE ("remote_addr", "progress_id")
);

I see the expected listing in the admin view.

admin listing for add file progress

Add file progress

Remote addr:
Progress id:
Last ts:

Date:  Today | Calendar
Time:  Now | Clock

However, there is nothing corresponding to a progress bar in FileUpload?. What am I doing wrong, if anything? If you don't have the time or inclination to debug my setup, could you provide a minimal working example?

Thanks, Faheem.

installation notes

INSTALLATION

The above docs don't really work. Here's a revised version.

1. Add #2070 to your source tree

Used 6469_streaming_file_upload.diff as uploaded to http://code.djangoproject.com/ticket/2070. When tested separately, this seems to work fine.

2. Add the latest patch (of this ticket) to your source tree.

Patched against svn 6469 + 6469_streaming_file_upload.diff, and added an empty django/contrib/uploadprogress/init.py.

3. Setup your middleware/models to track file progress.

Did the following as detailed below:

* Added 'django.contrib.uploadprogress.middleware.FileProgressDB' to

my MIDDLEWARE_CLASSES.

* Added 'django.contrib.uploadprogress' to my INSTALLED_APPS.
* Issued a ./manage.py syncdb

4. Set FILE_UPLOAD_DIR to a writable folder in your settings.py file.

Set this as

FILE_UPLOAD_DIR='/tmp'.

For (3), there are two options that come in the patch (as uploadprogress contrib):

* FileProgressCached? -- uses the cache framework for quicker ajax (will NOT > work if your cache framework doesn't)
* FileProgressDB -- uses Database models

Depending on what you choose, continue below.

See above.

FileProgressCached?

To enable this, add 'django.contrib.uploadprogress.middleware.FileProgressCached?' to your MIDDLEWARE_CLASSES.

FileProgressDB

To enable this:

* Add 'django.contrib.uploadprogress.middleware.FileProgressDB' to your MIDDLEWARE_CLASSES.
* Add 'django.contrib.uploadprogress' to your INSTALLED_APPS.
* Issue a ./manage.py syncdb

Update progress bar patch to 6602, with corrections. This should work with 6525_all_tests_pass.diff in issue 2070.

Sorry, the previous upload was incorrect.

I got the progress bar patch to work. There are two changes with respect to the previous patch.

1) (Minor): Added missing django/contrib/uploadprogress/init.py. Note that diff ignores empty files, which may explain why this keeps going missing.

2) For some reason, the pickled object corresponding to self.progress.text in _get_dict in the FileProgress? class in models.py is unicode, when it should of course be str. So pickle refuses to unpickle, and wackiness ensues. Well, no progress bar, anyway.

A workaround is to do

pickle.loads(str(self.progress_text))

as reflected in the patch. This isn't very satisfactory, however, so I'm hoping someone can tell me what is the problem here and how to fix it. I was hoping that the recent unicode related fix would take care of this, but apparently not.

Faheem.

I'm not sure if upload progress in the admin is really needed, so this might want to wait until we think about redesigning the admin interface.

I'm closing this fixed and marking it as 2070-docs. That is the documentation from the merger of #2070 describes how one can write this outside of Django's core.

I am very close to posting code to http://code.google.com/p/django-uploadutils/ that implements the feature described in this ticket.

Cheers,
Mike