FileField upload of large files looping and failing

This might well be a dupe of #8203. I gather from your report that the actual uploaded file is ok, and I assume that the problem you're having is with the temporary file, which is likely not cleaned up properly in Windows. Reopen if you estimate that your issue is different than #8203, but please provide more details if that's the case.

I dont think this is a duplicate. No temp file involved here.

When saving the object it uploads the file and then uploads it again and again and again, 200+ times until it fails. The file name of each file gets longer and longer as it appends _ to the new file's name. It appears that it would loop forever except that it eventually fails because the file name, with 200+ appended _'s exceeds the appearant OS filename limit of 215 characters.

So if I upload a file named "A" i will get 215 files uploaded before it fails. If my file name is "0123456789" i would get 205 files uploaded before it fails.

Fyi, the db object never does get created.

It may be my configuration but its not just my code, latest svn of django-photologue is failing the same way when I try to upload to a large zip file.

Hmmm, that's an interesting issue... At first glance, the fact that it keeps sending the file repeatedly seems like it is an issue with the front-end, not with Django. What is your front-end environment? Is there, for example, some javascript involved?

By the way, it is normal that it crashes when the filename reaches 215 characters as this is the allowed maximum in Windows.

Pulled the few scraps of js out of the form and retried. Still have the problem. It occurs in both Firefox2 and IE7.

Looking at the code in storage.py

class FileSystemStorage(Storage):
    """
    Standard filesystem storage
    """

    def __init__(self, location=settings.MEDIA_ROOT, base_url=settings.MEDIA_URL):
        self.location = os.path.abspath(location)
        self.base_url = base_url

    def _open(self, name, mode='rb'):
        return File(open(self.path(name), mode))

    def _save(self, name, content):
        full_path = self.path(name)

        directory = os.path.dirname(full_path)
        if not os.path.exists(directory):
            os.makedirs(directory)
        elif not os.path.isdir(directory):
            raise IOError("%s exists and is not a directory." % directory)

        # There's a potential race condition between get_available_name and
        # saving the file; it's possible that two threads might return the
        # same name, at which point all sorts of fun happens. So we need to
        # try to create the file, but if it already exists we have to go back
        # to get_available_name() and try again.

        while True:
            try:
                # This file has a file path that we can move.
                if hasattr(content, 'temporary_file_path'):
                    file_move_safe(content.temporary_file_path(), full_path)
                    content.close()

                # This is a normal uploadedfile that we can stream.
                else:
                    # This fun binary flag incantation makes os.open throw an
                    # OSError if the file already exists before we open it.
                    fd = os.open(full_path, os.O_WRONLY | os.O_CREAT | os.O_EXCL | getattr(os, 'O_BINARY', 0))
                    try:
                        locks.lock(fd, locks.LOCK_EX)
                        for chunk in content.chunks():
                            os.write(fd, chunk)
                    finally:
                        locks.unlock(fd)
                        os.close(fd)
            except OSError:
                # Ooops, we need a new file name.
                name = self.get_available_name(name)
                full_path = self.path(name)
            else:
                # OK, the file save worked. Break out of the loop.
                break
                
        return name

I would say, that based on what I'm observing, I'm stuck in the "while True" loop. Since my file is uploaded with a new name over and over (whether I use a temp directory or not) then I must be getting an OSerror on the Content.close() and on the locks.unlock(fd) or os.close(fd).

Mistake on previous assumption. I'm always going through file_move_safe(), regardless of whether or not I specify FILE_UPLOAD_TEMP_DIR (its just using C:\Windows\Temp if I don't). So, it's probably the same issue as #8203, just presenting in a different way. The "move" is working but I will bet I'm getting an OSerror on the attempt to unlock/close the temp file... so I do it all again and again. I'll stop testing on Windows and move over to a real OS 'till the patch goes in lol.