Opened 4 months ago

Closed 4 months ago

Last modified 4 months ago

#35619 closed Bug (invalid)

django.core.files.uploadfile.InMemoryUploadedFile ignores chunk_size

Reported by: XU GUI PING Owned by:
Component: File uploads/storage Version: 5.0
Severity: Normal Keywords:
Cc: XU GUI PING Triage Stage: Unreviewed
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

in class django.core.files.uploadfile.InMemoryUploadedFile

def chunks(self, chunk_size=None):

self.file.seek(0)
yield self.read() # this code, should be yield self.read(chunk_size)

This bug will cause us to fail when writing to the file system after reading the uploaded content, with the error "OSError: [Errno 5] Input/output error"

Change History (3)

comment:1 by XU GUI PING, 4 months ago

It doesn't seem to work. After the modification, the next problem will occur. The file will only read the beginning part.

comment:2 by Sarah Boyce, 4 months ago

Resolution: invalid
Severity: Release blockerNormal
Status: newclosed
Summary: django.core.files.uploadfile.InMemoryUploadedFile has a serious bugdjango.core.files.uploadfile.InMemoryUploadedFile ignores chunk_size

Hi, I believe what you're trying to report is that chunk_size is ignored in InMemoryUploadedFile.chunks(chunk_size=None).
From what I can see, this is by design and this is clarified in a comment within multiple_chunks:

    def multiple_chunks(self, chunk_size=None):
        # Since it's in memory, we'll never have multiple chunks.
        return False

Essentially, there is no real benefit of chunking an in memory file

I appreciate that this isn't explicitly documented and maybe we could add a clarifying note to the docs.

But I don't believe this is a bug

comment:3 by XU GUI PING, 4 months ago

I have now solved my problem by setting FILE_UPLOAD_MAX_MEMORY_SIZE. The problem occurred because my operating system does not support writing more than 60000+ bytes at a time, but the default FILE_UPLOAD_MAX_MEMORY_SIZE of Django is 2.5M, which is far beyond the range supported by my operating system. I think it is feasible for Django to set such a default value, but in my scenario, this does bring bugs. The benefits that MemoryFileUploadHandler brings to us are obvious, but it is not robust enough for my problem.

Note: See TracTickets for help on using tickets.
Back to Top