Updating the file contents of a Django FileField during upload results in I/O error

[Jeroen Jacobs]

I'm trying to encrypt the contents of a file that is being uploaded. This is the relevant code snippet:

class AppFile(models.Model):
    app_file = models.FileField(upload_to=upload_to, validators=[validate_file_size])
    encrypted_data_key = models.CharField(max_length=500, blank=True)

    def encrypt_file_with_data_key(self, data_key):
        cipher = Fernet(data_key)
        with self.app_file.open(mode='rb') as file:
            file_data = file.read()
            encrypted_data = cipher.encrypt(file_data)
        with self.app_file.open(mode='wb') as encrypted_file:
            encrypted_file.write(encrypted_data)

    def save(self, *args, **kwargs):
        if self._state.adding is True:

            # New image being uploaded
            encrypted_data_key, data_key = self.generate_data_key_from_vault()
            self.encrypted_data_key = encrypted_data_key

            # Encrypt the uploaded image file
            self.encrypt_file_with_data_key(data_key)

        super().save(args, kwargs)

I prefer this approach as this is agnostic of the StorageProvider being used. I also want to avoid detaching the file to a temporary folder, and re-attach it after encryption.

However, this results in the following error:

Traceback (most recent call last):
  File "/Users/jeroenjacobs/.pyenv/versions/myapp/lib/python3.11/site-packages/django/core/handlers/exception.py", line 55, in inner
    response = get_response(request)
               ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jeroenjacobs/.pyenv/versions/myapp/lib/python3.11/site-packages/django/core/handlers/base.py", line 197, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jeroenjacobs/.pyenv/versions/myapp/lib/python3.11/site-packages/django/views/generic/base.py", line 84, in view
    return self.dispatch(request, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jeroenjacobs/.pyenv/versions/myapp/lib/python3.11/site-packages/django/views/generic/base.py", line 119, in dispatch
    return handler(request, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jeroenjacobs/.pyenv/versions/myapp/lib/python3.11/site-packages/django/views/generic/edit.py", line 184, in post
    return super().post(request, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jeroenjacobs/.pyenv/versions/myapp/lib/python3.11/site-packages/django/views/generic/edit.py", line 153, in post
    return self.form_valid(form)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jeroenjacobs/.pyenv/versions/myapp/lib/python3.11/site-packages/django/contrib/messages/views.py", line 12, in form_valid
    response = super().form_valid(form)
               ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jeroenjacobs/.pyenv/versions/myapp/lib/python3.11/site-packages/django/views/generic/edit.py", line 135, in form_valid
    self.object = form.save()
                  ^^^^^^^^^^^
  File "/Users/jeroenjacobs/.pyenv/versions/myapp/lib/python3.11/site-packages/django/forms/models.py", line 548, in save
    self.instance.save()
  File "/Users/jeroenjacobs/PycharmProjects/myapp/mainapp/models.py", line 90, in save
    self.encrypt_file_with_data_key(data_key)
  File "/Users/jeroenjacobs/PycharmProjects/myapp/mainapp/models.py", line 77, in encrypt_file_with_data_key
    with self.app_file.open(mode='wb') as encrypted_file:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jeroenjacobs/.pyenv/versions/myapp/lib/python3.11/site-packages/django/db/models/fields/files.py", line 80, in open
    self.file.open(mode)
  File "/Users/jeroenjacobs/.pyenv/versions/myapp/lib/python3.11/site-packages/django/core/files/uploadedfile.py", line 115, in open
    self.file.seek(0)
ValueError: I/O operation on closed file.

Reading the contents doesn't seem to be a problem, but writing seems to generate an error, despite open being called.

[Natalia Bidart]

Hello,

This report seems better suited to be a support request. The best place to get answers to your issue is using any of the user support channels from ​this link.

Since the goal of this issue tracker is to track issues about Django itself, and your issue seems, at first, be located in your custom code, I'll be closing this ticket as invalid.
If, after debugging, you find out that this is indeed a bug in Django, please re-open with the specific details.

Thank you!

[Jeroen Jacobs]

I am reopening this ticket, as is this is clearly a bug and not a support request.

The error is not in my custom code! The error occurs when writing to a Django FileField which is opened for writing. This not custom code, I'm calling core Django code here.

Please take a good look at the code:

        with self.app_file.open(mode='wb') as encrypted_file:
            encrypted_file.write(encrypted_data)

app_file is a FileField and FileField is pure Django class, not custom code!

Please read the description again: I'm writing to a Django FileField which was successfully opened, and I'm getting an error that the file is closed! This is clearly a bug! Did you even try to reproduce it using my example code?

[Tim Graham]

Hi Jeroen, to confirm it's a bug, you should explain where Django is at fault.

[Jeroen Jacobs]

Getting an error that a file is closed when writing to it, when it was successfully opened for writing seems like a fault to me, no?

This is not a standard file object, but a FieldFile object if I understand the docs correctly (​https://docs.djangoproject.com/en/3.2/ref/models/fields/#filefield-and-fieldfile). It's certainly not a file permission issue.

[Jeroen Jacobs]

In the unlikely case it's not a bug, it's certainly undefined behaviour and maybe this should be documented somewhere.

So far, I've heard ZERO explanation why the reading the contents of newly uploaded file works without a problem, but writing gives this bizar error. IMHO there is a bug in FieldFile when you try to call open with mode wb on newly uploaded files.

If mode wb is not supported for INSERTS, and only for UPDATES, it should be documented somewhere.

No matter how you put it, mode wb does not work, and it could easily be verified if someone took 1 minute to actually run my example code.

[Natalia Bidart]

Hello Jeroen,

Please don't re-open tickets closed by Django maintainers/fellows. The code you shared, while indeed uses the Django provided FileField, does make a custom/non traditional use of it. Please reach out to the user support channels as suggested, where you can get help on whether the approach you took to encrypt the file content is sound or if there are other considerations that are needed.

I did read the ticket description, I understand you may feel frustrated but please follow the ​Django Code of Conduct, and please see ​the guidelines on reporting bugs.

The example code you provided is incomplete and does not work. There are undefined names and missing imports, and the stacktrace shows that a view and a form are involved which you did not provide. There is also lacking details of the settings of the project, such as the file storage backend that is being used.

As recommended, please reach out for help in the user forum or Discord channel, where there may be a bigger audience able to help in getting to the root of the issue. If this is indeed a Django bug, please reopen with a small yet complete Django project with the necessary models and other files to reproduce (I'd advice leaving encryption out of the code to reduce the scope of the reproducer).

[Mariusz Felisiak]

Duplicate of #33023.