File upload to a CIFS mount fails with Permissions Denied

[Derrick Jackson]

I upgraded from DJango 1.10.6 to 1.11 and can no longer use FileField to upload files larger than 2.5 MB. When an attempt is made to upload a larger file I get the error below occurs. Please note if I downgrade to 1.10.6 again the upload feature works just fine.

Django Version:	1.11
Exception Type:	PermissionError
Exception Value:	
[Errno 1] Operation not permitted
Exception Location:	/usr/lib/python3.5/shutil.py in copystat, line 195
Python Executable:	/usr/bin/python3
Python Version:	3.5.2

Environment:


Request Method: POST
Request URL: http://localhost:8000/orders/service/vpn/5/

Django Version: 1.11
Python Version: 3.5.2
Installed Applications:
['django.contrib.auth',
 'django.contrib.contenttypes',
 'django.contrib.sessions',
 'django.contrib.sites',
 'django.contrib.messages',
 'django.contrib.staticfiles',
 'impersonate',
 'stronghold',
 'django.contrib.admin',
 'django.contrib.admindocs',
 'ajax_select',
 'django_object_actions',
 'ckeditor',
 'mail_templated',
 'administrative',
 'changes',
 'decommission',
 'ecm',
 'feedback',
 'file_manager',
 'gunicorn',
 'orders',
 'old_net_chg_req',
 'server_request',
 'security',
 'shared',
 'system',
 'utilities',
 'downtime',
 'debug_toolbar']
Installed Middleware:
('django.middleware.security.SecurityMiddleware',
 'django.contrib.sessions.middleware.SessionMiddleware',
 'django.contrib.auth.middleware.AuthenticationMiddleware',
 'shared.middleware.CustomDowntimeMiddleware',
 'stronghold.middleware.LoginRequiredMiddleware',
 'django.middleware.common.CommonMiddleware',
 'django.middleware.csrf.CsrfViewMiddleware',
 'django.contrib.auth.middleware.SessionAuthenticationMiddleware',
 'shared.middleware.GlobalRequestMiddleware',
 'shared.middleware.UserTracebackMiddleware',
 'django.contrib.messages.middleware.MessageMiddleware',
 'impersonate.middleware.ImpersonateMiddleware',
 'django.middleware.clickjacking.XFrameOptionsMiddleware',
 'debug_toolbar.middleware.DebugToolbarMiddleware')



Traceback:

File "/usr/local/lib/python3.5/dist-packages/django/core/handlers/exception.py" in inner
  41.             response = get_response(request)

File "/usr/local/lib/python3.5/dist-packages/django/core/handlers/base.py" in _legacy_get_response
  249.             response = self._get_response(request)

File "/usr/local/lib/python3.5/dist-packages/django/core/handlers/base.py" in _get_response
  187.                 response = self.process_exception_by_middleware(e, request)

File "/usr/local/lib/python3.5/dist-packages/django/core/handlers/base.py" in _get_response
  185.                 response = wrapped_callback(request, *callback_args, **callback_kwargs)

File "/vagrant/orders/views_services.py" in service_vpn
  538.                     service_order_item.save()

File "/vagrant/orders/models.py" in save
  1150.         super(ServiceOrder, self).save(*args, **kwargs)

File "/usr/local/lib/python3.5/dist-packages/django/db/models/base.py" in save
  806.                        force_update=force_update, update_fields=update_fields)

File "/usr/local/lib/python3.5/dist-packages/django/db/models/base.py" in save_base
  836.             updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)

File "/usr/local/lib/python3.5/dist-packages/django/db/models/base.py" in _save_table
  900.                       for f in non_pks]

File "/usr/local/lib/python3.5/dist-packages/django/db/models/base.py" in <listcomp>
  900.                       for f in non_pks]

File "/usr/local/lib/python3.5/dist-packages/django/db/models/fields/files.py" in pre_save
  297.             file.save(file.name, file.file, save=False)

File "/usr/local/lib/python3.5/dist-packages/django/db/models/fields/files.py" in save
  95.         self.name = self.storage.save(name, content, max_length=self.field.max_length)

File "/usr/local/lib/python3.5/dist-packages/django/core/files/storage.py" in save
  54.         return self._save(name, content)

File "/usr/local/lib/python3.5/dist-packages/django/core/files/storage.py" in _save
  338.                     file_move_safe(content.temporary_file_path(), full_path)

File "/usr/local/lib/python3.5/dist-packages/django/core/files/move.py" in file_move_safe
  70.     copystat(old_file_name, new_file_name)

File "/usr/lib/python3.5/shutil.py" in copystat
  195.         follow_symlinks=follow)

Exception Type: PermissionError at /orders/service/vpn/5/
Exception Value: [Errno 1] Operation not permitted

[Tim Graham]

I'm not sure if there's much we can do with the report without a way to reproduce the problem. If you can ​bisect to determine the commit where the behavior changed, that would be useful.

[Derrick Jackson]

Tim,

After chatting with a colleague of mine we were able to determine that this commit ​f734e2d is where the issue I reported started. Specifically changing
file.save(file.name, file, save=False) to file.save(file.name, file.save, save=False) in files.py causes the copystat error.

Please let me know if you need additional details regarding this issue.
Thanks,
Derrick

[Derrick Jackson]

meant to re-open this ticket once I added my comment.

[Derrick Jackson]

Additional Details: In my case, the files are larger than 2.5 MB. The tmp directory is on a local linux file system and the final location is a mounted CIFS drive. This ​Stack Overflow post covers my exact situation.

[Tim Graham]

Thanks for the additional details. Can you offer a patch that fixes the issue for you? I guess a regression test might involve some mocking to simulate a CIFS mount.

[Derrick Jackson]

Tim... I will see what I can do. The tough part will be simulating a CIFS mount. I know the easy route would be to wrap line 296 of django/db/models/fields/files.py with a try/except but I know there's more to it.

try:
    file.save(file.name, file.file, save=False)
except Exception as e:
    file.save(file.name, file, save=False)

I'll keep digging into it.

[Derrick Jackson]

Tim,

A co-worker (pj@…) and I put together a suggested patch for the reported issue. We created the following pull request: ​https://github.com/django/django/pull/8486

[Tim Graham]

Looks okay at first glance. The next step is to add some tests, probably by mocking shutils.copystat().

[Derrick Jackson]

The requested modifications and tests have been made. Here is the corresponding Pull Request: ​PR

[Tim Graham <timograham@…>]

In 789c2901:

Fixed #28170 -- Fixed file_move_safe() crash when moving files to a CIFS mount.

[Tim Graham <timograham@…>]

In 7250393:

[1.11.x] Fixed #28170 -- Fixed file_move_safe() crash when moving files to a CIFS mount.

Backport of 789c290150a0a5e7312e152df281dbcaf4ec174e from master