Opened 12 years ago

Closed 11 years ago

Last modified 11 years ago

#18744 closed Cleanup/optimization (fixed)

NamedTemporaryFile opened in read mode cannot be written to by another process on Windows

Reported by: ty@… Owned by: Kevin Christopher Henry
Component: File uploads/storage Version: dev
Severity: Normal Keywords:
Cc: k@… Triage Stage: Accepted
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

The documentation for NamedTemporaryFile says that it exists so that on Windows so other processes can open the file, which is not possible using Python's implementation of NamedTemporaryFile because of the O_TEMPORARY flag that is passed by default.

However, if the temporary file is opened in read mode (r+), another process cannot write to the file, unless the temp file is closed before the outside process is called. This comes up with django_compressor when trying to use lessc (either https://github.com/duncansmart/less.js-windows or http://www.dotlesscss.org/).

I've attached a test that demonstrates the problem. On *nix, this test should pass.

Attachments (2)

test.py (1011 bytes ) - added by ty@… 12 years ago.
test.less (51 bytes ) - added by ty@… 12 years ago.

Download all attachments as: .zip

Change History (9)

by ty@…, 12 years ago

Attachment: test.py added

by ty@…, 12 years ago

Attachment: test.less added

comment:1 by Ramiro Morales, 12 years ago

Triage Stage: UnreviewedDesign decision needed

A more general question to the reporter of this ticket: What's the use case of having one piece of software (Django in this case) having a file open for reading (and possibly already reading from it) and then another process opening the same file for writing (and possibly writing to it)? This with independence of platform. Is there even an expected deterministic behavior such a scenarios should yield?

comment:2 by Tyler G. Hicks-Wright <ty@…>, 12 years ago

I ran into the bug in Jezdez's django_compressor (https://github.com/jezdez/django_compressor), which creates a temporary file for various CSS and Javascript preprocessors (e.g. Less, Sass, CoffeeScript) to write to. See https://github.com/jezdez/django_compressor/blob/develop/compressor/filters/base.py#L113.

It uses NamedTemporaryFile, with the expectation that the Windows version will behave identically to the *nix version, as the documentation states. However, many external processes will fail with some sort of permission error when they try to write to the temporary file that was created via NamedTemporaryFile and not closed.

My guess is that this will take some work to fix for Windows, given how Windows file handling works. If it's more work than it's worth, then at least the documentation should be updated to point out this deficiency.

comment:3 by Jacob, 11 years ago

Triage Stage: Design decision neededAccepted

comment:4 by Kevin Christopher Henry, 11 years ago

Cc: k@… added
Component: Core (Other)File uploads/storage
Has patch: set
Owner: changed from nobody to Kevin Christopher Henry
Status: newassigned
Type: BugCleanup/optimization
Version: 1.4master

As far as I can tell, the issue isn't about Django or NamedTemporaryFile but about how Windows deals with open files in general (or at least with how Python opens files on Windows). Django's NamedTemporaryFile is there to fix what is basically a bug in the Python implementation that prevents you from reopening the file in the same process on Windows; it does not address the more general problem of opening the same file for writing and reading in multiple processes on Windows. Even if we could figure out how to do that, this wouldn't be the right place to put it since the behavior would then be different depending on whether the file was opened here, or by open(), tempfile.mkstemp(), etc.

The documentation could probably stand to be clearer (for one thing, the link is broken), so I've attached a patch with an updated module docstring: https://github.com/django/django/pull/1640. If it's acceptable I will update the existing wiki page correspondingly.

However, we should really be thinking of this as code for internal use rather than as a tool provided and supported by Django. In addition to the confusion here, it's also the case that NamedTemporaryFile doesn't support the full set of keyword arguments used in Python 2.6+ and 3.0+. This might be fine given the way that Django uses it, but it wouldn't be good enough if trying to provide a proper drop-in replacement for Python's version.

Last edited 11 years ago by Kevin Christopher Henry (previous) (diff)

comment:5 by ty@…, 11 years ago

I agree, this is a reasonable fix, just to make it clear that this can't be considered the same on Windows as *nix.

Thanks!

comment:6 by Tim Graham <timograham@…>, 11 years ago

Resolution: fixed
Status: assignedclosed

In 59a34c43a8c3d62eaa400d48a9c26ed5400fc647:

Fixed #18744 -- Updated docstring to highlight limitations of NamedTemporaryFile

  • Noted that this does not allow for reading and writing the same open

file in different processes under Windows.

match the Python version.

comment:7 by Kevin Christopher Henry, 11 years ago

Since it exists, I updated the wiki entry. In my opinion it would be better to just delete it, though, as it's not really referenced anywhere; doesn't contain anything beyond what is in the module docstring; and concerns code that is meant for internal use.

Note: See TracTickets for help on using tickets.
Back to Top