Opened 5 years ago

Last modified 4 months ago

#23251 new Bug

Use a temporary folder to store uploaded files during tests

Reported by: Shai Berger Owned by:
Component: Testing framework Version: master
Severity: Normal Keywords: file storage upload
Cc: Marc Tamlyn Triage Stage: Accepted
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

Today, when running tests, Django uses the production storage for uploaded files -- meaning any tests which upload files will save copies of them, under different names, for every test run.

We need to treat this essentially like we treat mail -- during tests, a special file-storage should be set up to receive the uploads. Like with mail, it is probably better for this folder to be kept after the test run ends, and be cleared only when the tests are run again; but this is of lower priority.

This should be the default, or enabled easily in settings.

As @apollo13 noted, Django's own tests define an environment variable:

TEMP_DIR = tempfile.mkdtemp(prefix='django_')
os.environ['DJANGO_TEST_TEMP_DIR'] = TEMP_DIR

and all storages used in the suite use it or a subfolder. This alone, however, is not enough for user tests, as there is currently no way to define separate storages for test and production.

Change History (13)

comment:1 Changed 5 years ago by Tim Graham

Triage Stage: UnreviewedAccepted

comment:2 Changed 5 years ago by Pavel Shpilev

Owner: changed from nobody to Pavel Shpilev
Status: newassigned

comment:3 Changed 5 years ago by Berker Peksag

Needs documentation: unset
Needs tests: unset

Pavel, do you working on this? If not, I can take a look at this.

comment:4 Changed 5 years ago by Pavel Shpilev

Owner: Pavel Shpilev deleted
Status: assignednew

Oops, sorry. I was sure I deassigned myself.
Please, feel free to take it over.

comment:5 Changed 4 years ago by Carl Meyer

FWIW, there's also django-inmemorystorage, which is even simpler/faster. But it wouldn't allow leaving the files around after a failed test run for inspection.

comment:6 Changed 4 years ago by Marc Tamlyn

Cc: Marc Tamlyn added

As discussed on the mailing list this links nicely to the idea of a STORAGES setting, and a new temp storage backend for testing purposes like some of our other dummy back ends.

I think it also relates to ideas in #24721 about test extensions, so that the temo storage could (optionally?) be cleared in teardown/teardowntestdata or similar.

Last edited 4 years ago by Tim Graham (previous) (diff)

comment:7 Changed 4 years ago by Sasha Gaevsky

Owner: set to Sasha Gaevsky
Status: newassigned

comment:8 in reply to:  6 Changed 4 years ago by Sasha Gaevsky

Replying to mjtamlyn:

As discussed on the mailing list[1] this links nicely to the idea of a STORAGES setting, and a new temp storage backend for testing purposes like some of our other dummy back ends.

Link is pointing to changeset, not mailing list. Could you please update it? Thanks.

comment:9 Changed 4 years ago by Tim Graham

I edited comment 6 to add the link.

comment:10 in reply to:  9 Changed 4 years ago by Sasha Gaevsky

Replying to timgraham:

I edited comment 6 to add the link.

Thanks, I've reviewed.

So basically the idea is to introduce STORAGES setting, move there DEFAULT_FILE_STORAGE and STATICFILES_STORAGE settings and finally introduce separate storage backend for testing?

comment:11 Changed 4 years ago by Aymeric Augustin

I created #26029 to track the concept of multiple file storage backends, which may indeed make this easier to implement.

comment:12 Changed 8 months ago by Sasha Gaevsky

Owner: Sasha Gaevsky deleted
Status: assignednew

comment:13 Changed 4 months ago by אורי

In Speedy Net I defined TESTS_MEDIA_ROOT in settings, which is different than the production MEDIA_ROOT (it is defined only when running tests). All the files in the tests are saved there and it is deleted after the tests end. We might want to do something similar in Django for all users, and maybe define a default value to TESTS_MEDIA_ROOT.

I also defined a variable TESTS in settings, which is True only when running tests, and the tests asserts this value to True before running tests. This is for not to run tests accidentally with the production or other settings, but only with the tests settings. We might want to add a similar variable to the settings in Django.

You can see my commits (from today) in this branch:
https://github.com/speedy-net/speedy-net/commits/uri_main_branch_2019-06-30_a

Note: See TracTickets for help on using tickets.
Back to Top