LazyObject doesn't support multiprocessing on Python 2

[Johannes Maron]

Initial summary: AttributeError: 'FileSystemStorage' object has no attribute 'location'

I don't really want to bug anyone with python 2.7, for I couldn't care less. But, I stumbled upon something rather odd.

Easiest is to just check out the error on travis including the stack trace:
​https://travis-ci.org/codingjoe/django-stdimage/builds/94713726

Everything works fine in Python 3, the attribute is available, as it should. It it's an instance attribute and properly set in the __init__ method:
​https://github.com/django/django/blob/master/django/core/files/storage.py#L169-L172

I guess the error is here:
​https://github.com/django/django/blob/master/django/utils/_os.py#L24-L32

I'm not sure tho, because the method must return at least None, therefore I don't see how the attribute error is being raised.

[Claude Paroz]

Creating a storage instance as a function parameter ((DefaultStorage() for render_variations) looks a bit suspicious to me. I'm not convinced the problem is on Django's side.

[Johannes Maron]

Let me try to convince you then. I just ​changed it to the ​documented default_storage helper, which is nothing other than DefaultStorage().

I also just added a little test, that tests only storage.location. That works, that doesn't mean there isn't a problem tho.
See the tests don't even execute the case with a default storage.
It uses the FieldFile's storage.

Besides all that, I don't do any wizardry with the storages, nor in the settings. It must strike you, that there is a different behavior on python 2.7 and 3.*

[Johannes Maron]

I found the bug. It's the lazy object implementation of the default storage. It doesn't work in python 2 using multiprocessing.

This little snippet allows to produce the error.

def has_location(storage):
        return hasattr(storage, 'location')


def test_multiprocessing_lazy_objects(image_upload_file):
    SimpleModel.objects.create(image=image_upload_file)
    obj = SimpleModel.objects.last()
    storage = obj.image.storage

    p = Pool(2)
    assert all(p.map(has_location, [default_storage, storage]))

[Tim Graham]

Doesn't look like anyone will tackle this before we drop Python 2.