Opened 4 years ago

Closed 4 years ago

#25426 closed Bug (duplicate)

pickling SimpleLazyObject fails just after accessing related object of wrapped model instance.

Reported by: Iru Hwang Owned by: Iru Hwang
Component: Core (Serialization) Version: 1.8
Severity: Release blocker Keywords:
Cc: Triage Stage: Accepted
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description (last modified by Iru Hwang)

Reproduction step :

import pickle
import django
django.setup()

from django.contrib.auth.models import User
from django.utils.functional import SimpleLazyObject

u = User.objects.select_related('profile').get(id=153)
o = SimpleLazyObject(lambda :u)
pickle.dumps(o) # FAILS
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/pickle.py", line 1374, in dumps
    Pickler(file, protocol).dump(obj)
  File "/usr/local/lib/python2.7/pickle.py", line 224, in dump
    self.save(obj)
  File "/usr/local/lib/python2.7/pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "/usr/local/lib/python2.7/pickle.py", line 419, in save_reduce
    save(state)
  File "/usr/local/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/local/lib/python2.7/pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "/usr/local/lib/python2.7/pickle.py", line 661, in _batch_setitems
    for k, v in items:
RuntimeError: dictionary changed size during iteration
pickle.dumps(o) # SUCCEEDS 

This also fails :

u = User.objects.get(id=153)
o = SimpleLazyObject(lambda :u)
o.profile
pickle.dumps(o)

This looks to be related with change in Model.reduce in django 1.8
adding DJANGO_VERSION_PICKLE_KEY in dict

    def __reduce__(self):
        """                                                                                                                                                                                                                                                                     
        Provides pickling support. Normally, this just dispatches to Python's                                                                                                                                                                                                   
        standard handling. However, for models with deferred field loading, we                                                                                                                                                                                                  
        need to do things manually, as they're dynamically created classes and                                                                                                                                                                                                  
        only module-level classes can be pickled by the default path.                                                                                                                                                                                                           
        """
        data = self.__dict__
        data[DJANGO_VERSION_PICKLE_KEY] = get_version()
        if not self._deferred:
            class_id = self._meta.app_label, self._meta.object_name
            return model_unpickle, (class_id, [], simple_class_factory), data
        defers = []
        for field in self._meta.fields:
            if isinstance(self.__class__.__dict__.get(field.attname),
                          DeferredAttribute):
                defers.append(field.attname)
        model = self._meta.proxy_for_model
        class_id = model._meta.app_label, model._meta.object_name
        return (model_unpickle, (class_id, defers, deferred_class_factory), data)

Attachments (1)

patch_25426.diff (2.9 KB) - added by Iru Hwang 4 years ago.

Download all attachments as: .zip

Change History (12)

comment:1 Changed 4 years ago by Iru Hwang

Description: modified (diff)

comment:2 Changed 4 years ago by Iru Hwang

Description: modified (diff)

comment:3 Changed 4 years ago by Iru Hwang

Severity: NormalRelease blocker

comment:4 Changed 4 years ago by Iru Hwang

This is how this happens :

  1. pickling lazy object starts.
  2. wrapped user object is pickled without DJANGO_VERSION_PICKLE_KEY in its dict. (SimpleLazyObject.__reduce_ex__ #instancemethod)
  3. related object "profile" is pickled. It also has a reference to the user object. ( profile.user, due to one-to-one relationship (Model.__reduce__ #instancemethod)
  4. the user object(profile.user) is pickled again with same dict used at step 1. DJANGO_VERSION_PICKLE_KEY is set.(Model.__reduce__ #instancemethod)
  5. pickling the user object fails due to dictionary changed size during iteration.
  6. pickling lazy object fails.
Last edited 4 years ago by Iru Hwang (previous) (diff)

comment:5 Changed 4 years ago by Iru Hwang

This happens always when cPickle is used. Does not happen when proto >=2 with python implementation of pickle.

comment:6 Changed 4 years ago by Iru Hwang

Has patch: set

Changed 4 years ago by Iru Hwang

Attachment: patch_25426.diff added

comment:7 Changed 4 years ago by Iru Hwang

the patch is for saving django version info as unpickler's argument instead of setting it as model instance's attribute which results in RuntimeError depending on pickling protocol.

Last edited 4 years ago by Iru Hwang (previous) (diff)

comment:8 Changed 4 years ago by Iru Hwang

added test for this issue and updated existing one
https://github.com/django/django/compare/master...knifenomad:ticket_25426

Last edited 4 years ago by Iru Hwang (previous) (diff)

comment:9 Changed 4 years ago by Tim Graham

Triage Stage: UnreviewedAccepted

comment:10 Changed 4 years ago by Iru Hwang

Owner: changed from nobody to Iru Hwang
Status: newassigned

comment:11 Changed 4 years ago by Iru Hwang

Resolution: duplicate
Status: assignedclosed

closing since this duplicates https://code.djangoproject.com/ticket/25389

Note: See TracTickets for help on using tickets.
Back to Top