Opened 3 years ago

Closed 2 years ago

Last modified 2 years ago

#21430 closed New feature (fixed)

Raise Warning when unpickling Models and QuerySet from different version

Reported by: FunkyBob Owned by: ANUBHAV JOSHI
Component: Database layer (models, ORM) Version: master
Severity: Normal Keywords:
Cc: loic84 Triage Stage: Accepted
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

Currently the errors from doing this are not obvious at all, and make debugging a nightmare.

Having a specific exception to look for in such cases would greatly simplify everyones job.

Change History (15)

comment:1 Changed 3 years ago by Anssi Kääriäinen

Needs documentation: unset
Needs tests: unset
Patch needs improvement: unset

I think this is a good idea. We should invent a new exception type UnsupportedUnpickleException or something like that.

The implementation doesn't need to be more complicated than adding version = django's major version to the pickled state in __getstate__ and then checking that the version is compatible in __setstate__. For first try we should just raise error if versions differ, later on we might want to try to actually recover from pickling errors. Unfortunately Query and all that it depends on seems to be way too complex to successfully do cross-version unpickling.

comment:2 Changed 3 years ago by Anssi Kääriäinen

Triage Stage: UnreviewedAccepted

By the way, I don't think we should leave this to just QuerySet. I know that at least Model unpickling can have this same issue. But for model unpickling we might be able to actually do cross-version conversions in __getstate__() and __setstate__(). Again, first try lets just throw an error.

comment:3 Changed 3 years ago by prasoon2211

Owner: changed from nobody to prasoon2211
Status: newassigned

comment:4 Changed 3 years ago by prasoon2211

A PR has been filed for this : https://github.com/django/django/pull/2082

comment:5 Changed 2 years ago by ANUBHAV JOSHI

Owner: changed from prasoon2211 to ANUBHAV JOSHI

Working on this in my GSoC project.

comment:6 Changed 2 years ago by ANUBHAV JOSHI

Version: 1.6master

comment:7 in reply to:  2 Changed 2 years ago by ANUBHAV JOSHI

Replying to akaariai:

By the way, I don't think we should leave this to just QuerySet. I know that at least Model unpickling can have this same issue. But for model unpickling we might be able to actually do cross-version conversions in __getstate__() and __setstate__(). Again, first try lets just throw an error.

Can you please comment out your thoughts regarding cross-version conversion. I mean if you have thought of any way to pickle Models than the normal way or some other strategy.

Last edited 2 years ago by ANUBHAV JOSHI (previous) (diff)

comment:8 Changed 2 years ago by ANUBHAV JOSHI

First we will deal with raising errors. Cross-version conversions in Model would be dealt with separately.
Regarding raising errors, we have reached where we are raising errors when version is different. Any ideas on how we can proceed further and check if versions are compatible or not.

comment:9 Changed 2 years ago by ANUBHAV JOSHI

Summary: Raise explicit error when unpickling QuerySet from incompatible versionRaise explicit error when unpickling Models and QuerySet from incompatible version

comment:10 Changed 2 years ago by loic84

We discussed this on IRC with Anubhav and Simon.

My understanding of the issue is that we can't detect if a pickle is really incompatible since unpickling gives us silently corrupted objects rather than hard failures.

The proposed patch (https://github.com/coder9042/django/compare/gsoc_21430) raises an exception every time a pickle from another major version is detected, based on the premise that we "try" not to break pickle in minor version. While it's true that minor releases are less likely to break pickling, a security issue or a data-loss issue could easily require a change in the ORM that would break pickling.

I proposed we used runtime warnings rather than exceptions when any version mismatch is detected, these can easily be ignored if you've never cared/encountered the problem, but when you are affected, you'll clearly get a pointer to the issue. Another option is to mix the two approaches, forbid pickles across major versions with exceptions, and only warn about pickles across minor version.

Our "release checklist" documentation could also add a recommendation of clearing pickled objects when upgrading Django.

comment:11 Changed 2 years ago by loic84

Cc: loic84 added

comment:12 Changed 2 years ago by Anssi Kääriäinen

RuntimeWarnings are OK to me. The idea is to make problems in this area easier to detect, and warnings will suffice for that.

As for making model pickling compatible between major versions, lets split that issue into separate ticket.

So, the proposed plan for this ticket is: raise RuntimeWarning on every detected major or minor version change for both models and querysets.

comment:13 Changed 2 years ago by ANUBHAV JOSHI

Has patch: set
Summary: Raise explicit error when unpickling Models and QuerySet from incompatible versionRaise Warning when unpickling Models and QuerySet from different version

comment:14 Changed 2 years ago by Tim Graham <timograham@…>

Resolution: fixed
Status: assignedclosed

In 42736ac8e8c31137131714013951249a09e6e7d4:

Fixed #21430 -- Added a RuntimeWarning when unpickling Models and QuerySets from a different Django version.

Thanks FunkyBob for the suggestion, prasoon2211 for the initial patch,
and akaariai, loic, and charettes for helping in shaping the patch.

comment:15 Changed 2 years ago by Tim Graham <timograham@…>

In 80f4487d17a0040e9be35e7ee6ac478bafe6504a:

Fixed #22867 -- Memoized django.utils.version.get_git_changeset().

This improves pickling speed in prelease versions of Django; refs #21430.

Note: See TracTickets for help on using tickets.
Back to Top