Opened 10 years ago

Closed 8 years ago

#24007 closed Bug (wontfix)

Unable to unpickle models from an external script

Reported by: Claude Paroz Owned by: nobody
Component: Core (Other) Version: 1.7
Severity: Normal Keywords:
Cc: Triage Stage: Accepted
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

I'm using Python rq stack to delegate some tasks from my Django app to a queued system. In some of those tasks, models are pickled and then unpickled by rq. Unfortunately, with Django 1.7, I'm getting AppRegistryNotReady errors:

Traceback (most recent call last):
  File "/usr/local/bin/rqworker", line 9, in <module>
    load_entry_point('rq==0.4.6', 'console_scripts', 'rqworker')()
  File "/usr/local/lib/python2.7/dist-packages/rq/scripts/rqworker.py", line 100, in main
    w.work(burst=args.burst)
  File "/usr/local/lib/python2.7/dist-packages/rq/worker.py", line 358, in work
    self.execute_job(job)
  File "/usr/local/lib/python2.7/dist-packages/rq/worker.py", line 422, in execute_job
    self.main_work_horse(job)
  File "/usr/local/lib/python2.7/dist-packages/rq/worker.py", line 457, in main_work_horse
    success = self.perform_job(job)
  File "/usr/local/lib/python2.7/dist-packages/rq/worker.py", line 473, in perform_job
    job.func_name,
  File "/usr/local/lib/python2.7/dist-packages/rq/job.py", line 226, in func_name
    self._unpickle_data()
  File "/usr/local/lib/python2.7/dist-packages/rq/job.py", line 194, in _unpickle_data
    self._func_name, self._instance, self._args, self._kwargs = unpickle(self.data)
  File "/usr/local/lib/python2.7/dist-packages/rq/job.py", line 48, in unpickle
    obj = loads(pickled_string)
  File "/usr/local/lib/python2.7/dist-packages/django/db/models/base.py", line 1464, in model_unpickle
    model = apps.get_model(*model_id)
  File "/usr/local/lib/python2.7/dist-packages/django/apps/registry.py", line 199, in get_model
    self.check_models_ready()
  File "/usr/local/lib/python2.7/dist-packages/django/apps/registry.py", line 131, in check_models_ready
    raise AppRegistryNotReady("Models aren't loaded yet.")
django.core.exceptions.AppRegistryNotReady: (AppRegistryNotReady("Models aren't loaded yet.",), <function model_unpickle

As you can see in the stack trace, my code has no chance to fix this as it is not called before the unpickling process.

One idea could be to check the app registry state in model_unpickle and populate it if needed. Thoughts?

Change History (21)

comment:1 by Tim Graham, 10 years ago

Triage Stage: UnreviewedAccepted

comment:3 by Claude Paroz, 10 years ago

Oh, and I'd really like to backport this to 1.7 as it appears to be a regression.

comment:4 by Tim Graham, 10 years ago

Triage Stage: AcceptedReady for checkin

comment:5 by Claude Paroz <claude@…>, 10 years ago

Resolution: fixed
Status: newclosed

In 108b8bf852c76855ed98f5abe55db1da845598e7:

Fixed #24007 -- Ensure apps registry's ready before unpickling models

This prevents AppRegistryNotReady errors when unpickling Django
models from an external script.

comment:6 by Claude Paroz <claude@…>, 10 years ago

In 47912d9f2b8f55044aef5b8ed9b006e8dd8ce976:

[1.7.x] Fixed #24007 -- Ensure apps registry's ready before unpickling models

This prevents AppRegistryNotReady errors when unpickling Django
models from an external script.
Backport of 108b8bf85 from master.

comment:7 by Collin Anderson, 10 years ago

comment:8 by Tim Graham <timograham@…>, 10 years ago

In 4fb38b73073dcd54f4af41a2af14dbc36f5d9f33:

Allowed Python executable names other than python in test from refs #24007.

comment:9 by Tim Graham, 10 years ago

Has patch: unset
Resolution: fixed
Status: closednew
Triage Stage: Ready for checkinAccepted

Another problem with the test when running it outside the tests directory:

$ ./tests/runtests.py model_regress
Testing against Django installed in '/home/tim/code/django/django'
Creating test database for alias 'default'...
Creating test database for alias 'other'...
.............s..Traceback (most recent call last):
  File "/home/tim/code/django/tmpCkJ7D2.py", line 9, in <module>
    article = pickle.loads(data)
  File "/usr/lib/python2.7/pickle.py", line 1382, in loads
    return Unpickler(file).load()
  File "/usr/lib/python2.7/pickle.py", line 858, in load
    dispatch[key](self)
  File "/usr/lib/python2.7/pickle.py", line 1133, in load_reduce
    value = func(*args)
  File "/home/tim/code/django/django/db/models/base.py", line 1662, in model_unpickle
    apps.populate(settings.INSTALLED_APPS)
  File "/home/tim/code/django/django/apps/registry.py", line 85, in populate
    app_config = AppConfig.create(entry)
  File "/home/tim/code/django/django/apps/config.py", line 87, in create
    module = import_module(entry)
  File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
ImportError: No module named model_regress
F.
======================================================================
FAIL: test_unpickling_when_appregistrynotready (model_regress.test_pickle.ModelPickleTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/tim/code/django/tests/model_regress/test_pickle.py", line 88, in test_unpickling_when_appregistrynotready
    self.fail("Unable to reload model pickled data")
AssertionError: Unable to reload model pickled data

----------------------------------------------------------------------

comment:10 by Claude Paroz, 10 years ago

One solution: https://github.com/django/django/pull/3758, but I may miss some simpler resolution.

comment:11 by Tim Graham, 10 years ago

Has patch: set
Triage Stage: AcceptedReady for checkin

comment:12 by Claude Paroz <claude@…>, 10 years ago

In 1d9fc5caa947ff4ee72180185e91a9a145171712:

Made model_regress unpickling test CWD-independent

Refs #24007. Thanks Tim Graham for his help with the patch.

comment:13 by Tim Graham <timograham@…>, 10 years ago

In 995be4a1375340a53668dd80444756d77302000d:

Fixed refs #24007 test on Windows.

comment:14 by Tim Graham <timograham@…>, 10 years ago

In ea1865223886295674422923a2a9d38bacb0564d:

[1.7.x] Made model_regress unpickling test CWD-independent

Refs #24007. Thanks Tim Graham for his help with the patch.

Backport of 1d9fc5caa947ff4ee72180185e91a9a145171712 and
995be4a1375340a53668dd80444756d77302000d from master

comment:15 by Tim Graham, 10 years ago

Resolution: fixed
Status: newclosed

comment:16 by Tim Graham <timograham@…>, 9 years ago

In 5338ff4808c822a8b00e90154b884b7be3011e60:

Fixed #24160 -- Fixed model_regress test on Windows; refs #24007.

comment:17 by Tim Graham <timograham@…>, 9 years ago

In cef3f805c21c029d017e6704565f9f2f5746a1e5:

[1.7.x] Fixed #24160 -- Fixed model_regress test on Windows; refs #24007.

Backport of 5338ff4808c822a8b00e90154b884b7be3011e60 from master

comment:18 by Tim Graham, 8 years ago

Triage Stage: Ready for checkinAccepted

Reopening per discussion on django-dev which suggests this should be reverted.

PR

comment:19 by Tim Graham, 8 years ago

Resolution: fixed
Status: closednew

comment:20 by Tim Graham <timograham@…>, 8 years ago

In 10a16280:

Refs #24007 -- Removed an apps.populate() call in model unpickling that can cause deadlocks.

comment:21 by Tim Graham, 8 years ago

Resolution: wontfix
Status: newclosed
Note: See TracTickets for help on using tickets.
Back to Top