Context Navigation

← Previous Ticket
Next Ticket →

#24007 closed Bug (wontfix)

Unable to unpickle models from an external script

Reported by:	Claude Paroz	Owned by:	nobody
Component:	Core (Other)	Version:	1.7
Severity:	Normal	Keywords:
Cc:		Triage Stage:	Accepted
Has patch:	yes	Needs documentation:	no
Needs tests:	no	Patch needs improvement:	no
Easy pickings:	no	UI/UX:	no

Description

I'm using Python rq stack to delegate some tasks from my Django app to a queued system. In some of those tasks, models are pickled and then unpickled by rq. Unfortunately, with Django 1.7, I'm getting AppRegistryNotReady errors:

Traceback (most recent call last):
  File "/usr/local/bin/rqworker", line 9, in <module>
    load_entry_point('rq==0.4.6', 'console_scripts', 'rqworker')()
  File "/usr/local/lib/python2.7/dist-packages/rq/scripts/rqworker.py", line 100, in main
    w.work(burst=args.burst)
  File "/usr/local/lib/python2.7/dist-packages/rq/worker.py", line 358, in work
    self.execute_job(job)
  File "/usr/local/lib/python2.7/dist-packages/rq/worker.py", line 422, in execute_job
    self.main_work_horse(job)
  File "/usr/local/lib/python2.7/dist-packages/rq/worker.py", line 457, in main_work_horse
    success = self.perform_job(job)
  File "/usr/local/lib/python2.7/dist-packages/rq/worker.py", line 473, in perform_job
    job.func_name,
  File "/usr/local/lib/python2.7/dist-packages/rq/job.py", line 226, in func_name
    self._unpickle_data()
  File "/usr/local/lib/python2.7/dist-packages/rq/job.py", line 194, in _unpickle_data
    self._func_name, self._instance, self._args, self._kwargs = unpickle(self.data)
  File "/usr/local/lib/python2.7/dist-packages/rq/job.py", line 48, in unpickle
    obj = loads(pickled_string)
  File "/usr/local/lib/python2.7/dist-packages/django/db/models/base.py", line 1464, in model_unpickle
    model = apps.get_model(*model_id)
  File "/usr/local/lib/python2.7/dist-packages/django/apps/registry.py", line 199, in get_model
    self.check_models_ready()
  File "/usr/local/lib/python2.7/dist-packages/django/apps/registry.py", line 131, in check_models_ready
    raise AppRegistryNotReady("Models aren't loaded yet.")
django.core.exceptions.AppRegistryNotReady: (AppRegistryNotReady("Models aren't loaded yet.",), <function model_unpickle

As you can see in the stack trace, my code has no chance to fix this as it is not called before the unpickling process.

One idea could be to check the app registry state in model_unpickle and populate it if needed. Thoughts?

Change History (21)

comment:1 by Tim Graham, 9 years ago

Triage Stage:	Unreviewed → Accepted

comment:2 by Claude Paroz, 9 years ago

Has patch:	set

https://github.com/django/django/pull/3747

comment:3 by Claude Paroz, 9 years ago

Oh, and I'd really like to backport this to 1.7 as it appears to be a regression.

comment:4 by Tim Graham, 9 years ago

Triage Stage:	Accepted → Ready for checkin

comment:5 by Claude Paroz <claude@…>, 9 years ago

Resolution:	→ fixed
Status:	new → closed

In 108b8bf852c76855ed98f5abe55db1da845598e7:

Fixed #24007 -- Ensure apps registry's ready before unpickling models

This prevents AppRegistryNotReady errors when unpickling Django
models from an external script.

comment:6 by Claude Paroz <claude@…>, 9 years ago

In 47912d9f2b8f55044aef5b8ed9b006e8dd8ce976:

[1.7.x] Fixed #24007 -- Ensure apps registry's ready before unpickling models

This prevents AppRegistryNotReady errors when unpickling Django
models from an external script.
Backport of 108b8bf85 from master.

comment:7 by Collin Anderson, 9 years ago

Regression in the test.
https://github.com/django/django/pull/3749

comment:8 by Tim Graham <timograham@…>, 9 years ago

In 4fb38b73073dcd54f4af41a2af14dbc36f5d9f33:

Allowed Python executable names other than python in test from refs #24007.

comment:9 by Tim Graham, 9 years ago

Has patch:	unset
Resolution:	fixed
Status:	closed → new
Triage Stage:	Ready for checkin → Accepted

Another problem with the test when running it outside the tests directory:

$ ./tests/runtests.py model_regress
Testing against Django installed in '/home/tim/code/django/django'
Creating test database for alias 'default'...
Creating test database for alias 'other'...
.............s..Traceback (most recent call last):
  File "/home/tim/code/django/tmpCkJ7D2.py", line 9, in <module>
    article = pickle.loads(data)
  File "/usr/lib/python2.7/pickle.py", line 1382, in loads
    return Unpickler(file).load()
  File "/usr/lib/python2.7/pickle.py", line 858, in load
    dispatch[key](self)
  File "/usr/lib/python2.7/pickle.py", line 1133, in load_reduce
    value = func(*args)
  File "/home/tim/code/django/django/db/models/base.py", line 1662, in model_unpickle
    apps.populate(settings.INSTALLED_APPS)
  File "/home/tim/code/django/django/apps/registry.py", line 85, in populate
    app_config = AppConfig.create(entry)
  File "/home/tim/code/django/django/apps/config.py", line 87, in create
    module = import_module(entry)
  File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
ImportError: No module named model_regress
F.
======================================================================
FAIL: test_unpickling_when_appregistrynotready (model_regress.test_pickle.ModelPickleTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/tim/code/django/tests/model_regress/test_pickle.py", line 88, in test_unpickling_when_appregistrynotready
    self.fail("Unable to reload model pickled data")
AssertionError: Unable to reload model pickled data

----------------------------------------------------------------------