#13335 closed (fixed)
Revision 12950 crashes complex interdependent models in directory
Reported by: | Matt Harasymczuk | Owned by: | nobody |
---|---|---|---|
Component: | Database layer (models, ORM) | Version: | dev |
Severity: | Keywords: | ||
Cc: | Triage Stage: | Accepted | |
Has patch: | yes | Needs documentation: | no |
Needs tests: | yes | Patch needs improvement: | no |
Easy pickings: | no | UI/UX: | no |
Description
my project is to big to hold models in one file
I am using models in directory structure
apps/
- myapp/ -- admin/ -- models/ -- views/
...etc
I did this from this post
http://www.acooke.org/cute/UsingaDire0.html
and it worked for very long time
Revision 12950 breaks it!
http://code.djangoproject.com/changeset/12950
exactly loading.py throw unhanded exception
If I downgrade to 12949 it is ok
traceback after upgrade to 12950
> python manage.py shell Traceback (most recent call last): File "manage.py", line 11, in <module> execute_manager( settings ) File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/django/core/management/__init__.py", line 438, in execute_manager utility.execute() File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/django/core/management/__init__.py", line 379, in execute self.fetch_command(subcommand).run_from_argv(self.argv) File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/django/core/management/base.py", line 196, in run_from_argv self.execute(*args, **options.__dict__) File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/django/core/management/base.py", line 223, in execute output = self.handle(*args, **options) File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/django/core/management/base.py", line 352, in handle return self.handle_noargs(**options) File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/django/core/management/commands/shell.py", line 18, in handle_noargs loaded_models = get_models() File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/django/db/models/loading.py", line 156, in get_models self._populate() File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/django/db/models/loading.py", line 60, in _populate self.load_app(app_name, True) File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/django/db/models/loading.py", line 87, in load_app models = import_module('.models', app_name) File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/django/utils/importlib.py", line 35, in import_module __import__(name) File "/Users/haras/Source/WebApps/current/crm/src/crm/contract/models/__init__.py", line 7, in <module> from crm.contract.models.contract import Contract File "/Users/haras/Source/WebApps/current/crm/src/crm/contract/models/contract.py", line 12, in <module> from crm.date.models.season import Season File "/Users/haras/Source/WebApps/current/crm/src/crm/date/models/__init__.py", line 7, in <module> from crm.date.models.month import Month File "/Users/haras/Source/WebApps/current/crm/src/crm/date/models/month.py", line 26, in <module> from ..admin import month File "/Users/haras/Source/WebApps/current/crm/src/crm/date/admin/month.py", line 10, in <module> admin.site.register( Month ) File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/django/contrib/admin/sites.py", line 94, in register validate(admin_class, model) File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/django/contrib/admin/validation.py", line 20, in validate models.get_apps() File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/django/db/models/loading.py", line 104, in get_apps self._populate() File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/django/db/models/loading.py", line 60, in _populate self.load_app(app_name, True) File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/django/db/models/loading.py", line 87, in load_app models = import_module('.models', app_name) File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/django/utils/importlib.py", line 35, in import_module __import__(name) File "/Users/haras/Source/WebApps/current/crm/src/crm/customer/models/__init__.py", line 7, in <module> from crm.customer.models.country import Country File "/Users/haras/Source/WebApps/current/crm/src/crm/customer/models/country.py", line 40, in <module> from ..admin import country File "/Users/haras/Source/WebApps/current/crm/src/crm/customer/admin/country.py", line 18, in <module> admin.site.register( Country, CountryAdmin ) File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/django/contrib/admin/sites.py", line 94, in register validate(admin_class, model) File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/django/contrib/admin/validation.py", line 20, in validate models.get_apps() File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/django/db/models/loading.py", line 104, in get_apps self._populate() File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/django/db/models/loading.py", line 60, in _populate self.load_app(app_name, True) File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/django/db/models/loading.py", line 87, in load_app models = import_module('.models', app_name) File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/django/utils/importlib.py", line 35, in import_module __import__(name) File "/Users/haras/Source/WebApps/current/crm/src/crm/group/models/__init__.py", line 7, in <module> from crm.group.models.group import Group File "/Users/haras/Source/WebApps/current/crm/src/crm/group/models/group.py", line 10, in <module> from crm.date.helpers.default_season import default_season File "/Users/haras/Source/WebApps/current/crm/src/crm/date/helpers/default_season.py", line 8, in <module> from ..models.season import Season File "/Users/haras/Source/WebApps/current/crm/src/crm/date/models/season.py", line 28, in <module> from ..admin import season File "/Users/haras/Source/WebApps/current/crm/src/crm/date/admin/season.py", line 17, in <module> admin.site.register( Season, SeasonAdmin ) File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/django/contrib/admin/sites.py", line 94, in register validate(admin_class, model) File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/django/contrib/admin/validation.py", line 20, in validate models.get_apps() File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/django/db/models/loading.py", line 104, in get_apps self._populate() File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/django/db/models/loading.py", line 60, in _populate self.load_app(app_name, True) File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/django/db/models/loading.py", line 87, in load_app models = import_module('.models', app_name) File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/django/utils/importlib.py", line 35, in import_module __import__(name) File "/Users/haras/Source/WebApps/current/crm/src/crm/feedback/models/__init__.py", line 7, in <module> from crm.feedback.models.feedback import Feedback File "/Users/haras/Source/WebApps/current/crm/src/crm/feedback/models/feedback.py", line 12, in <module> from category import Category File "/Users/haras/Source/WebApps/current/crm/src/crm/feedback/models/category.py", line 30, in <module> from ..admin import category File "/Users/haras/Source/WebApps/current/crm/src/crm/feedback/admin/category.py", line 16, in <module> admin.site.register( Category, CategoryAdmin ) File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/django/contrib/admin/sites.py", line 94, in register validate(admin_class, model) File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/django/contrib/admin/validation.py", line 20, in validate models.get_apps() File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/django/db/models/loading.py", line 104, in get_apps self._populate() File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/django/db/models/loading.py", line 60, in _populate self.load_app(app_name, True) File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/django/db/models/loading.py", line 87, in load_app models = import_module('.models', app_name) File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/django/utils/importlib.py", line 35, in import_module __import__(name) File "/Users/haras/Source/WebApps/current/crm/src/crm/payment/models/__init__.py", line 7, in <module> from crm.payment.models.payment import Payment File "/Users/haras/Source/WebApps/current/crm/src/crm/payment/models/payment.py", line 12, in <module> from crm.contract.models.contract import Contract ImportError: cannot import name Contract
Attachments (1)
Change History (15)
comment:1 by , 15 years ago
Summary: | Revision 12950 crashes models in directory → Revision 12950 crashes complex interdependent models in directory |
---|
comment:2 by , 15 years ago
I'd also be interested to know if turning off DEBUG makes the problem go away. Is it just the admin validation triggering the issue or is there a problem even without that? If the latter, the traceback from the DEBUG off case might help with recreating a simpler scenario to demonstrate the problem.
comment:3 by , 15 years ago
OK, I figured out a simpler scenario to trigger the problem. Given one app, blah
, with a models directory containing __init__.py
:
from blah.models.bar import Bar from blah.models.foo import Foo __all__ = ['Foo', 'Bar']
blah/models/bar.py
:
from django.db import models from ..admin import foo class Bar(models.Model): name = models.CharField(max_length=5) class Meta: app_label = 'blah'
blah/models/foo.py
:
from django.db import models class Foo(models.Model): name = models.CharField(max_length=5) class Meta: app_label = 'blah'
The blah
app also has an admin directory containing foo.py
:
from django.contrib import admin from blah.models.foo import Foo admin.site.register(Foo)
Plus another app listed in INSTALLED_APPS
, blahblah
, with a simple models.py
:
from blah.models.bar import Bar
Loading the blah app starts with an import of Bar from the blah/models/__init__.py
file. Before Bar is defined, the admin module for Foo is imported, triggering registration of Foo. With debug on, this results in a call to the admin validation code, which first attempts to ensure that all apps and models are fully loaded. This leads to an attempt to load the blahblah
models file, which since it attempts to import Bar (where we started), results in an ImportError.
Setting DEBUG to False makes the problem go away, since then we don't have an attempt to ensure all all models are fully loaded while we are in-process of importing a model. Also, moving the admin import in bar.py
after the definition of Bar fixes the problem. Finally this scenario does also fail on Django 1.0.4:
kmt@lbox:~/software/web/foo$ python manage.py shell Traceback (most recent call last): File "manage.py", line 11, in <module> execute_manager(settings) File "/home/kmt/django/Django-1.0.4/django/core/management/__init__.py", line 361, in execute_manager utility.execute() File "/home/kmt/django/Django-1.0.4/django/core/management/__init__.py", line 306, in execute self.fetch_command(subcommand).run_from_argv(self.argv) File "/home/kmt/django/Django-1.0.4/django/core/management/base.py", line 192, in run_from_argv self.execute(*args, **options.__dict__) File "/home/kmt/django/Django-1.0.4/django/core/management/base.py", line 219, in execute output = self.handle(*args, **options) File "/home/kmt/django/Django-1.0.4/django/core/management/base.py", line 348, in handle return self.handle_noargs(**options) File "/home/kmt/django/Django-1.0.4/django/core/management/commands/shell.py", line 18, in handle_noargs loaded_models = get_models() File "/home/kmt/django/Django-1.0.4/django/db/models/loading.py", line 136, in get_models self._populate() File "/home/kmt/django/Django-1.0.4/django/db/models/loading.py", line 57, in _populate self.load_app(app_name, True) File "/home/kmt/django/Django-1.0.4/django/db/models/loading.py", line 72, in load_app mod = __import__(app_name, {}, {}, ['models']) File "/home/kmt/software/web/foo/blah/models/__init__.py", line 1, in <module> from blah.models.bar import Bar File "/home/kmt/software/web/foo/blah/models/bar.py", line 3, in <module> from ..admin import foo File "/home/kmt/software/web/foo/blah/admin/foo.py", line 5, in <module> admin.site.register(Foo) File "/home/kmt/django/Django-1.0.4/django/contrib/admin/sites.py", line 77, in register validate(admin_class, model) File "/home/kmt/django/Django-1.0.4/django/contrib/admin/validation.py", line 22, in validate models.get_apps() File "/home/kmt/django/Django-1.0.4/django/db/models/loading.py", line 97, in get_apps self._populate() File "/home/kmt/django/Django-1.0.4/django/db/models/loading.py", line 57, in _populate self.load_app(app_name, True) File "/home/kmt/django/Django-1.0.4/django/db/models/loading.py", line 72, in load_app mod = __import__(app_name, {}, {}, ['models']) File "/home/kmt/software/web/foo/blahblah/models.py", line 1, in <module> from blah.models.bar import Bar ImportError: cannot import name Bar
So it's a case of an error situation that would have been reported by 1.0 but got hidden by the import changes in 1.1, and is now no longer hidden. Not sure what the best way to deal with that is.
by , 15 years ago
Attachment: | 13335.diff added |
---|
comment:4 by , 15 years ago
Has patch: | set |
---|---|
milestone: | → 1.2 |
Needs tests: | set |
Triage Stage: | Unreviewed → Accepted |
Possible fix attached. It fixes the problem for the scenario I came up with; I'd like to confirm it fixes the problem for the more complicated original scenario. Does not yet integrate tests.
follow-ups: 6 7 comment:5 by , 15 years ago
sample case I use my models
I hope this would help to solve problem
module crm.customer.models.country
""" Mateusz Harasymczuk mateusz@harasymczuk.pl www.mateusz.harasymczuk.pl """ from django.db import models from django.utils.translation import ugettext as _ from ..helpers import address_unify from ..helpers import string_unify from country import Country DEFAULT_COUNTRY = 1 class Document ( models.Model ): name = models.CharField( unique=True, max_length=20, verbose_name=_( 'Name' ) ) country = models.ForeignKey( Country, verbose_name=_( 'Country' ), default=DEFAULT_COUNTRY ) comment = models.TextField( verbose_name=_( 'Comment' ), unique=False, null=True, blank=True ) def __unicode__( self ): return '%s (%s)' % ( _( self.name ), _( self.country.name ) ) class Meta (): ordering = ( 'name', ) verbose_name = _( 'Document' ) verbose_name_plural = _( 'Documents' ) app_label = 'customer' from ..admin import document
module crm.customer.admin.document
from django.contrib import admin from ..models.country import Country class CountryAdmin ( admin.ModelAdmin ): list_select_related = True search_fields = ['name', 'iso_alpha_2', 'iso_alpha_3'] list_display = ( 'continent', 'name', 'iso_alpha_2', 'iso_alpha_3' ) list_display_links = ( 'name', ) list_filter = ( 'continent', ) admin.site.register( Country, CountryAdmin )
follow-up: 8 comment:6 by , 15 years ago
I pasted wrong code, this is ok
but you can see, all my models are importing its own admin module from file which is located in ../admin/ directory
(models are in model dir, admin files in admin dir, views in views dir, and so on...)
crm.contract.admin.document
from django.contrib import admin from ..models.document import Document import settings class DocumentAdmin ( admin.ModelAdmin ): class Media(): #js = ( settings.MEDIA_URL + '/js/crm/contract.js', ) css = { 'all': ( settings.MEDIA_URL + '/css/crm/contract/admin.css', ) } list_select_related = True search_fields = ['name', 'type'] list_display = ( 'name', 'type', 'sample_view' ) list_display_links = ( 'name', ) admin.site.register( Document, DocumentAdmin )
comment:7 by , 15 years ago
Replying to haras:
sample case I use my models
I hope this would help to solve problem
I don't need partial sample code, I've already recreated the basic problem evident in the traceback from the description. What I did not recreate was a project containing at least six different apps, interdependent in ways I cannot necessarily see from the traceback. The easiest way to see if the attached fix corrects the problem in your specific case is for you to try it out. I believe it will fix the problem, but it's clear your project is pretty extensive, and it's hard to be 100% sure that there is not something else in how you have things set up that is going to cause an issue.
follow-up: 9 comment:8 by , 15 years ago
Replying to haras:
I pasted wrong code, this is ok
but you can see, all my models are importing its own admin module from file which is located in ../admin/ directory
(models are in model dir, admin files in admin dir, views in views dir, and so on...)
Not that it really matters, but why are your models modules importing admin modules? That's (part of) the crux of the problem that is leading to a circular import, and is generally not necessary. Admin modules are certainly dependent on the models, but the reverse dependence seems a bit unusual.
comment:9 by , 15 years ago
Replying to kmtracey:
Not that it really matters, but why are your models modules importing admin modules? That's (part of) the crux of the problem that is leading to a circular import, and is generally not necessary. Admin modules are certainly dependent on the models, but the reverse dependence seems a bit unusual.
in other case admin models are not visible in admin panel
I partially solved circular import problem with importing admin model at the end of file with models, not at the beginning, it seems odd, but it works :}
I have to admit that I forgot about method I use in models case, adding admin models imports to admin/init.py and then relaying on admin autodiscover,
Yes this solves problem
I removed all
from ..admin import module
and it does not breaks at this revision anymore,
moreover I updated to HEAD - most recent one, from svn and there is no longer any problem
I have to apologize for posting this problem, but this revision breaks (or fixes) inappropriate method used by me
comment:10 by , 15 years ago
After applying your patch to r12950
to my unmodified revision (this without correcting ..admin problem)
produces much less traceback, but still
and no DEBUG = False, does not correct it for me
Validating models... Unhandled exception in thread started by <function inner_run at 0x101b0a1b8> Traceback (most recent call last): File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/django/core/management/commands/runserver.py", line 48, in inner_run self.validate(display_num_errors=True) File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/django/core/management/base.py", line 250, in validate num_errors = get_validation_errors(s, app) File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/django/core/management/validation.py", line 28, in get_validation_errors for (app_name, error) in get_app_errors().items(): File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/django/db/models/loading.py", line 142, in get_app_errors self._populate() File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/django/db/models/loading.py", line 63, in _populate self.load_app(app_name) File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/django/db/models/loading.py", line 83, in load_app models = import_module('.models', app_name) File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/django/utils/importlib.py", line 35, in import_module __import__(name) File "/Users/haras/Source/WebApps/current/crm/srcOld/crm/payment/models/__init__.py", line 7, in <module> from crm.payment.models.payment import Payment File "/Users/haras/Source/WebApps/current/crm/srcOld/crm/payment/models/payment.py", line 12, in <module> from crm.contract.models.contract import Contract File "/Users/haras/Source/WebApps/current/crm/srcOld/crm/contract/models/__init__.py", line 7, in <module> from crm.contract.models.contract import Contract File "/Users/haras/Source/WebApps/current/crm/srcOld/crm/contract/models/contract.py", line 14, in <module> from crm.group.models.group import Group File "/Users/haras/Source/WebApps/current/crm/srcOld/crm/group/models/__init__.py", line 7, in <module> from crm.group.models.group import Group File "/Users/haras/Source/WebApps/current/crm/srcOld/crm/group/models/group.py", line 91, in <module> from ..admin import group File "/Users/haras/Source/WebApps/current/crm/srcOld/crm/group/admin/group.py", line 8, in <module> from crm.contract.models.contract import Contract ImportError: cannot import name Contract
comment:11 by , 15 years ago
OK, this last problem is straightforward circular import bug in the app that was mistakenly hidden by the pre-r12950 behavior of blindly swallowing import errors on an attempt to load an app's models. The right fix for that is to remove the circular import in the app, which from the comments above I gather has been done successfully.
While removing all the ..admin imports from models modules may have also resolved the original problem described in this ticket, the report did bring to light that the r12950 code does not correctly handle import errors resulting from nested calls to load_app, so the fix in the patch is still necessary. Just need to add a test first....
comment:12 by , 15 years ago
Resolution: | → fixed |
---|---|
Status: | new → closed |
This is a bit more complicated than "models in a directory" not working after r12950. Models in a directory do still work, what isn't working is this particular case, which from the traceback involves the models modules importing admin modules, triggering admin registration, which is triggering admin validation, which in turn triggers more app loading, more model imports, more admin imports and registrations, more validation, until a point is reached where we try to import something that we're still in the process of importing. Netting down the trace, what we've got is:
That last one is triggering an ImportError because we are already in the process of importing Contract way up top. Prior to r12950 that import error would be suppressed. Now it's not.
Trying to come up with a somewhat simpler scenario to recreate the issue, but haven't managed to do that yet. I'd be interested to know if this case worked on Django 1.0?