Code

Opened 8 years ago

Closed 14 months ago

#2438 closed Bug (worksforme)

django.db.models get_apps() return different results according to the server used

Reported by: kilian <kilian.cavalotti@…> Owned by: nobody
Component: Core (Other) Version: master
Severity: Normal Keywords:
Cc: kilian.cavalotti@…, mattimustang@…, oliver@…, lrekucki@… Triage Stage: Accepted
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

Using apache2 + mod_python, the admin interface do not display all of the applications listed in INSTALLED_APPS, while all those applications correctly appear in the main admin page using the integrated web server.

Weirder is the fact that, for the missing app, the 'should-be' admin URLs for adding or editing objects (admin/app/object/add/) can be successfully accessed. It seems that the missing application is correctly seen and working (objects actually can be managed with direct URLs), but it's not shown on the main admin page.

I narrowed the problem to the fact that get_apps() from django.db.models does not return the same list when called from mod_python or from the integrated server. More precisely, it seems that for one of the applications, in db/models/loading.py, the load_app() function checking for 'models' attribute behaves differently according to the server used.

The hasattr(mod, 'models') call returns False for the hostManage application using mod_python, and True for the same application using the integrated server. Thus, it's not listed in _app_list, and is not displayed on the admin page.

Adding

h = hasattr(mod, 'models')
assert False, h

in the load_app() function from db/models/loading.py gives the following with mod_python:

VariableValue
app_name'dNA.hostManage'
hFalse
mod<module 'dNA.hostManage' from '/var/www/dNA/hostManage/__init__.pyc'>

I removed *.pyc, and checked permissions on my folders and directories, but they seems to be ok since I the hasattr(mod, 'models') works from the shell:

>>> mod = __import__('hostManage', '', '', ['models'])
>>> mod
<module 'hostManage' from '/var/www/dNA/hostManage/__init__.pyc'>
>>> hasattr(mod, 'models')
True

So, I can't figure out why, but the hasattr(mod, 'models') function returns different values according to the server used. And I guess it's not thewanted behavior. ;)


for reference, this issue has been discussed there and there

Attachments (0)

Change History (21)

comment:1 Changed 8 years ago by mtredinnick

  • Status changed from new to assigned

comment:2 Changed 8 years ago by mattimustang@…

  • Cc mattimustang@… added

I hit this exact same issue yesterday too when cutting over from django's dev server to apache2/mod_python 3.2.8. I have an app that calls get_model() and it wasn't working under mod_python.

The snippet below demonstrates it:

>>> from django.db.models.loading import get_model
>>> model = get_model('asset', 'Asset')
>>> repr(model)
'None'
>>> from nong.asset.models import Asset
>>> model = get_model('asset', 'Asset')
>>> repr(model)
"<class 'nong.asset.models.Asset'>"

comment:3 Changed 8 years ago by mtredinnick

The mattimustang comment has nothing to do with the initial bug report, as far as I can see. However, it is addressed/fixed in [3490].

comment:4 Changed 8 years ago by mtredinnick

A status update:

After a lot of back and forth in email with Kilian, we now understand why this is happening, although a fix still remains to be found. Kilian had a model that was using a queryset on a model as a constructor to another field. Something like this:

class ModelA(models.Model):
   # ...

class ModelB(models.Model):
   some_field = models.CharField(choices = ModelA.objects.all(), ...)

and this was causing some problems with the app/model cache in django.db.models.loading.

The fix for now is "don't to do that", but I would like to get a better solution in the end.

comment:5 Changed 8 years ago by oliver@…

  • Cc oliver@… added

Interesting. I encountered a very similar (or the same) issue this morning, and tracked down this bug report.

One other twist that may or may not be related to the original case. In my case everything works (or fails to) exactly as described by killian, but removing certain lines from my models makes them reappear on the django admin interface:

class Item( models.Model ):

objects = ItemManager()


#The limit_choices_to in this line will cause this model to disaapear from django admin
type_code = models.ForeignKey( DbCode, null=False, blank=False, limit_choices_to = { 'parent_codeexact' : DbCode.objects.filter( label='ItemType')[0].id } )

DbCode in this code is just another Model used to store constants in the database (a code table). Eliminating limit_choices_to causes this model to appear. Perhaps that will help someone else find a workaround.

In another instance one of my Models which loads a queryset at global scope won't be visible on the admin page unless I remove the global query:

# A bunch of model and manager definitions preceed this slightly hackish code
try:

foo = codes
print "codes defined"

except:

try:

codes = DbCode.objects.build_code_objects()

except:

print "Warning codes object not built."

There do appear to be differences in import using the test server and mod_python/fastcgi. Tracing another bug in my code recently I found that:

from django.contrib import auth.models.User

would fail under mod_python/fastcgi but worked fine with the test server. (syntax is from memory, ymmv)

Hope this helps!

comment:6 Changed 8 years ago by anonymous

Yuck. Here are those snippets wikified, my appologies:

class Item( models.Model ):
    objects = ItemManager()
    
    #The limit_choices_to in this line will cause this model to disaapear from django admin
    type_code     = models.ForeignKey( DbCode, null=False, blank=False, limit_choices_to = { 'parent_code__exact' : DbCode.objects.filter( label='ItemType')[0].id } )

and

#Global scope
try:
    foo = codes
    print "codes defined"
except:
    try:
        codes = DbCode.objects.build_code_objects()
    except:
        print "Warning codes object not built."

comment:7 Changed 7 years ago by Simon G. <dev@…>

  • Triage Stage changed from Unreviewed to Accepted

comment:8 Changed 7 years ago by portland@…

This problem seems to have popped up again in a different context. First, some version numbers (joy):

Django SVN 5609
Python 2.5.1
mod_python 3.2.10
Apache 2.3.3

I'm trying to implement django-comment-utils, the package James Bennett cooked up for comment moderation. I'm using revision 72 of the SVN codebase, and I'm encountering the same issues with the admin site (app doesn't show up in admin list, though its add/edit/list pages are still available and work fine). The culprit appears to be the exact same problem&mdash;get_apps() returns a different set of applications depending on whether I'm calling it from mod_python, the development server, or shell (the latter two work, the former doesn't). Specifically, if I turn on moderation on a model, the application it's associated with disappears. (See http://code.google.com/p/django-comment-utils/issues/detail?id=4 for more detail)

The tricky part is none of the affected models uses a QuerySet in constructing a model field, and a cursory glance at the comment-utils code reveals nothing similar. Here's the model itself:

class Story(models.Model):
    head = models.CharField(maxlength=255)
    deck = models.CharField(maxlength=255, blank=True, null=True)
    slug = models.SlugField(prepopulate_from=("head",), help_text='Automatically written based on the headline. If nothing shows up here, try typing in the headline instead of copying and pasting.')
    section = models.ForeignKey(Section)
    issue = models.ForeignKey(Issue)
    label = models.CharField(maxlength=255, blank=True, help_text='ex. "Signed Editorial," "Feature." Optional.')
    content = models.TextField()
    summary = models.TextField(help_text='Sum up the story in a single paragraph.')
    section_order = models.PositiveSmallIntegerField("Order in section", help_text="Determines the order of all stories in the section, with lower numbers at the top (so a story with order priority 1 would be at the top).")
    enable_comments = models.BooleanField(default=True)

    class Admin:
        fields = (
            ('Top matter', {'fields': ('head', 'deck', 'summary', 'label')}),
            ('Content', {'fields': ('content',), 'classes': 'richedit'}),
            ('Metadata', {'fields': ('issue', 'section', 'section_order', 'slug')}),
            ('Comments', {'fields': ('enable_comments',)}),
        )
        list_display    = ('head', 'list_authors', 'issue', 'section',)
        list_filter     = ('issue', 'section',)
        search_fields   = ['head', 'deck']
        ordering        = ('-issue',)

        js = ('js/journal/formatting-controls.js',)

    class Meta:
        verbose_name_plural     = "Stories"
        unique_together         = (("head","issue"),("issue","section","section_order"))
        ordering                = ('-issue', 'section_order')
    
    def __unicode__(self):
        return self.head

    def get_absolute_url(self):
        return ('journal.stories.views.detail_story', (), {
            'datestring': self.issue.pub_date.strftime("%Y-%m-%d"),
            'section': self.section.slug,
            'slug': self.slug})
    get_absolute_url = models.permalink(get_absolute_url)

comment:9 Changed 7 years ago by portland@…

Some further digging into the issue: I think the problem is Django isn't handling some class references very well. The culprit in my case turned out to be that in order to turn on comment moderation, I had to insert the following code somewhere:

class StoryModerator(AlwaysModerate):
    akismet = True
    akismet_auto_delete = True
    enable_field = 'enable_comments'

moderator.register(Story, StoryModerator)

where Story is the name of the model I'm trying to moderate. If I put that code in the same models.py where Story lives, the trouble with the admin app and get_apps() not working properly surfaces. If I pull the code out of models.py and call it from a separate file (in this case a file imported by views.py) then everything works fine. Similarly, the problem with using QuerySets in the field constructor above may not be the use of a QuerySet in and of itself, but specifically the call to another model in the same models.py file.

This might not be news to anyone except me, but it wasn't obvious from the comments above and I thought it was worth mentioning just in case.

comment:10 Changed 7 years ago by mtredinnick

  • Resolution set to fixed
  • Status changed from assigned to closed

(In [5919]) Rewrote portions of the app- and model-cache initialisation to handle some corner cases. It is now possible to use m2m relations before everything is imported and still get the right results later when importing is complete. Also, get_apps() should always return the same results, so apps won't randomly disappear in the admin interface.

Also reorganised the structure of loading.py, since the number of global variables was exploding. The public API is still backwards compatible.

Fixed #1796 and #2438 (he claims, optimistically).

comment:11 Changed 7 years ago by aaraines@…

  • Resolution fixed deleted
  • Status changed from closed to reopened
  • Version set to SVN

I've been hitting a problem with loading.py the last week or so, in the same vein as this thread. One app doesn't return in get_apps(), and it's because of the hasattr(mod, "models") check in load_app(). It started happening after I added a particular model the file, but I've tried removing it and the admin still leaves out the app (it's a clean deploy, so no pyc is left around). I've added a getattr() in load_app() to see what exactly is failing, but it only gives me an ImportError, which is what I already knew.

Works fine with runserver, but not with mod_python 3.3.1/Python 2.4.4 or 2.5.1. I'm at version 6023 of django/trunk.
The models file is at http://draines.com/tmp/foo-models.py.txt.

comment:12 Changed 7 years ago by Aidas Bendoraitis aka Archatas

I experienced this bug/feature as well. After analyzing the core code of django I found out a workaround for the problem.

If you need to assign some limit_choices_to or default value which is related to the instances of other models, you can create a function which returns the necessary value and assign the function (not called) to the limit_choices_to or default attributes. It won't be called at the load time, so the modules will be loaded correctly, but the function will be called at run time, for instance, when the administration is loaded.

comment:13 Changed 6 years ago by semmons99@…

I had similar problems with my apps not all showing up when using apache2.2 vs. runserver. I added the directory to the project to my python path in httpd.conf instead of just the parent folder:

Before:

<Location "/">
  SetHandler python-program
  PythonPath "['c:/DjangoSites'] + sys.path"
  PythonHandler django.core.handlers.modpython
  SetEnv DJANGO_SETTINGS_MODULE yourdata.settings
  PythonDebug On
</Location>

After:

<Location "/">
  SetHandler python-program
  PythonPath "['c:/DjangoSites', 'c:/DjangoSites/yourdata'] + sys.path"
  PythonHandler django.core.handlers.modpython
  SetEnv DJANGO_SETTINGS_MODULE yourdata.settings
  PythonDebug On
</Location>

comment:14 Changed 6 years ago by kilian

I'm posting the following comment on behalf of Dennis Parker:


I just experienced a slightly different form of this problem.



I have several apps in my site, and one of them stopped showing up on the admin home page, but was accessible by direct url.  The usual "works in dev server, doesn't work in mod_python" stuff applies. Python2.5, Apache 2.0.x and 2.2.x latest mod_python, django 7716. I read the comments on this ticket, and was able to change my code to eliminate the problem.



In my case, the app (call it "App A") that was creating the problem was NOT the one that failed to appear (call it "App B"). The offending app (App A) only had a couple of admin enabled model classes, so I don't know how much of that module was being processed successfully. What I do know is that App A had some (non-model) classes in it that did some object creation of shared instances as class attributes of other classes in the same module.



Based on a hunch generated by the comments on this page, I moved the class attribute bearing, non-model classes in App A to a different module, and App B showed up in the admin page again.



The only additional piece of data that might be relevant is that the non-model classes in question use SQLAlchemy to access a different database schema (not my model schema), and I am using the SQLAlchemy connection pooling trick for my django database (MySQL) backend, so there may be some database connection startup issues involved.



I don't know if this will help debug the ultimate problem, but I hope that anyone else who has this problem will see that they should look for initialization issues in all their app models, not just the app that disappears.

comment:15 Changed 6 years ago by brosner

  • Component changed from Admin interface to Core framework

comment:16 Changed 3 years ago by lrekucki

  • Cc lrekucki@… added

comment:17 Changed 3 years ago by lrekucki

  • Type changed from defect to Bug

comment:18 Changed 3 years ago by lrekucki

  • Severity changed from normal to Normal

comment:19 Changed 2 years ago by aaugustin

  • UI/UX unset

Change UI/UX from NULL to False.

comment:20 Changed 2 years ago by aaugustin

  • Easy pickings unset

Change Easy pickings from NULL to False.

comment:21 Changed 14 months ago by claudep

  • Resolution set to worksforme
  • Status changed from reopened to closed

No new info since 5 years, and mod_python is now gone. Reopen if reproducible with recent code.

Add Comment

Modify Ticket

Change Properties
<Author field>
Action
as closed
as The resolution will be set. Next status will be 'closed'
The resolution will be deleted. Next status will be 'new'
Author


E-mail address and user name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.