django.db.models get_apps() return different results according to the server used

I hit this exact same issue yesterday too when cutting over from django's dev server to apache2/mod_python 3.2.8. I have an app that calls get_model() and it wasn't working under mod_python.

The snippet below demonstrates it:

>>> from django.db.models.loading import get_model
>>> model = get_model('asset', 'Asset')
>>> repr(model)
'None'
>>> from nong.asset.models import Asset
>>> model = get_model('asset', 'Asset')
>>> repr(model)
"<class 'nong.asset.models.Asset'>"

The mattimustang comment has nothing to do with the initial bug report, as far as I can see. However, it is addressed/fixed in [3490].

A status update:

After a lot of back and forth in email with Kilian, we now understand why this is happening, although a fix still remains to be found. Kilian had a model that was using a queryset on a model as a constructor to another field. Something like this:

class ModelA(models.Model):
   # ...

class ModelB(models.Model):
   some_field = models.CharField(choices = ModelA.objects.all(), ...)

and this was causing some problems with the app/model cache in django.db.models.loading.

The fix for now is "don't to do that", but I would like to get a better solution in the end.

Interesting. I encountered a very similar (or the same) issue this morning, and tracked down this bug report.

One other twist that may or may not be related to the original case. In my case everything works (or fails to) exactly as described by killian, but removing certain lines from my models makes them reappear on the django admin interface:

class Item( models.Model ):

objects = ItemManager?()

#The limit_choices_to in this line will cause this model to disaapear from django admin type_code = models.ForeignKey( DbCode?, null=False, blank=False, limit_choices_to = { 'parent_codeexact' : DbCode?.objects.filter( label='ItemType?')[0].id } )

DbCode? in this code is just another Model used to store constants in the database (a code table). Eliminating limit_choices_to causes this model to appear. Perhaps that will help someone else find a workaround.

In another instance one of my Models which loads a queryset at global scope won't be visible on the admin page unless I remove the global query:

# A bunch of model and manager definitions preceed this slightly hackish code try:

foo = codes print "codes defined"

except:

try:

codes = DbCode?.objects.build_code_objects()

except:

print "Warning codes object not built."

There do appear to be differences in import using the test server and mod_python/fastcgi. Tracing another bug in my code recently I found that:

from django.contrib import auth.models.User

would fail under mod_python/fastcgi but worked fine with the test server. (syntax is from memory, ymmv)

Hope this helps!

Yuck. Here are those snippets wikified, my appologies:

class Item( models.Model ):
    objects = ItemManager()
    
    #The limit_choices_to in this line will cause this model to disaapear from django admin
    type_code     = models.ForeignKey( DbCode, null=False, blank=False, limit_choices_to = { 'parent_code__exact' : DbCode.objects.filter( label='ItemType')[0].id } )

and

#Global scope
try:
    foo = codes
    print "codes defined"
except:
    try:
        codes = DbCode.objects.build_code_objects()
    except:
        print "Warning codes object not built."

This problem seems to have popped up again in a different context. First, some version numbers (joy):

Django SVN 5609
Python 2.5.1
mod_python 3.2.10
Apache 2.3.3

I'm trying to implement django-comment-utils, the package James Bennett cooked up for comment moderation. I'm using revision 72 of the SVN codebase, and I'm encountering the same issues with the admin site (app doesn't show up in admin list, though its add/edit/list pages are still available and work fine). The culprit appears to be the exact same problem—get_apps() returns a different set of applications depending on whether I'm calling it from mod_python, the development server, or shell (the latter two work, the former doesn't). Specifically, if I turn on moderation on a model, the application it's associated with disappears. (See http://code.google.com/p/django-comment-utils/issues/detail?id=4 for more detail)

The tricky part is none of the affected models uses a QuerySet? in constructing a model field, and a cursory glance at the comment-utils code reveals nothing similar. Here's the model itself:

class Story(models.Model):
    head = models.CharField(maxlength=255)
    deck = models.CharField(maxlength=255, blank=True, null=True)
    slug = models.SlugField(prepopulate_from=("head",), help_text='Automatically written based on the headline. If nothing shows up here, try typing in the headline instead of copying and pasting.')
    section = models.ForeignKey(Section)
    issue = models.ForeignKey(Issue)
    label = models.CharField(maxlength=255, blank=True, help_text='ex. "Signed Editorial," "Feature." Optional.')
    content = models.TextField()
    summary = models.TextField(help_text='Sum up the story in a single paragraph.')
    section_order = models.PositiveSmallIntegerField("Order in section", help_text="Determines the order of all stories in the section, with lower numbers at the top (so a story with order priority 1 would be at the top).")
    enable_comments = models.BooleanField(default=True)

    class Admin:
        fields = (
            ('Top matter', {'fields': ('head', 'deck', 'summary', 'label')}),
            ('Content', {'fields': ('content',), 'classes': 'richedit'}),
            ('Metadata', {'fields': ('issue', 'section', 'section_order', 'slug')}),
            ('Comments', {'fields': ('enable_comments',)}),
        )
        list_display    = ('head', 'list_authors', 'issue', 'section',)
        list_filter     = ('issue', 'section',)
        search_fields   = ['head', 'deck']
        ordering        = ('-issue',)

        js = ('js/journal/formatting-controls.js',)

    class Meta:
        verbose_name_plural     = "Stories"
        unique_together         = (("head","issue"),("issue","section","section_order"))
        ordering                = ('-issue', 'section_order')
    
    def __unicode__(self):
        return self.head

    def get_absolute_url(self):
        return ('journal.stories.views.detail_story', (), {
            'datestring': self.issue.pub_date.strftime("%Y-%m-%d"),
            'section': self.section.slug,
            'slug': self.slug})
    get_absolute_url = models.permalink(get_absolute_url)

Some further digging into the issue: I think the problem is Django isn't handling some class references very well. The culprit in my case turned out to be that in order to turn on comment moderation, I had to insert the following code somewhere:

class StoryModerator(AlwaysModerate):
    akismet = True
    akismet_auto_delete = True
    enable_field = 'enable_comments'

moderator.register(Story, StoryModerator)

where Story is the name of the model I'm trying to moderate. If I put that code in the same models.py where Story lives, the trouble with the admin app and get_apps() not working properly surfaces. If I pull the code out of models.py and call it from a separate file (in this case a file imported by views.py) then everything works fine. Similarly, the problem with using QuerySets? in the field constructor above may not be the use of a QuerySet? in and of itself, but specifically the call to another model in the same models.py file.

This might not be news to anyone except me, but it wasn't obvious from the comments above and I thought it was worth mentioning just in case.

(In [5919]) Rewrote portions of the app- and model-cache initialisation to handle some corner cases. It is now possible to use m2m relations before everything is imported and still get the right results later when importing is complete. Also, get_apps() should always return the same results, so apps won't randomly disappear in the admin interface.

Also reorganised the structure of loading.py, since the number of global variables was exploding. The public API is still backwards compatible.

Fixed #1796 and #2438 (he claims, optimistically).

I've been hitting a problem with loading.py the last week or so, in the same vein as this thread. One app doesn't return in get_apps(), and it's because of the hasattr(mod, "models") check in load_app(). It started happening after I added a particular model the file, but I've tried removing it and the admin still leaves out the app (it's a clean deploy, so no pyc is left around). I've added a getattr() in load_app() to see what exactly is failing, but it only gives me an ImportError, which is what I already knew.

Works fine with runserver, but not with mod_python 3.3.1/Python 2.4.4 or 2.5.1. I'm at version 6023 of django/trunk. The models file is at http://draines.com/tmp/foo-models.py.txt.

I experienced this bug/feature as well. After analyzing the core code of django I found out a workaround for the problem.

If you need to assign some limit_choices_to or default value which is related to the instances of other models, you can create a function which returns the necessary value and assign the function (not called) to the limit_choices_to or default attributes. It won't be called at the load time, so the modules will be loaded correctly, but the function will be called at run time, for instance, when the administration is loaded.

I had similar problems with my apps not all showing up when using apache2.2 vs. runserver. I added the directory to the project to my python path in httpd.conf instead of just the parent folder:

Before:

<Location "/">
  SetHandler python-program
  PythonPath "['c:/DjangoSites'] + sys.path"
  PythonHandler django.core.handlers.modpython
  SetEnv DJANGO_SETTINGS_MODULE yourdata.settings
  PythonDebug On
</Location>

After:

<Location "/">
  SetHandler python-program
  PythonPath "['c:/DjangoSites', 'c:/DjangoSites/yourdata'] + sys.path"
  PythonHandler django.core.handlers.modpython
  SetEnv DJANGO_SETTINGS_MODULE yourdata.settings
  PythonDebug On
</Location>

I'm posting the following comment on behalf of Dennis Parker:

I just experienced a slightly different form of this problem.

I have several apps in my site, and one of them stopped showing up on the admin home page, but was accessible by direct url.  The usual "works in dev server, doesn't work in mod_python" stuff applies. Python2.5, Apache 2.0.x and 2.2.x latest mod_python, django 7716. I read the comments on this ticket, and was able to change my code to eliminate the problem.

In my case, the app (call it "App A") that was creating the problem was NOT the one that failed to appear (call it "App B"). The offending app (App A) only had a couple of admin enabled model classes, so I don't know how much of that module was being processed successfully. What I do know is that App A had some (non-model) classes in it that did some object creation of shared instances as class attributes of other classes in the same module.

Based on a hunch generated by the comments on this page, I moved the class attribute bearing, non-model classes in App A to a different module, and App B showed up in the admin page again.

The only additional piece of data that might be relevant is that the non-model classes in question use SQLAlchemy to access a different database schema (not my model schema), and I am using the SQLAlchemy connection pooling trick for my django database (MySQL) backend, so there may be some database connection startup issues involved.

I don't know if this will help debug the ultimate problem, but I hope that anyone else who has this problem will see that they should look for initialization issues in all their app models, not just the app that disappears.