Iteration of a Many To Many Relationship raises Field Error

[callen]

As seen here: ​http://gist.github.com/337546

Cannot resolve keyword 'bimport' into field. Choices are: archive_column, archive_order, article, construct, day, edition, frequency, frequency_text, from_email, html_template, id, is_public, list_name, name, newsletterarticle, newsletterarticles, slug, subject, subject_type, subscription, text_template, time

# Caused by

# this is a many-to-many relationship
for nl in bimport.newsletters.all():

# Here's the trace:

for nl in bimport.newsletters.all():
File "/Library/Python/2.5/site-packages/django/db/models/manager.py", line 105, in all
return self.get_query_set()

File "/Library/Python/2.5/site-packages/django/db/models/fields/related.py", line 424, in get_query_set

return superclass.get_query_set(self)._next_is_sticky().filter((self.core_filters))

File "/Library/Python/2.5/site-packages/django/db/models/query.py", line 498, in filter
return self._filter_or_exclude(False, *args, kwargs)

File "/Library/Python/2.5/site-packages/django/db/models/query.py", line 516, in _filter_or_exclude

clone.query.add_q(Q(*args, kwargs))

File "/Library/Python/2.5/site-packages/django/db/models/sql/query.py", line 1675, in add_q
can_reuse=used_aliases)

File "/Library/Python/2.5/site-packages/django/db/models/sql/query.py", line 1569, in add_filter
negate=negate, process_extras=process_extras)

File "/Library/Python/2.5/site-packages/django/db/models/sql/query.py", line 1737, in setup_joins
"Choices are: %s" % (name, ", ".join(names)))

[Karen Tracey]

I don't see specifics of the models in use here linked anywhere -- you've shown a single line of code without any context for what its variables might be, exactly. Without specifics of the models (preferably reduced to a minimal form, removing extraneous fields that are not required to recreate the error) and some context for what's happened prior to that single line of code, I don't see how anyone could figure out what might be going on. Also, please use WikiFormatting and preview when pasting tracebacks and code.

[callen]

Give me a chance to edit it before you close it.

[callen]

Replying to callen:

As seen here: ​http://gist.github.com/337546

Cannot resolve keyword 'bimport' into field. Choices are: archive_column, archive_order, article, construct, day, edition, frequency, frequency_text, from_email, html_template, id, is_public, list_name, name, newsletterarticle, newsletterarticles, slug, subject, subject_type, subscription, text_template, time

# Caused by

# this is a many-to-many relationship
for nl in bimport.newsletters.all():

Relevant models here:

class Bimport(models.Model):
    content_type = models.ForeignKey(ContentType)
    source = models.ForeignKey(Source)
    import_date = models.DateField(auto_now_add = True)
    datafile = models.FileField(upload_to='bimport/uploads')
    newsletters = models.ManyToManyField(Newsletter, help_text=None, null=True)
    error = models.CharField(max_length=200)
    status = models.CharField(choices=STATUS_CHOICES, default='N', max_length=100)

class Newsletter(models.Model):
    name = models.CharField(max_length=50)
    html_template = models.CharField(max_length=300, default="merx/newsletters/default.html",verbose_name='HTML Template')
    text_template = models.CharField(max_length=300, default="merx/newsletters/default.txt",verbose_name='Text Template')
    day = models.IntegerField(null=True, blank=True, choices=DAYS)
    time = models.TimeField(null=True, blank=True)
    frequency = models.IntegerField(blank=True,null=True,choices=FREQUENCY)
    frequency_text = models.CharField(max_length=200,blank=True,null=True,verbose_name='Frequency Display Text')
    slug = models.SlugField()
    construct = models.ForeignKey(Construct,blank=True,null=True)
    is_public = models.BooleanField(blank=False,null=False,default=False,verbose_name='Public')
    list_name = models.CharField(max_length=30, blank=True, null=True)
    archive_column = models.IntegerField(blank=False,null=False,default=0)
    archive_order = models.IntegerField(blank=False,null=False,default=0)
    from_email = models.CharField(
        max_length=300, 
        null=True, 
        blank=True,
        help_text="Please use the following format: [Newsletter Name] <[email]@domain.com>"
        )
    subject = models.CharField(max_length=300,blank=True,null=True,help_text='Can contain one string substitute as defined by subject type.')
    subject_type = models.IntegerField(blank=True,null=True,choices=SUBJECT_TYPE,help_text="Object used as string substitute for subject.")

Here's the trace:

for nl in bimport.newsletters.all():
File "/Library/Python/2.5/site-packages/django/db/models/manager.py", line 105, in all
return self.get_query_set()

File "/Library/Python/2.5/site-packages/django/db/models/fields/related.py", line 424, in get_query_set

return superclass.get_query_set(self)._next_is_sticky().filter(**(self.core_filters))

File "/Library/Python/2.5/site-packages/django/db/models/query.py", line 498, in filter
return self._filter_or_exclude(False, *args, **kwargs)

File "/Library/Python/2.5/site-packages/django/db/models/query.py", line 516, in _filter_or_exclude

clone.query.add_q(Q(*args, **kwargs))

File "/Library/Python/2.5/site-packages/django/db/models/sql/query.py", line 1675, in add_q
can_reuse=used_aliases)

File "/Library/Python/2.5/site-packages/django/db/models/sql/query.py", line 1569, in add_filter
negate=negate, process_extras=process_extras)

File "/Library/Python/2.5/site-packages/django/db/models/sql/query.py", line 1737, in setup_joins
"Choices are: %s" % (name, ", ".join(names)))

[Karen Tracey]

What is the variable bimport? Guessing the obvious -- that it is an instance of Bimport -- I cannot recreate the problem:

>>> from ttt.models import Bimport
>>> bimport = Bimport.objects.all()[0]
>>> bimport.newsletters.all()
[<Newsletter: Newsletter object>]
>>> import django
>>> django.get_version()
u'1.2 beta 1 SVN-12805'
>>> for nl in bimport.newsletters.all():
...     print nl.name
...
first
>>>

[callen]

Replying to kmtracey:

What is the variable bimport? Guessing the obvious -- that it is an instance of Bimport -- I cannot recreate the problem:

>>> from ttt.models import Bimport
>>> bimport = Bimport.objects.all()[0]
>>> bimport.newsletters.all()
[<Newsletter: Newsletter object>]
>>> import django
>>> django.get_version()
u'1.2 beta 1 SVN-12805'
>>> for nl in bimport.newsletters.all():
...     print nl.name
...
first
>>>

You're correct that bimport is just a Bimport instance. As I said, the issue cannot be reproduced in a shell because it works for me as well.

In [1]: from bimport.models import Bimport

In [2]: bi = Bimport.objects.get(id = 20)

In [3]: for nl in bi.newsletters.all():
   ...:     print nl
   ...:     
Soccer America Classifieds
Youth Soccer Insider

I can do that too. However, when it's running in the actual web app:

ipdb> 
> /Users/callen/code/socceramerica/bimport/controllers.py(36)process_bimport()
     35         bi.save()
---> 36         if int(bi.content_type.id) == 116:
     37             code = import_subs(bi)

ipdb> 
> /Users/callen/code/socceramerica/bimport/controllers.py(37)process_bimport()
     36         if int(bi.content_type.id) == 116:
---> 37             code = import_subs(bi)
     38             if code == 1:

ipdb> bi
Out[0]: <Bimport: Bimport object>

ipdb> pdoc bi
Bimport(id, content_type_id, source_id, import_date, datafile, error, status)

ipdb> pinfo bi
Type:		Bimport
Base Class:	<class 'bimport.models.Bimport'>
String Form:	Bimport object
Namespace:	Locals
File:		/Users/callen/code/socceramerica/bimport/models.py
Source:
class Bimport(models.Model):
    content_type = models.ForeignKey(ContentType)
    source = models.ForeignKey(Source)
    import_date = models.DateField(auto_now_add = True)
    datafile = models.FileField(upload_to='bimport/uploads')
    newsletters = models.ManyToManyField(Newsletter, help_text=None, null=True)
    error = models.CharField(max_length=200)
    status = models.CharField(choices=STATUS_CHOICES, default='N', max_length=100)

ipdb> bi.newsletters.all()
*** FieldError: Cannot resolve keyword 'bimport' into field. Choices are: archive_column, archive_order, article, construct, day, edition, frequency, frequency_text, from_email, html_template, id, is_public, list_name, name, newsletterarticle, newsletterarticles, slug, subject, subject_type, subscription, text_template, time

[callen]

These are the view + controllers that are causing the error.

​http://gist.github.com/337876

View:

def bimport_add(request):
    error = None
    if request.method == "POST":
        biform = BimportForm(request.POST, request.FILES)
        if biform.is_valid():
            bimport = biform.save()
            #p = Popen(['python', 'utils/processing.py', '%s' % bimport.id])
            process_bimport(bimport)
    else:
        biform = BimportForm()
    return render_to_response('dashboard/form.html', {
        'form':biform,
        'error':error
    },  context_instance=RequestContext(request))

Controller:

import xlrd
from merx.models import Subscription, Newsletter
import datetime
from dashboard.controllers import add_time
from time import sleep
 
def load_sheet(datafile):
    return xlrd.open_workbook(datafile).sheet_by_index(0)
    
def import_subs(bimport):
    sh = load_sheet(bimport.datafile.path)
    for r in range(sh.nrows):
        email = sh.cell(r, 0)
        for nl in bimport.newsletters.all():
            sub = Subscription.objects.create(
                newsletter = nl, 
                email = email, 
                public = 0, 
                status = 1, 
                start_date = datetime.date.today(),
                end_date = add_time(datetime.date.today(), 12)
                )
            bis = Bimport_Subscription.objects.create(
                bimport = bimport,
                email = sub.email,
                conversion_date = None
            )
    return 1
    
def process_bimport(bimport):
    import ipdb; ipdb.set_trace()
    bi = bimport
    if not bi.status == 'P':
        bi.status = 'P'
        bi.save()
        if int(bi.content_type.id) == 116:
            code = import_subs(bi)
            if code == 1:
                bi.status = 'D'
                bi.error = ''
                bi.save()
            else:
                bi.status = 'E'
                
        else:
            bi.status = 'E'
            bi.error = "Content type unrecognized, don't know what to do with this bulk import."
            bi.save()

[callen]

I've found a reliable way to reproduce it, via the "shell_plus" manage.py command provided by django-command-extensions.

I've included the shell session that reproduced the error as well as including the source to the shell_plus command.

This gives me a couple guesses as to what is going on, but I'm not 100% certain.

patras:socceramerica callen$ python ./manage.py shell_plus
From 'sites' autoload: Site
From 'auth' autoload: Permission, Group, User, Message
From 'contenttypes' autoload: ContentType
From 'sessions' autoload: Session
From 'admin' autoload: LogEntry
From 'comments' autoload: Comment, CommentFlag
From 'flatpages' autoload: FlatPage
From 'redirects' autoload: Redirect
From 'venus' autoload: MenuType, SubType, Category, Author, Type, Tag, Article, ArticleMeta, Change, ArticleOffsite, TypeFormField, InlineImage, Note, SphinxCounter
From 'mailer' autoload: Message, DontSendEntry, MessageLog
From 'base' autoload: Category, Type, Region, Tournament
From 'account' autoload: State, Country, Source, Involvement, InvolvementLevel, TransactionType, PaymentType, CreditCardType, UpdateType, Effort, MembershipType, Address, UserExtra, CreditCard, Transaction, Update, Notification, Decline, Issue, IssuePurchases
From 'merx' autoload: Construct, Newsletter, NewsletterArticle, MasterList, Subscription, Edition, EditionContent, ApiEdition, ApiCall, ApiLog, BadEmail
From 'migration' autoload: SaTournament, AuthorMap, OldState, OldCountry, CommentAuthorMap, SubscriptionHash, Image, ArticleTags, ArticleType, Articles, ArticlesAdmin, ArticlesDigest, ArticlesImages, ArticlesMagazine, Authors, AuthorsBlog, AwardsYsa, Dborderitems, Dborderpayments, Cctemp, MigrationCc, CompRenewals, Members, MembersHistory, MembersLogChanges, MembersLogVisits, MembersRemoved, MembersSubEmail, MembersSubMagazine, Pemailexpiration, Pmemberships, WorldCupWatchComments, MlsConfidentialComments, YouthSoccerInsiderComments, SoccerBusinessInsiderComments, GrassrootsSoccerBizComments, SoccerTalkComments, PaypalUser
Python 2.5.1 (r251:54863, Feb  6 2009, 19:02:12) 
Type "copyright", "credits" or "license" for more information.

IPython 0.9.1 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object'. ?object also works, ?? prints more.

In [1]: Newsletter.objects.get(id = 2)
---------------------------------------------------------------------------
DoesNotExist                              Traceback (most recent call last)

/Users/callen/code/socceramerica/<ipython console> in <module>()

/Library/Python/2.5/site-packages/django/db/models/manager.pyc in get(self, *args, **kwargs)
    118 
    119     def get(self, *args, **kwargs):
--> 120         return self.get_query_set().get(*args, **kwargs)
    121 
    122     def get_or_create(self, **kwargs):

/Library/Python/2.5/site-packages/django/db/models/query.pyc in get(self, *args, **kwargs)
    303         if not num:
    304             raise self.model.DoesNotExist("%s matching query does not exist."
--> 305                     % self.model._meta.object_name)
    306         raise self.model.MultipleObjectsReturned("get() returned more than one %s -- it returned %s! Lookup parameters were %s"
    307                 % (self.model._meta.object_name, num, kwargs))

DoesNotExist: Newsletter matching query does not exist.

In [2]: Newsletter.objects.get(id = 10)
---------------------------------------------------------------------------
DoesNotExist                              Traceback (most recent call last)

/Users/callen/code/socceramerica/<ipython console> in <module>()

/Library/Python/2.5/site-packages/django/db/models/manager.pyc in get(self, *args, **kwargs)
    118 
    119     def get(self, *args, **kwargs):
--> 120         return self.get_query_set().get(*args, **kwargs)
    121 
    122     def get_or_create(self, **kwargs):

/Library/Python/2.5/site-packages/django/db/models/query.pyc in get(self, *args, **kwargs)
    303         if not num:
    304             raise self.model.DoesNotExist("%s matching query does not exist."
--> 305                     % self.model._meta.object_name)
    306         raise self.model.MultipleObjectsReturned("get() returned more than one %s -- it returned %s! Lookup parameters were %s"
    307                 % (self.model._meta.object_name, num, kwargs))

DoesNotExist: Newsletter matching query does not exist.

In [3]: Newsletter.objects.get(id = 80)
Out[3]: <Newsletter: Soccer on TV>

In [4]: Bimport
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)

/Users/callen/code/socceramerica/<ipython console> in <module>()

NameError: name 'Bimport' is not defined

In [5]: from bimport.models import *

In [6]: Bimport
Out[6]: <class 'bimport.models.Bimport'>

In [7]: pinfo Bimport
Type:		ModelBase
Base Class:	<class 'django.db.models.base.ModelBase'>
String Form:	<class 'bimport.models.Bimport'>
Namespace:	Interactive
File:		/Users/callen/code/socceramerica/bimport/models.py
Docstring:
    Bimport(id, content_type_id, source_id, import_date, datafile, error, status)

Constructor information:
Definition:	Bimport(self, *args, **kwargs)


In [8]: Bimport.objects.get(id = 20)
Out[8]: <Bimport: Bimport object>

In [9]: Bimport.objects.get(id = 20).status
Out[9]: u'P'

In [10]: Bimport.objects.get(id = 20).newsletters
Out[10]: <django.db.models.fields.related.ManyRelatedManager object at 0x16ae8d0>

In [11]: Bimport.objects.get(id = 20).newsletters.all()
---------------------------------------------------------------------------
FieldError                                Traceback (most recent call last)

/Users/callen/code/socceramerica/<ipython console> in <module>()

/Library/Python/2.5/site-packages/django/db/models/manager.pyc in all(self)
    103 
    104     def all(self):
--> 105         return self.get_query_set()
    106 
    107     def count(self):

/Library/Python/2.5/site-packages/django/db/models/fields/related.pyc in get_query_set(self)
    422 
    423         def get_query_set(self):
--> 424             return superclass.get_query_set(self)._next_is_sticky().filter(**(self.core_filters))
    425 
    426         # If the ManyToMany relation has an intermediary model,

/Library/Python/2.5/site-packages/django/db/models/query.pyc in filter(self, *args, **kwargs)
    496         set.
    497         """
--> 498         return self._filter_or_exclude(False, *args, **kwargs)
    499 
    500     def exclude(self, *args, **kwargs):

/Library/Python/2.5/site-packages/django/db/models/query.pyc in _filter_or_exclude(self, negate, *args, **kwargs)
    514             clone.query.add_q(~Q(*args, **kwargs))
    515         else:
--> 516             clone.query.add_q(Q(*args, **kwargs))
    517         return clone
    518 

/Library/Python/2.5/site-packages/django/db/models/sql/query.pyc in add_q(self, q_object, used_aliases)
   1673                 else:
   1674                     self.add_filter(child, connector, q_object.negated,
-> 1675                             can_reuse=used_aliases)
   1676                 if connector == OR:
   1677                     # Aliases that were newly added or not used at all need to

/Library/Python/2.5/site-packages/django/db/models/sql/query.pyc in add_filter(self, filter_expr, connector, negate, trim, can_reuse, process_extras)
   1567             field, target, opts, join_list, last, extra_filters = self.setup_joins(
   1568                     parts, opts, alias, True, allow_many, can_reuse=can_reuse,
-> 1569                     negate=negate, process_extras=process_extras)
   1570         except MultiJoin, e:
   1571             self.split_exclude(filter_expr, LOOKUP_SEP.join(parts[:e.level]),

/Library/Python/2.5/site-packages/django/db/models/sql/query.pyc in setup_joins(self, names, opts, alias, dupe_multis, allow_many, allow_explicit_fk, can_reuse, negate, process_extras)
   1735                     names = opts.get_all_field_names() + self.aggregate_select.keys()
   1736                     raise FieldError("Cannot resolve keyword %r into field. "
-> 1737                             "Choices are: %s" % (name, ", ".join(names)))
   1738 
   1739             if not allow_many and (m2m or not direct):

FieldError: Cannot resolve keyword 'bimport' into field. Choices are: archive_column, archive_order, article, construct, day, edition, frequency, frequency_text, from_email, html_template, id, is_public, list_name, name, newsletterarticle, newsletterarticles, slug, subject, subject_type, subscription, text_template, time

shell_plus from django-command-extensions

{{{import os
from django.core.management.base import NoArgsCommand
from optparse import make_option

class Command(NoArgsCommand):

option_list = NoArgsCommand.option_list + (

make_option('--plain', action='store_true', dest='plain',

help='Tells Django to use plain Python, not IPython.'),

make_option('--no-pythonrc', action='store_true', dest='no_pythonrc',

help='Tells Django to use plain Python, not IPython.'),

)
help = "Like the 'shell' command but autoloads the models of all installed Django apps."

requires_model_validation = True

def handle_noargs(self, options):

# XXX: (Temporary) workaround for ticket #1796: force early loading of all
# models from installed apps. (this is fixed by now, but leaving it here
# for people using 0.96 or older trunk (pre [5919]) versions.
from django.db.models.loading import get_models, get_apps
loaded_models = get_models()

use_plain = options.get('plain', False)
use_pythonrc = not options.get('no_pythonrc', True)

# Set up a dictionary to serve as the environment for the shell, so
# that tab completion works on objects that are imported at runtime.
# See ticket 5082.
from django.conf import settings
imported_objects = {'settings': settings}
for app_mod in get_apps():

app_models = get_models(app_mod)
if not app_models:

continue

model_labels = ", ".join([model.name for model in app_models])
print self.style.SQL_COLTYPE("From '%s' autoload: %s" % (app_mod.name.split('.')[-2], model_labels))
for model in app_models:

try:

imported_objects[model.name] = getattr(import(app_mod.name, {}, {}, model.name), model.name)

except AttributeError, e:

print self.style.ERROR_OUTPUT("Failed to import '%s' from '%s' reason: %s" % (model.name, app_mod.name.split('.')[-2], str(e)))
continue

try:

if use_plain:

# Don't bother loading IPython, because the user wants plain Python.
raise ImportError

import IPython
# Explicitly pass an empty list as arguments, because otherwise IPython
# would use sys.argv from this script.
shell = IPython.Shell.IPShell(argv=[], user_ns=imported_objects)
shell.mainloop()

except ImportError:

# Using normal Python shell
import code
try: # Try activating rlcompleter, because it's handy.

import readline

except ImportError:

pass

else:

# We don't have to wrap the following import in a 'try', because
# we already know 'readline' was imported successfully.
import rlcompleter
readline.set_completer(rlcompleter.Completer(imported_objects).complete)
readline.parse_and_bind("tab:complete")

# We want to honor both $PYTHONSTARTUP and .pythonrc.py, so follow system
# conventions and get $PYTHONSTARTUP first then import user.
if use_pythonrc:

pythonrc = os.environ.get("PYTHONSTARTUP")
if pythonrc and os.path.isfile(pythonrc):

try:

execfile(pythonrc)

except NameError:

pass

# This will import .pythonrc.py as a side-effect
import user

code.interact(local=imported_objects)

}}}

[callen]

Despite not getting auto-imported by shell_plus, bimport is indeed part of my "installed apps".

INSTALLED_APPS = (
    'django.contrib.sites',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.markup',
    'django.contrib.sessions',
    'django.contrib.admin', 
    'django.contrib.comments',
    'django.contrib.flatpages',
    'django.contrib.redirects',
    'django.contrib.humanize',
    'django.contrib.sitemaps',
    'django_filters',
    'pagination',
    'venus',
    'mailer',
    'base',
    'account',
    'bimport',
    'djangosphinx',
    'utils',
    'middleware',
    'merx',
    'migration',
    'johnny',
)

[callen]

Local_settings installed_apps was overriding the one in settings.py. Didn't have 'bimport' in the tuple.

Fixing that seemed to deal with the newsletter M2M issue. The kind of error this produced is odd and not at all expected.

I'd consider the strange error output it produced a matter worth looking into unless there's a known explanation.

[Russell Keith-Magee]

I'm going to call this a strange presentation of #10405.

If I'm understanding the extremely verbose reproduction instructions, the root of this problem is a model that is referenced in a m2m that belongs to an app that isn't (due to some configuration problem) included in INSTALLED_APPS. Using shell_plus seems to change the way the problem manifests, but using similar conditions and a normal shell, I get "'str' object has no attribute '_default_manager'" errors, which is the presentation under #10405.

I would also note that this particular error case is caught by manage.py validate. You can't use runserver, syncdb, or most of the other management commands without triggering a validation error if you have an m2m to a model in an app that isn't in INSTALLED_APPS.

If I'm misunderstood the root cause here, a *terse* set of reproduction instructions -- preferably that doesn't involve shell_plus -- would be helpful.