`values()` returns empty list after `extra(order_by)`

[Andy Durdin]

Given any model with an integer 'id' column, if you perform an extra() query that creates a new column and orders by it, .values() fails even if selecting values from a different column:

>>> qs = MyModel.objects.extra({'new_number': '-id'}, order_by=['-new_number']).values('id')
>>> list(qs)
[]
>>> qs.count()
66670

If you try to display the query, a FieldError is raised:

>>> print qs.query
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File ".../django/db/models/sql/query.py", line 113, in __str__
    sql, params = self.as_sql()
  File ".../django/db/models/sql/query.py", line 397, in as_sql
    ordering, ordering_group_by = self.get_ordering()
  File ".../django/db/models/sql/query.py", line 962, in get_ordering
    self.model._meta, default_order=asc):
  File ".../django/db/models/sql/query.py", line 991, in find_ordering_name
    opts, alias, False)
  File ".../django/db/models/sql/query.py", line 1737, in setup_joins
    "Choices are: %s" % (name, ", ".join(names)))
FieldError: Cannot resolve keyword 'new_number' into field. Choices are: id, name

It is arguable whether such a query should be valid or not, but the behaviour here is inconsistent: an empty list when evaluating the queryset, getting the number of rows on count(), and a FieldError on most other operations.

This behaviour was noticed under Django 1.1

[Russell Keith-Magee]

I can't reproduce this on 1.1, 1.1.X or trunk. I get the FieldError consistently, which to my reading is the right result (since you can't order by a created column that you aren't actually selecting).

[Andy Durdin]

I can still reproduce this on a fresh project, on both django 1.1.1 and trunk (revision 12855). I was running with python 2.6.1, with both mysql 5.0.77 and sqlite 3.6.12, all running on Mac OS X 10.6.2. I will attach the minimal project to this ticket for reference.

You are right, the expected result for each of the three statements is a FieldError.

[Andy Durdin]

Minimal project that shows the error.

[Gabriel Hurley]

I wouldn't be surprised if this is another instance of the Python 2.6.1 bug I encountered the other day on #13171 and #13172... I never did pinpoint it, but it had something to do with the exception not bubbling up through the len method of QuerySet in django.db.models.query.

I did a stacktrace from where the error was supposed to be raised and found that it ran through this line of code, which is where the Exception vanishes:

​http://code.djangoproject.com/browser/django/trunk/django/db/models/query.py#L81

When the queryset is converted to a list there, it calls len, the exception disappears, and an empty list is returned instead of the FieldError (or whichever error you're looking for). Raising the error manually before that line will return the error, raising it after that line of code it is simply ignored. I spent over an hour hacking at the Django code to try and fix it to no avail. The bug was pretty clearly in Python itself.

Upgrading Python to a more recent version is my solution. Though it seems this problem is gonna keep popping up since 2.6.1 is the default Snow Leopard install.

[Karen Tracey]

The Python bug that swallows any exceptions raised by __len__ is here: ​http://bugs.python.org/issue1242657

We have run into trouble with it before, see #7786. Malcolm notes in one of the comments that he tried at one point to avoid doing anything in __len__ that could raise an exception but that had terrible performance impacts. Given that, I'm not sure there is much we can do here.

[Gabriel Hurley]

Ah, very interesting bug report there. Thanks Karen! I like the comment "This function never fails.
Accordingly, it will mask exceptions raised in either method." That's just asking for trouble. Ha!

Perhaps, since 2.6.1 is installed on Snow Leopard by default, a note somewhere in the docs would be helpful... like a small item in:

​http://docs.djangoproject.com/en/1.1/intro/install/#install-python

Pointing out the problem might save a few people. While it seems obvious that outdated bugfix releases of Python are going to have bugs, it seems like this particular one might be worth noting since it may affect a fair number of people.

Just a thought.

[Russell Keith-Magee]

If this were just an issue with Python 2.6.1, documenting it would probably be the appropriate solution. However, given that 2.6.1 is the default install on Snow Leopard, this is going to keep biting us.

I'm afraid I don't have any helpful suggestions, though.

[Jacob]

I've created #15867 to track the documentation issue. I agree with Malcolm's note on #7786 that avoiding exceptions in __len__ is basically impossible, so I'm marking this as a dup.