Opened 14 years ago

Closed 13 years ago

#13204 closed Bug (duplicate)

`values()` returns empty list after `extra(order_by)`

Reported by: Andy Durdin Owned by: nobody
Component: Database layer (models, ORM) Version: 1.1
Severity: Normal Keywords:
Cc: me@…, Gabriel Hurley Triage Stage: Accepted
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

Given any model with an integer 'id' column, if you perform an extra() query that creates a new column and orders by it, .values() fails even if selecting values from a different column:

>>> qs = MyModel.objects.extra({'new_number': '-id'}, order_by=['-new_number']).values('id')
>>> list(qs)
[]
>>> qs.count()
66670

If you try to display the query, a FieldError is raised:

>>> print qs.query
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File ".../django/db/models/sql/query.py", line 113, in __str__
    sql, params = self.as_sql()
  File ".../django/db/models/sql/query.py", line 397, in as_sql
    ordering, ordering_group_by = self.get_ordering()
  File ".../django/db/models/sql/query.py", line 962, in get_ordering
    self.model._meta, default_order=asc):
  File ".../django/db/models/sql/query.py", line 991, in find_ordering_name
    opts, alias, False)
  File ".../django/db/models/sql/query.py", line 1737, in setup_joins
    "Choices are: %s" % (name, ", ".join(names)))
FieldError: Cannot resolve keyword 'new_number' into field. Choices are: id, name

It is arguable whether such a query should be valid or not, but the behaviour here is inconsistent: an empty list when evaluating the queryset, getting the number of rows on count(), and a FieldError on most other operations.

This behaviour was noticed under Django 1.1

Attachments (1)

repl13204.tar.bz2 (4.7 KB ) - added by Andy Durdin 14 years ago.
Minimal project that shows the error.

Download all attachments as: .zip

Change History (12)

comment:1 by Andy Durdin, 14 years ago

Cc: me@… added

comment:2 by Russell Keith-Magee, 14 years ago

Resolution: worksforme
Status: newclosed

I can't reproduce this on 1.1, 1.1.X or trunk. I get the FieldError consistently, which to my reading is the right result (since you can't order by a created column that you aren't actually selecting).

comment:3 by Andy Durdin, 14 years ago

Resolution: worksforme
Status: closedreopened

I can still reproduce this on a fresh project, on both django 1.1.1 and trunk (revision 12855). I was running with python 2.6.1, with both mysql 5.0.77 and sqlite 3.6.12, all running on Mac OS X 10.6.2. I will attach the minimal project to this ticket for reference.

You are right, the expected result for each of the three statements is a FieldError.

by Andy Durdin, 14 years ago

Attachment: repl13204.tar.bz2 added

Minimal project that shows the error.

comment:4 by Gabriel Hurley, 14 years ago

I wouldn't be surprised if this is another instance of the Python 2.6.1 bug I encountered the other day on #13171 and #13172... I never did pinpoint it, but it had something to do with the exception not bubbling up through the len method of QuerySet in django.db.models.query.

I did a stacktrace from where the error was supposed to be raised and found that it ran through this line of code, which is where the Exception vanishes:

http://code.djangoproject.com/browser/django/trunk/django/db/models/query.py#L81

When the queryset is converted to a list there, it calls len, the exception disappears, and an empty list is returned instead of the FieldError (or whichever error you're looking for). Raising the error manually before that line will return the error, raising it after that line of code it is simply ignored. I spent over an hour hacking at the Django code to try and fix it to no avail. The bug was pretty clearly in Python itself.

Upgrading Python to a more recent version is my solution. Though it seems this problem is gonna keep popping up since 2.6.1 is the default Snow Leopard install.

comment:5 by Gabriel Hurley, 14 years ago

Cc: Gabriel Hurley added

comment:6 by Karen Tracey, 14 years ago

The Python bug that swallows any exceptions raised by __len__ is here: http://bugs.python.org/issue1242657

We have run into trouble with it before, see #7786. Malcolm notes in one of the comments that he tried at one point to avoid doing anything in __len__ that could raise an exception but that had terrible performance impacts. Given that, I'm not sure there is much we can do here.

comment:7 by Gabriel Hurley, 14 years ago

Ah, very interesting bug report there. Thanks Karen! I like the comment "This function never fails.
Accordingly, it will mask exceptions raised in either method." That's just asking for trouble. Ha!

Perhaps, since 2.6.1 is installed on Snow Leopard by default, a note somewhere in the docs would be helpful... like a small item in:

http://docs.djangoproject.com/en/1.1/intro/install/#install-python

Pointing out the problem might save a few people. While it seems obvious that outdated bugfix releases of Python are going to have bugs, it seems like this particular one might be worth noting since it may affect a fair number of people.

Just a thought.

comment:8 by Russell Keith-Magee, 14 years ago

Triage Stage: UnreviewedAccepted

If this were just an issue with Python 2.6.1, documenting it would probably be the appropriate solution. However, given that 2.6.1 is the default install on Snow Leopard, this is going to keep biting us.

I'm afraid I don't have any helpful suggestions, though.

comment:9 by Luke Plant, 13 years ago

Type: Bug

comment:10 by Luke Plant, 13 years ago

Severity: Normal

comment:11 by Jacob, 13 years ago

Resolution: duplicate
Status: reopenedclosed

I've created #15867 to track the documentation issue. I agree with Malcolm's note on #7786 that avoiding exceptions in __len__ is basically impossible, so I'm marking this as a dup.

Note: See TracTickets for help on using tickets.
Back to Top