Opened 7 years ago

Closed 6 years ago

#13204 closed Bug (duplicate)

`values()` returns empty list after `extra(order_by)`

Reported by: Andy Durdin Owned by: nobody
Component: Database layer (models, ORM) Version: 1.1
Severity: Normal Keywords:
Cc: me@…, Gabriel Hurley Triage Stage: Accepted
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: UI/UX:


Given any model with an integer 'id' column, if you perform an extra() query that creates a new column and orders by it, .values() fails even if selecting values from a different column:

>>> qs = MyModel.objects.extra({'new_number': '-id'}, order_by=['-new_number']).values('id')
>>> list(qs)
>>> qs.count()

If you try to display the query, a FieldError is raised:

>>> print qs.query
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File ".../django/db/models/sql/", line 113, in __str__
    sql, params = self.as_sql()
  File ".../django/db/models/sql/", line 397, in as_sql
    ordering, ordering_group_by = self.get_ordering()
  File ".../django/db/models/sql/", line 962, in get_ordering
    self.model._meta, default_order=asc):
  File ".../django/db/models/sql/", line 991, in find_ordering_name
    opts, alias, False)
  File ".../django/db/models/sql/", line 1737, in setup_joins
    "Choices are: %s" % (name, ", ".join(names)))
FieldError: Cannot resolve keyword 'new_number' into field. Choices are: id, name

It is arguable whether such a query should be valid or not, but the behaviour here is inconsistent: an empty list when evaluating the queryset, getting the number of rows on count(), and a FieldError on most other operations.

This behaviour was noticed under Django 1.1

Attachments (1)

repl13204.tar.bz2 (4.7 KB) - added by Andy Durdin 7 years ago.
Minimal project that shows the error.

Download all attachments as: .zip

Change History (12)

comment:1 Changed 7 years ago by Andy Durdin

Cc: me@… added
Needs documentation: unset
Needs tests: unset
Patch needs improvement: unset

comment:2 Changed 7 years ago by Russell Keith-Magee

Resolution: worksforme
Status: newclosed

I can't reproduce this on 1.1, 1.1.X or trunk. I get the FieldError consistently, which to my reading is the right result (since you can't order by a created column that you aren't actually selecting).

comment:3 Changed 7 years ago by Andy Durdin

Resolution: worksforme
Status: closedreopened

I can still reproduce this on a fresh project, on both django 1.1.1 and trunk (revision 12855). I was running with python 2.6.1, with both mysql 5.0.77 and sqlite 3.6.12, all running on Mac OS X 10.6.2. I will attach the minimal project to this ticket for reference.

You are right, the expected result for each of the three statements is a FieldError.

Changed 7 years ago by Andy Durdin

Attachment: repl13204.tar.bz2 added

Minimal project that shows the error.

comment:4 Changed 7 years ago by Gabriel Hurley

I wouldn't be surprised if this is another instance of the Python 2.6.1 bug I encountered the other day on #13171 and #13172... I never did pinpoint it, but it had something to do with the exception not bubbling up through the len method of QuerySet in django.db.models.query.

I did a stacktrace from where the error was supposed to be raised and found that it ran through this line of code, which is where the Exception vanishes:

When the queryset is converted to a list there, it calls len, the exception disappears, and an empty list is returned instead of the FieldError (or whichever error you're looking for). Raising the error manually before that line will return the error, raising it after that line of code it is simply ignored. I spent over an hour hacking at the Django code to try and fix it to no avail. The bug was pretty clearly in Python itself.

Upgrading Python to a more recent version is my solution. Though it seems this problem is gonna keep popping up since 2.6.1 is the default Snow Leopard install.

comment:5 Changed 7 years ago by Gabriel Hurley

Cc: Gabriel Hurley added

comment:6 Changed 7 years ago by Karen Tracey

The Python bug that swallows any exceptions raised by __len__ is here:

We have run into trouble with it before, see #7786. Malcolm notes in one of the comments that he tried at one point to avoid doing anything in __len__ that could raise an exception but that had terrible performance impacts. Given that, I'm not sure there is much we can do here.

comment:7 Changed 7 years ago by Gabriel Hurley

Ah, very interesting bug report there. Thanks Karen! I like the comment "This function never fails.
Accordingly, it will mask exceptions raised in either method." That's just asking for trouble. Ha!

Perhaps, since 2.6.1 is installed on Snow Leopard by default, a note somewhere in the docs would be helpful... like a small item in:

Pointing out the problem might save a few people. While it seems obvious that outdated bugfix releases of Python are going to have bugs, it seems like this particular one might be worth noting since it may affect a fair number of people.

Just a thought.

comment:8 Changed 7 years ago by Russell Keith-Magee

Triage Stage: UnreviewedAccepted

If this were just an issue with Python 2.6.1, documenting it would probably be the appropriate solution. However, given that 2.6.1 is the default install on Snow Leopard, this is going to keep biting us.

I'm afraid I don't have any helpful suggestions, though.

comment:9 Changed 6 years ago by Luke Plant

Type: Bug

comment:10 Changed 6 years ago by Luke Plant

Severity: Normal

comment:11 Changed 6 years ago by Jacob

Resolution: duplicate
Status: reopenedclosed

I've created #15867 to track the documentation issue. I agree with Malcolm's note on #7786 that avoiding exceptions in __len__ is basically impossible, so I'm marking this as a dup.

Note: See TracTickets for help on using tickets.
Back to Top