Opened 6 years ago

Closed 5 years ago

#13204 closed Bug (duplicate)

`values()` returns empty list after `extra(order_by)`

Reported by: adurdin Owned by: nobody
Component: Database layer (models, ORM) Version: 1.1
Severity: Normal Keywords:
Cc: me@…, gabrielhurley Triage Stage: Accepted
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: UI/UX:


Given any model with an integer 'id' column, if you perform an extra() query that creates a new column and orders by it, .values() fails even if selecting values from a different column:

>>> qs = MyModel.objects.extra({'new_number': '-id'}, order_by=['-new_number']).values('id')
>>> list(qs)
>>> qs.count()

If you try to display the query, a FieldError is raised:

>>> print qs.query
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File ".../django/db/models/sql/", line 113, in __str__
    sql, params = self.as_sql()
  File ".../django/db/models/sql/", line 397, in as_sql
    ordering, ordering_group_by = self.get_ordering()
  File ".../django/db/models/sql/", line 962, in get_ordering
    self.model._meta, default_order=asc):
  File ".../django/db/models/sql/", line 991, in find_ordering_name
    opts, alias, False)
  File ".../django/db/models/sql/", line 1737, in setup_joins
    "Choices are: %s" % (name, ", ".join(names)))
FieldError: Cannot resolve keyword 'new_number' into field. Choices are: id, name

It is arguable whether such a query should be valid or not, but the behaviour here is inconsistent: an empty list when evaluating the queryset, getting the number of rows on count(), and a FieldError on most other operations.

This behaviour was noticed under Django 1.1

Attachments (1)

repl13204.tar.bz2 (4.7 KB) - added by adurdin 6 years ago.
Minimal project that shows the error.

Download all attachments as: .zip

Change History (12)

comment:1 Changed 6 years ago by adurdin

  • Cc me@… added
  • Needs documentation unset
  • Needs tests unset
  • Patch needs improvement unset

comment:2 Changed 6 years ago by russellm

  • Resolution set to worksforme
  • Status changed from new to closed

I can't reproduce this on 1.1, 1.1.X or trunk. I get the FieldError consistently, which to my reading is the right result (since you can't order by a created column that you aren't actually selecting).

comment:3 Changed 6 years ago by adurdin

  • Resolution worksforme deleted
  • Status changed from closed to reopened

I can still reproduce this on a fresh project, on both django 1.1.1 and trunk (revision 12855). I was running with python 2.6.1, with both mysql 5.0.77 and sqlite 3.6.12, all running on Mac OS X 10.6.2. I will attach the minimal project to this ticket for reference.

You are right, the expected result for each of the three statements is a FieldError.

Changed 6 years ago by adurdin

Minimal project that shows the error.

comment:4 Changed 6 years ago by gabrielhurley

I wouldn't be surprised if this is another instance of the Python 2.6.1 bug I encountered the other day on #13171 and #13172... I never did pinpoint it, but it had something to do with the exception not bubbling up through the len method of QuerySet in django.db.models.query.

I did a stacktrace from where the error was supposed to be raised and found that it ran through this line of code, which is where the Exception vanishes:

When the queryset is converted to a list there, it calls len, the exception disappears, and an empty list is returned instead of the FieldError (or whichever error you're looking for). Raising the error manually before that line will return the error, raising it after that line of code it is simply ignored. I spent over an hour hacking at the Django code to try and fix it to no avail. The bug was pretty clearly in Python itself.

Upgrading Python to a more recent version is my solution. Though it seems this problem is gonna keep popping up since 2.6.1 is the default Snow Leopard install.

comment:5 Changed 6 years ago by gabrielhurley

  • Cc gabrielhurley added

comment:6 Changed 6 years ago by kmtracey

The Python bug that swallows any exceptions raised by __len__ is here:

We have run into trouble with it before, see #7786. Malcolm notes in one of the comments that he tried at one point to avoid doing anything in __len__ that could raise an exception but that had terrible performance impacts. Given that, I'm not sure there is much we can do here.

comment:7 Changed 6 years ago by gabrielhurley

Ah, very interesting bug report there. Thanks Karen! I like the comment "This function never fails.
Accordingly, it will mask exceptions raised in either method." That's just asking for trouble. Ha!

Perhaps, since 2.6.1 is installed on Snow Leopard by default, a note somewhere in the docs would be helpful... like a small item in:

Pointing out the problem might save a few people. While it seems obvious that outdated bugfix releases of Python are going to have bugs, it seems like this particular one might be worth noting since it may affect a fair number of people.

Just a thought.

comment:8 Changed 6 years ago by russellm

  • Triage Stage changed from Unreviewed to Accepted

If this were just an issue with Python 2.6.1, documenting it would probably be the appropriate solution. However, given that 2.6.1 is the default install on Snow Leopard, this is going to keep biting us.

I'm afraid I don't have any helpful suggestions, though.

comment:9 Changed 5 years ago by lukeplant

  • Type set to Bug

comment:10 Changed 5 years ago by lukeplant

  • Severity set to Normal

comment:11 Changed 5 years ago by jacob

  • Resolution set to duplicate
  • Status changed from reopened to closed

I've created #15867 to track the documentation issue. I agree with Malcolm's note on #7786 that avoiding exceptions in __len__ is basically impossible, so I'm marking this as a dup.

Note: See TracTickets for help on using tickets.
Back to Top