Opened 7 years ago

Closed 6 years ago

#3141 closed defect (fixed)

can't use count() on querysets when extra() has been used with params.

Reported by: mrmachine <real dot human at mrmachine dot net> Owned by: deepak
Component: Database layer (models, ORM) Version: master
Severity: normal Keywords: queryset extra params count, qs-rf-fixed
Cc: simon@…, real.human@…, heckj@… Triage Stage: Accepted
Has patch: yes Needs documentation: no
Needs tests: yes Patch needs improvement: no
Easy pickings: UI/UX:


there seems to be a bug in django/db/models/ when using extra() with params on a queryset. the queryset is generated correctly, i can access it, iterate through it, but i can't count() it.

>>> from django.contrib.auth.models import User
>>> u = User.objects.filter(pk=1).extra(select={'foo': '%s'}, params=['1'])
>>> u[0].foo
>>> u.count()
Traceback (most recent call last):
  File "<console>", line 1, in ?
  File "/Volumes/Singularity/Projects/django/django/db/models/", line 202, in count
    cursor.execute("SELECT COUNT(*)" + sql, params)
  File "/Volumes/Singularity/Projects/django/django/db/backends/", line 19, in execute
TypeError: not all arguments converted during string formatting
>>> u = User.objects.filter(pk=1).extra(select={'foo': '1'})
>>> u[0].foo
>>> u.count()

Attachments (1)

extra_param_fix.diff (618 bytes) - added by deepak <deep.thukral@…> 7 years ago.

Download all attachments as: .zip

Change History (9)

comment:1 Changed 7 years ago by heckj@…

  • Cc heckj@… added

I just ran into a similiar situation this evening with the queryset not putting the arguments in the correct ordering, specifically when tacking on two "extra" sets to an otherwise kind of plain jane queryset. I'm trying to work it down to an easily reproducible case or example. In my case, it also appears to be related to using .count() at the tail end of the queryset chain.

If I pull back the queryset, force it into a list, then I can count it - but that sort of defeats the purpose of being able to use the count.

Ah - I narrowed it down to when it's using an extra(select=...) statement. In django/db/models/ around line 202, it's invoking:
cursor.execute("SELECT COUNT(*)" + sql, params)

The additional "select" param is added to the list, but the additional "select" component isn't, resulting in the "not all arguments converted" error.

Not sure how to nessecarily resolve this though.

comment:2 Changed 7 years ago by Simon G. <dev@…>

  • Triage Stage changed from Unreviewed to Accepted

comment:3 Changed 7 years ago by mtredinnick

  • Keywords count, qs-rf-fixed added; count removed

Changed 7 years ago by deepak <deep.thukral@…>

comment:4 Changed 7 years ago by deepak <deep.thukral@…>

  • Has patch set
  • Needs tests set
  • Owner changed from nobody to deepak
  • Version set to SVN

Main problem is that while logging sql, we are not handling extras in sql queries. I fixed this issue, but needs test.

comment:5 Changed 6 years ago by amit.ramon@…

The problem still exists with the latest svn version. It seems to a problem with using select with parameters in extra (extra where with parameters is working fine).

comment:6 Changed 6 years ago by mtredinnick

(In [7340]) queryset-refactor: Fixed up extra(select=...) calls with parameters so that the
parameters are substituted in correctly in all cases. This introduces an extra
argument to extra() for this purpose; no alternative there.

Also fixed values() to work if you don't specify *all* the extra select aliases
in the values() call.

Refs #3141.

comment:7 Changed 6 years ago by Simon Litchfield <simon@…>

  • Cc simon@… added

comment:8 Changed 6 years ago by mtredinnick

  • Resolution set to fixed
  • Status changed from new to closed

(In [7477]) Merged the queryset-refactor branch into trunk.

This is a big internal change, but mostly backwards compatible with existing
code. Also adds a couple of new features.

Fixed #245, #1050, #1656, #1801, #2076, #2091, #2150, #2253, #2306, #2400, #2430, #2482, #2496, #2676, #2737, #2874, #2902, #2939, #3037, #3141, #3288, #3440, #3592, #3739, #4088, #4260, #4289, #4306, #4358, #4464, #4510, #4858, #5012, #5020, #5261, #5295, #5321, #5324, #5325, #5555, #5707, #5796, #5817, #5987, #6018, #6074, #6088, #6154, #6177, #6180, #6203, #6658

Add Comment

Modify Ticket

Change Properties
<Author field>
as closed
as The resolution will be set. Next status will be 'closed'
The resolution will be deleted. Next status will be 'new'

E-mail address and user name can be saved in the Preferences.

Note: See TracTickets for help on using tickets.