Opened 13 years ago

Closed 9 years ago

Last modified 7 years ago

#18247 closed Bug (fixed)

Filtering on aggregate annotations with a Decimal value doesn't work

Reported by: elmopl@… Owned by: Michael Tänzer
Component: Database layer (models, ORM) Version: dev
Severity: Normal Keywords: decimal filter annotate having
Cc: zbyte64, dylan@…, Michael Tänzer Triage Stage: Ready for checkin
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

When I run

$ python manage.py version
1.4
$ python manage.py shell
[...]
>>> for r in StoreBill.objects.annotate( amount = Sum('parts__amount') ).filter( date = '2012-03-28' ): r.amount
... 
Decimal('1.05')
>>> for r in StoreBill.objects.annotate( amount = Sum('parts__amount') ).filter( date = '2012-03-28', amount = Decimal("1.05") ): a = r.amount
... 
>>> print connection.queries[-1]
{'time': '0.002', 'sql': u'SELECT "spendings_storebill"."id", "spendings_storebill"."name", "spendings_storebill"."date", SUM("spendings_cost"."amount") AS "amount" FROM "spendings_storebill" LEFT OUTER JOIN "spendings_storebillpart" ON ("spendings_storebill"."id" = "spendings_storebillpart"."bill_id") LEFT OUTER JOIN "spendings_cost" ON ("spendings_storebillpart"."cost_ptr_id" = "spendings_cost"."id") WHERE ("spendings_storebill"."date" = 2012-03-28 ) GROUP BY "spendings_storebill"."id", "spendings_storebill"."name", "spendings_storebill"."date" HAVING SUM("spendings_cost"."amount") = 1.05 '}

As you can see from first query there is a result matching second one, yet I get no results for latter. If I run the generated query I get expected result (left only "amount" column):

sqlite> SELECT  SUM("spendings_cost"."amount") AS "amount" FROM "spendings_storebill" LEFT OUTER JOIN "spendings_storebillpart" ON ("spendings_storebill"."id" = "spendings_storebillpart"."bill_id") LEFT OUTER JOIN "spendings_cost" ON ("spendings_storebillpart"."cost_ptr_id" = "spendings_cost"."id") GROUP BY "spendings_storebill"."id", "spendings_storebill"."name", "spendings_storebill"."date" HAVING SUM("spendings_cost"."amount") = 1.05  LIMIT 21;
1.05

Change History (18)

comment:1 by zbyte64, 13 years ago

Cc: zbyte64 added
Type: UncategorizedBug

I believe I have experienced the same issue and it only seems to apply to Django 1.4 with sqlite.

from django.db import models

class Transaction(models.Model):
    tag = models.CharField(max_length=5)
    amount = models.DecimalField(max_digits=19, decimal_places=4)

def test_transaction():
    Transaction.objects.all().delete()
    Transaction.objects.create(amount=5, tag='a')
    Transaction.objects.create(amount=4, tag='a')
    Transaction.objects.create(amount=1, tag='b')
    Transaction.objects.create(amount='1.5', tag='b')
    Transaction.objects.create(amount=1, tag='c')
    
    qs = Transaction.objects.values('tag').annotate(sumamount=models.Sum('amount'))
    print qs #displays 3 results
    
    qs = Transaction.objects.values('tag').annotate(sumamount=models.Sum('amount')).filter(sumamount__gt=3)
    print qs #displays 0 results when using sqlite + Django 1.4, displays 1 result when using mysql or Django 1.3.1

comment:2 by anonymous, 13 years ago

Resolution: invalid
Status: newclosed

I was able to replicate that on just sqlite3 (C API), so it is not a bug with Django or driver.
Marking as invalid, as the bug is outside Django itself.

comment:3 by fengb, 12 years ago

Resolution: invalid
Status: closedreopened

Per http://article.gmane.org/gmane.comp.db.sqlite.general/75696 , aggregate functions should not have affinity so typecasting Decimal into a string will never have matching values.

I think changing the typecast of Decimal into float is "good enough". Internally, sqlite3 already represents these values as floats so I don't think there would be too detrimental to make it slightly more explicit. Otherwise, we would need separate casting engines depending on whether aggregation is used.

Last edited 12 years ago by fengb (previous) (diff)

comment:4 by Aymeric Augustin, 12 years ago

Triage Stage: UnreviewedDesign decision needed

comment:5 by Aymeric Augustin, 12 years ago

Status: reopenednew

comment:6 by Aymeric Augustin, 12 years ago

Triage Stage: Design decision neededAccepted

comment:7 by Rigel Di Scala, 11 years ago

#22117 was just closed as a duplicate of this ticket.

comment:8 by dylan@…, 11 years ago

Cc: dylan@… added
Version: 1.4master

in reply to:  8 comment:9 by dylan@…, 11 years ago

Replying to dylan@…:
As noted in #22117 this is still an issue: https://code.djangoproject.com/ticket/22117#comment:1

comment:10 by Shai Berger, 11 years ago

#22128 asked for the limitation to be documented; I closed it as a duplicate of this. If this bug is wontfix'd, we should then add appropriate warnings on the SQLite-specific notes.

comment:11 by Michael Tänzer, 9 years ago

Cc: Michael Tänzer added
Keywords: annotate having added
Owner: changed from nobody to Michael Tänzer
Status: newassigned

comment:12 by Michael Tänzer, 9 years ago

Has patch: set

On sqlite the SUM() of a decimal column doesn't have a NUMERIC type so when comparing it to a string literal (which a Decimal gets converted to in django) it is not compared as NUMERIC as may be expected but as something else (maybe string?).

I added a CAST to NUMERIC type when using a SQL function that is expected to return a Decimal.

All tests pass under SQLite.

comment:13 by Tim Graham, 9 years ago

Summary: filter and Decimal equality doesn't workFiltering on aggregate annotations with a Decimal value doesn't work
Triage Stage: AcceptedReady for checkin

comment:14 by Tim Graham <timograham@…>, 9 years ago

Resolution: fixed
Status: assignedclosed

In 3bbaf84d:

Fixed #18247 -- Added cast to NUMERIC for Decimals on sqlite

On sqlite the SUM() of a decimal column doesn't have a NUMERIC type so
when comparing it to a string literal (which a Decimal gets converted to
in Django) it is not compared as expected.

comment:15 by Tuomas Suutari, 8 years ago

Shouldn't this be backported to the current LTS release (1.8) too? The commit seems to cherry-pick cleanly to stable/1.8.x branch.

comment:16 by Tim Graham <timograham@…>, 8 years ago

In 068d7568:

Refs #18247 -- Fixed SQLite QuerySet filtering on decimal result of Least and Greatest.

comment:17 by Tim Graham <timograham@…>, 8 years ago

In 8484cf4c:

[1.11.x] Refs #18247 -- Fixed SQLite QuerySet filtering on decimal result of Least and Greatest.

Backport of 068d75688f28f9a1530b771ae1d625f41e98cd82 from master

comment:18 by Tim Graham <timograham@…>, 7 years ago

In c3c6c92d:

Refs #18247 -- Fixed filtering on CombinedExpression(output_field=DecimalField()) annotation on SQLite.

Note: See TracTickets for help on using tickets.
Back to Top