Opened 13 years ago

Last modified 10 years ago

#15559 new Bug

Distinct queries will cause errors with some custom model fields

Reported by: urandom Owned by: nobody
Component: Database layer (models, ORM) Version: dev
Severity: Normal Keywords:
Cc: Michael Radziej Triage Stage: Accepted
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

Starting with 1.3 rc1, some queries are marked as distinct by the system.

If a model now uses a custom field, that relies on a special DB column (like Postgresql's 'point' column),
the database will immediately throw an error, since such a column cannot be compared, and thus unique values cannot be found.

It would probably be a good idea to allow fields to specify whether they cannot be distinct:
custom = MyCustomField(distinctless=True)

which would turn this ordinary distinct query:
SELECT DISTINCT id, number, custom FROM foo;

into:
SELECT DISTINCT ON(id, number) FROM FOO;

Discussion link:
https://groups.google.com/forum/#!topic/django-developers/iO8z3iLpgg4

Change History (15)

comment:1 by Michael Radziej, 13 years ago

Cc: Michael Radziej added

comment:2 by Karen Tracey, 13 years ago

Component: UncategorizedDatabase layer (models, ORM)
Keywords: regression blocker added
milestone: 1.3
Triage Stage: UnreviewedAccepted
Version: 1.3-beta1.3-rc1

Not sure we want to be adding a new option between now and 1.3 final, but seems like we need to do something (minimum document the change as a backwards-incompatibility?).

comment:3 by Luke Plant, 13 years ago

We could also revert the fix in [15607], and either:

  • leave #11707 open
  • Fix #11707 by doing duplicate elimination in Python

comment:4 by Erin Kelly, 13 years ago

I discovered another problem with the distinct queries from [15607]. Using Oracle, it breaks foreign keys to models that have TextFields. This is because Oracle won't let you do SELECT DISTINCT with a select list that includes a LOB column.

+1 for reverting.

comment:5 by Michael Radziej, 13 years ago

ikelly, does Oracle support SELECT DISTINCT ON (id) if there's a LOB column in the SELECT list?

in reply to:  5 comment:6 by Erin Kelly, 13 years ago

Replying to mir:

ikelly, does Oracle support SELECT DISTINCT ON (id) if there's a LOB column in the SELECT list?

No. The analytic query that emulates DISTINCT ON works using a regular DISTINCT query and selecting for each column the first value in that column for the corresponding DISTINCT ON partition in place of the actual value for the row. The result is that rows that are equivalent over the DISTINCT ON partition become equivalent over the entire row and so are filtered out by the DISTINCT keyword. That means you're still effectively doing a SELECT DISTINCT with a LOB column.

comment:7 by Luke Plant, 13 years ago

Resolution: fixed
Status: newclosed

In [15791]:

Fixed #15559 - distinct queries introduced by [15607] cause errors with some custom model fields

This patch just reverts [15607] until a more satisfying solution can be
found.

Refs #11707

comment:8 by Luke Plant, 13 years ago

In [15792]:

[1.2.X] Fixed #15559 - distinct queries introduced by [15607] cause errors with some custom model fields

This patch just reverts [15607] until a more satisfying solution can be
found.

Refs #11707

Backport of [15791] from trunk.

comment:9 by Luke Plant, 13 years ago

As you can see, I decided to revert [15607], because of the support for reverting listed above and on the mailing list, and because of the closeness of the 1.3 release.

comment:10 by Luke Plant, 13 years ago

Severity: Normal
Type: Uncategorized

There are other places where we are using distinct (see [15526]), so it is possible that this bug is only fixed in the one instance where the reporter experienced a regression. A more satisfying general solution is probably still needed, but a new bug should be opened for that.

in reply to:  10 comment:11 by Carl Meyer, 13 years ago

Keywords: regression blocker removed
milestone: 1.3
Resolution: fixed
Status: closedreopened
Type: UncategorizedBug
Version: 1.3-rc1SVN

Replying to lukeplant:

There are other places where we are using distinct (see [15526]), so it is possible that this bug is only fixed in the one instance where the reporter experienced a regression. A more satisfying general solution is probably still needed, but a new bug should be opened for that.

I don't see any need to open a new bug - it would just be a copy-paste of this one. The description for this bug is still fully applicable (and was never fixed, just worked around). Reopening and removing 1.3-blocker keywords.

comment:12 by Aymeric Augustin, 12 years ago

UI/UX: unset

Change UI/UX from NULL to False.

comment:13 by Aymeric Augustin, 12 years ago

Easy pickings: unset

Change Easy pickings from NULL to False.

comment:14 by Aymeric Augustin, 11 years ago

Status: reopenednew

in reply to:  3 comment:15 by Chris Wesseling, 10 years ago

Replying to lukeplant:

We could also revert the fix in [15607], and either:

  • leave #11707 open
  • Fix #11707 by doing duplicate elimination in Python

I'm revisiting #11707 on a sprint. I implemented the duplicate elimination 2 years ago with ojii, but it never got RFC'ed.

Since then I thought: DISTINCT logically equivalent with a GROUP BY on all fields. Not all databases support DISTINCT ON, but a GROUP BY on just the pk should do the trick (at least in #11707).

Note: See TracTickets for help on using tickets.
Back to Top