Distinct queries will cause errors with some custom model fields

[urandom]

Starting with 1.3 rc1, some queries are marked as distinct by the system.

If a model now uses a custom field, that relies on a special DB column (like Postgresql's 'point' column),
the database will immediately throw an error, since such a column cannot be compared, and thus unique values cannot be found.

It would probably be a good idea to allow fields to specify whether they cannot be distinct:
custom = MyCustomField(distinctless=True)

which would turn this ordinary distinct query:
SELECT DISTINCT id, number, custom FROM foo;

into:
SELECT DISTINCT ON(id, number) FROM FOO;

Discussion link:
​https://groups.google.com/forum/#!topic/django-developers/iO8z3iLpgg4

[Karen Tracey]

Not sure we want to be adding a new option between now and 1.3 final, but seems like we need to do something (minimum document the change as a backwards-incompatibility?).

[Luke Plant]

We could also revert the fix in [15607], and either:

• leave #11707 open
• Fix #11707 by doing duplicate elimination in Python

[Ian Kelly]

I discovered another problem with the distinct queries from [15607]. Using Oracle, it breaks foreign keys to models that have TextFields. This is because Oracle won't let you do SELECT DISTINCT with a select list that includes a LOB column.

+1 for reverting.

[Michael Radziej]

ikelly, does Oracle support SELECT DISTINCT ON (id) if there's a LOB column in the SELECT list?

[Ian Kelly]

Replying to mir:

ikelly, does Oracle support SELECT DISTINCT ON (id) if there's a LOB column in the SELECT list?

No. The analytic query that emulates DISTINCT ON works using a regular DISTINCT query and selecting for each column the first value in that column for the corresponding DISTINCT ON partition in place of the actual value for the row. The result is that rows that are equivalent over the DISTINCT ON partition become equivalent over the entire row and so are filtered out by the DISTINCT keyword. That means you're still effectively doing a SELECT DISTINCT with a LOB column.

[Luke Plant]

In [15791]:

Fixed #15559 - distinct queries introduced by [15607] cause errors with some custom model fields

This patch just reverts [15607] until a more satisfying solution can be
found.

Refs #11707

[Luke Plant]

In [15792]:

[1.2.X] Fixed #15559 - distinct queries introduced by [15607] cause errors with some custom model fields

This patch just reverts [15607] until a more satisfying solution can be
found.

Refs #11707

Backport of [15791] from trunk.

[Luke Plant]

As you can see, I decided to revert [15607], because of the support for reverting listed above and on the mailing list, and because of the closeness of the 1.3 release.

[Luke Plant]

There are other places where we are using distinct (see [15526]), so it is possible that this bug is only fixed in the one instance where the reporter experienced a regression. A more satisfying general solution is probably still needed, but a new bug should be opened for that.

[Carl Meyer]

Replying to lukeplant:

There are other places where we are using distinct (see [15526]), so it is possible that this bug is only fixed in the one instance where the reporter experienced a regression. A more satisfying general solution is probably still needed, but a new bug should be opened for that.

I don't see any need to open a new bug - it would just be a copy-paste of this one. The description for this bug is still fully applicable (and was never fixed, just worked around). Reopening and removing 1.3-blocker keywords.

[Aymeric Augustin]

Change UI/UX from NULL to False.

[Aymeric Augustin]

Change Easy pickings from NULL to False.

[Chris Wesseling]

Replying to lukeplant:

We could also revert the fix in [15607], and either:

• leave #11707 open
• Fix #11707 by doing duplicate elimination in Python

I'm revisiting #11707 on a sprint. I implemented the duplicate elimination 2 years ago with ojii, but it never got RFC'ed.

Since then I thought: DISTINCT logically equivalent with a GROUP BY on all fields. Not all databases support DISTINCT ON, but a GROUP BY on just the pk should do the trick (at least in #11707).