Default filters: pacth to add QuerySet support for random plus a new shuffle filter

[h3]

I believe the way the random filter works right now is asinine or at least not very useful.

When working in templates, 90% of the lists we have to deal with are QuerySet instances, so for this filter to choke when we pass it an object list is not very helpful.

I've seen a proposed patch for this posted three years ago, but it was rejected on the ground that it required a new db hit and thus was problematic for large querysets. Fair enough.

My patch use the queryset.order_by('?') method so it (theoretically) doesn't require a new database hit.

It also introduce a new "shuffle" filter, which useful when you want to randomize a list or a queryset instead of just picking one randomly.

[h3]

Sorry for the typo .. looks like I have a hard time writing the word "patch" today.

[Anssi Kääriäinen]

I am not sure this is a good idea. The problem is that qs.order_by('?')[0] might be really expensive. If I am not mistaken, the DB must go through the whole result set and sort everything. Even if the DB returns just one row, that can be expensive. I am afraid that users of random do not realize this, as how could a fetch of just a single object cost much...

Also, in the patch you use len(qs). That means the queryset gets evaluated (or at least count(*) is evaluated) there. try - except indexerror is better.

Setting this to design decision needed.

[anonymous]

order_by('?') *does* cause an additional query, if the queryset has already been evaluated. order_by() is handled by the database.

If the queryset has already been evaluated, you could just transform it to a list and perform the ordinary random/shuffle operations on that, I suppose... would that be faster?

[h3]

If the queryset has already been evaluated, you could just transform it to a list and perform the ordinary random/shuffle operations on that, I suppose... would that be faster?

I will have to make some tests to tell

I am not sure this is a good idea. The problem is that qs.order_by('?')[0] might be really expensive. If I am not mistaken, the DB must go through the whole result set and sort everything. Even if the DB returns just one row, that can be expensive. I am afraid that users of random do not realize this, as how could a fetch of just a single object cost much...

Also, in the patch you use len(qs). That means the queryset gets evaluated (or at least count(*) is evaluated) there. try - except indexerror is better.

This strengthen my feeling that these functionality (random/shuffle filter for qs) should really be included by django core.

What I did was the "obvious" way of doing it and it's how a lot of django developers will implement it the day they'll need this functionality.

Now the question is, what's the optimal way to do it..

[Jacob]

I still agree with what Malcolm wrote three years ago:

I'm inclined to say "no" on this feature. If you find yourself wanting to use random selection a lot on large querysets that wouldn't otherwise have their results used and you're happy to pay for the extra database query, you could write your own tag that does the same thing.

To expand a bit, the reason Django has a custom tag/filter API is precisely so that we don't have to include every potential feature as a built-in. To me, the shuffle tag doesn't cross the widely-useful-enough threshold. (Frankly, it's probably something better done in the view anyway.)

As for the changes to the random filter: I very much don't like special-casing QuerySet there. It's an explicit coupling of the template language to the model layer, and we should be *removing* those couplings, not introducing more of them.

Given that, I'm going to mark this wontfix. Thanks for the suggestions; sorry to have to turn 'em down!