Opened 4 years ago

Closed 4 years ago

#16647 closed New feature (wontfix)

Default filters: pacth to add QuerySet support for random plus a new shuffle filter

Reported by: h3 Owned by: nobody
Component: Template system Version: 1.3
Severity: Normal Keywords:
Cc: Triage Stage: Design decision needed
Has patch: yes Needs documentation: yes
Needs tests: yes Patch needs improvement: yes
Easy pickings: no UI/UX: no

Description

I believe the way the random filter works right now is asinine or at least not very useful.

When working in templates, 90% of the lists we have to deal with are QuerySet instances, so for this filter to choke when we pass it an object list is not very helpful.

I've seen a proposed patch for this posted three years ago, but it was rejected on the ground that it required a new db hit and thus was problematic for large querysets. Fair enough.

My patch use the queryset.order_by('?') method so it (theoretically) doesn't require a new database hit.

It also introduce a new "shuffle" filter, which useful when you want to randomize a list or a queryset instead of just picking one randomly.

Attachments (1)

django-defaultfilters-random-shuffle.path (1.4 KB) - added by h3 4 years ago.

Download all attachments as: .zip

Change History (6)

comment:1 Changed 4 years ago by h3

  • Needs documentation unset
  • Needs tests unset
  • Patch needs improvement unset

Sorry for the typo .. looks like I have a hard time writing the word "patch" today.

comment:2 Changed 4 years ago by akaariai

  • Needs documentation set
  • Needs tests set
  • Patch needs improvement set
  • Triage Stage changed from Unreviewed to Design decision needed

I am not sure this is a good idea. The problem is that qs.order_by('?')[0] might be really expensive. If I am not mistaken, the DB must go through the whole result set and sort everything. Even if the DB returns just one row, that can be expensive. I am afraid that users of random do not realize this, as how could a fetch of just a single object cost much...

Also, in the patch you use len(qs). That means the queryset gets evaluated (or at least count(*) is evaluated) there. try - except indexerror is better.

Setting this to design decision needed.

comment:3 Changed 4 years ago by anonymous

order_by('?') *does* cause an additional query, if the queryset has already been evaluated. order_by() is handled by the database.

If the queryset has already been evaluated, you could just transform it to a list and perform the ordinary random/shuffle operations on that, I suppose... would that be faster?

comment:4 Changed 4 years ago by h3

If the queryset has already been evaluated, you could just transform it to a list and perform the ordinary random/shuffle operations on that, I suppose... would that be faster?

I will have to make some tests to tell

I am not sure this is a good idea. The problem is that qs.order_by('?')[0] might be really expensive. If I am not mistaken, the DB must go through the whole result set and sort everything. Even if the DB returns just one row, that can be expensive. I am afraid that users of random do not realize this, as how could a fetch of just a single object cost much...

Also, in the patch you use len(qs). That means the queryset gets evaluated (or at least count(*) is evaluated) there. try - except indexerror is better.

This strengthen my feeling that these functionality (random/shuffle filter for qs) should really be included by django core.

What I did was the "obvious" way of doing it and it's how a lot of django developers will implement it the day they'll need this functionality.

Now the question is, what's the optimal way to do it..

comment:5 Changed 4 years ago by jacob

  • Resolution set to wontfix
  • Status changed from new to closed

I still agree with what Malcolm wrote three years ago:

I'm inclined to say "no" on this feature. If you find yourself wanting to use random selection a lot on large querysets that wouldn't otherwise have their results used and you're happy to pay for the extra database query, you could write your own tag that does the same thing.

To expand a bit, the reason Django has a custom tag/filter API is precisely so that we don't have to include every potential feature as a built-in. To me, the shuffle tag doesn't cross the widely-useful-enough threshold. (Frankly, it's probably something better done in the view anyway.)

As for the changes to the random filter: I very much don't like special-casing QuerySet there. It's an explicit coupling of the template language to the model layer, and we should be *removing* those couplings, not introducing more of them.

Given that, I'm going to mark this wontfix. Thanks for the suggestions; sorry to have to turn 'em down!

Note: See TracTickets for help on using tickets.
Back to Top