Opened 11 years ago

Closed 3 years ago

#1453 closed New feature (worksforme)

generic views sometimes inefficient with large data sets - especially archive_index

Reported by: hugo Owned by: zefciu
Component: Generic views Version:
Severity: Normal Keywords:
Cc: Triage Stage: Accepted
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: yes
Easy pickings: no UI/UX: no


archive_index will give a list of years with available data to the template in the context. To discover these years, it will do the following SQL:

SELECT DATE_TRUNC(\'year\', "logviewer_message"."time")
FROM "logviewer_message"
WHERE ("logviewer_message"."channel_id" = 2 AND 
       "logviewer_message"."time" <= 2006-03-02 12:07:20.758531)

This is rather icky if you have very much data in your database. For example the IRC logger currently has over 100000 messages stored for the #django channel. It will do an index scan for the timestamp - but that is allways "now" in archive_index, so it won't help much. It will essentially dig through all rows sequentially.

I think either this list of years should be thrown out or should be made optional (add a with_list_of_years parameter to the generic view that people can set if they need this behaviour).

Attachments (1)

issue_1453.diff (8.5 KB) - added by zefciu 5 years ago.

Download all attachments as: .zip

Change History (10)

comment:1 Changed 11 years ago by Esaj

How about putting an object in the context which can lazily get a list of years or months? I think it's more KISS than adding another parameter :) Getting a list of months in archive_index would actually be quite useful, in particular for blogs.

comment:2 Changed 10 years ago by Simon G. <dev@…>

Triage Stage: UnreviewedDesign decision needed

comment:3 Changed 9 years ago by Jacob

Triage Stage: Design decision neededSomeday/Maybe

comment:4 Changed 5 years ago by Łukasz Rekucki

Severity: normalNormal
Type: defectNew feature

This can be easily solved in class-based views.

comment:5 Changed 5 years ago by zefciu

Easy pickings: unset
Owner: changed from nobody to zefciu
Status: newassigned
UI/UX: unset

Changed 5 years ago by zefciu

Attachment: issue_1453.diff added

comment:6 Changed 5 years ago by zefciu

Has patch: set

I have created a patch that makes date_list in both class-based and function-based views lazy. Also created some tests to check the behavior of function-based archive-index (there were none before) and to ensure date_list is a Promise

comment:7 Changed 4 years ago by Aymeric Augustin

Triage Stage: Someday/MaybeAccepted

comment:8 Changed 3 years ago by Tim Graham

Patch needs improvement: set

Patch doesn't apply cleanly to master.

comment:9 Changed 3 years ago by Marc Tamlyn

Resolution: worksforme
Status: assignedclosed

My instinct is that if you have the problem then you should override your view and make get_date_list return None, or however you wish to handle it. Wrapping the date_list in lazy objects adds some complexity, and saves errors in the defaults, but only if you don't actually use date_list in your template. If you do use date_list then you have the same performance problem. It is easy enough to avoid the performance issue by overriding a single method in the CBV.

Note: See TracTickets for help on using tickets.
Back to Top