Opened 10 years ago

Closed 22 months ago

#1453 closed New feature (worksforme)

generic views sometimes inefficient with large data sets - especially archive_index

Reported by: hugo Owned by: zefciu
Component: Generic views Version:
Severity: Normal Keywords:
Cc: Triage Stage: Accepted
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: yes
Easy pickings: no UI/UX: no


archive_index will give a list of years with available data to the template in the context. To discover these years, it will do the following SQL:

SELECT DATE_TRUNC(\'year\', "logviewer_message"."time")
FROM "logviewer_message"
WHERE ("logviewer_message"."channel_id" = 2 AND 
       "logviewer_message"."time" <= 2006-03-02 12:07:20.758531)

This is rather icky if you have very much data in your database. For example the IRC logger currently has over 100000 messages stored for the #django channel. It will do an index scan for the timestamp - but that is allways "now" in archive_index, so it won't help much. It will essentially dig through all rows sequentially.

I think either this list of years should be thrown out or should be made optional (add a with_list_of_years parameter to the generic view that people can set if they need this behaviour).

Attachments (1)

issue_1453.diff (8.5 KB) - added by zefciu 4 years ago.

Download all attachments as: .zip

Change History (10)

comment:1 Changed 10 years ago by Esaj

How about putting an object in the context which can lazily get a list of years or months? I think it's more KISS than adding another parameter :) Getting a list of months in archive_index would actually be quite useful, in particular for blogs.

comment:2 Changed 9 years ago by Simon G. <dev@…>

  • Triage Stage changed from Unreviewed to Design decision needed

comment:3 Changed 8 years ago by jacob

  • Triage Stage changed from Design decision needed to Someday/Maybe

comment:4 Changed 5 years ago by lrekucki

  • Severity changed from normal to Normal
  • Type changed from defect to New feature

This can be easily solved in class-based views.

comment:5 Changed 4 years ago by zefciu

  • Easy pickings unset
  • Owner changed from nobody to zefciu
  • Status changed from new to assigned
  • UI/UX unset

Changed 4 years ago by zefciu

comment:6 Changed 4 years ago by zefciu

  • Has patch set

I have created a patch that makes date_list in both class-based and function-based views lazy. Also created some tests to check the behavior of function-based archive-index (there were none before) and to ensure date_list is a Promise

comment:7 Changed 3 years ago by aaugustin

  • Triage Stage changed from Someday/Maybe to Accepted

comment:8 Changed 2 years ago by timo

  • Patch needs improvement set

Patch doesn't apply cleanly to master.

comment:9 Changed 22 months ago by mjtamlyn

  • Resolution set to worksforme
  • Status changed from assigned to closed

My instinct is that if you have the problem then you should override your view and make get_date_list return None, or however you wish to handle it. Wrapping the date_list in lazy objects adds some complexity, and saves errors in the defaults, but only if you don't actually use date_list in your template. If you do use date_list then you have the same performance problem. It is easy enough to avoid the performance issue by overriding a single method in the CBV.

Note: See TracTickets for help on using tickets.
Back to Top