generic views sometimes inefficient with large data sets - especially archive_index

[hugo]

archive_index will give a list of years with available data to the template in the context. To discover these years, it will do the following SQL:

SELECT DATE_TRUNC(\'year\', "logviewer_message"."time")
FROM "logviewer_message"
WHERE ("logviewer_message"."channel_id" = 2 AND 
       "logviewer_message"."time" <= 2006-03-02 12:07:20.758531)
GROUP BY 1 ORDER BY 1 ASC

This is rather icky if you have very much data in your database. For example the IRC logger currently has over 100000 messages stored for the #django channel. It will do an index scan for the timestamp - but that is allways "now" in archive_index, so it won't help much. It will essentially dig through all rows sequentially.

I think either this list of years should be thrown out or should be made optional (add a with_list_of_years parameter to the generic view that people can set if they need this behaviour).

[Esaj]

How about putting an object in the context which can lazily get a list of years or months? I think it's more KISS than adding another parameter :) Getting a list of months in archive_index would actually be quite useful, in particular for blogs.

[Łukasz Rekucki]

This can be easily solved in class-based views.

[zefciu]

I have created a patch that makes date_list in both class-based and function-based views lazy. Also created some tests to check the behavior of function-based archive-index (there were none before) and to ensure date_list is a Promise

[Tim Graham]

Patch doesn't apply cleanly to master.

[Marc Tamlyn]

My instinct is that if you have the problem then you should override your view and make get_date_list return None, or however you wish to handle it. Wrapping the date_list in lazy objects adds some complexity, and saves errors in the defaults, but only if you don't actually use date_list in your template. If you do use date_list then you have the same performance problem. It is easy enough to avoid the performance issue by overriding a single method in the CBV.