generic view archive_year is querying all objects, unexplained.

[Steven Wagner <stevenwagner@…>]

Here is my urls.py , pretty much identical to djangobook.

info_dict = {
    'queryset': Report.objects.all(),
    'date_field': 'date',
    'make_object_list': 'False',
}

urlpatterns = patterns('django.views.generic.date_based',
   (r'^report/(?P<year>\d{4})/(?P<month>[a-z]{3})/(?P<day>\w{1,2})/(?P<slug>[-\w]+)/$', 'object_detail', dict(info_dict, slug_field='slug')),
   (r'^report/(?P<year>\d{4})/(?P<month>[a-z]{3})/(?P<day>\w{1,2})/$',                  'archive_day',   info_dict),
   (r'^report/(?P<year>\d{4})/(?P<month>[a-z]{3})/$',                                   'archive_month', info_dict),
   (r'^report/(?P<year>\d{4})/$',                                                       'archive_year',  info_dict),
   (r'^report/?$',                                                                     'archive_index', info_dict),
)

Here are the queries I see in my sql log, as the python process climbs in memory usage and 100cpu until django server crashes.

SELECT CAST(DATE_FORMAT(`reports`.`date`, '%Y-%m-01 00:00:00') AS DATETIME)  
FROM `reports` 
WHERE `reports`.`date` IS NOT NULL AND (`reports`.`date` <= '2007-05-23' AND `reports`.`date` BETWEEN '2007-01-01 00:00:00' AND '2007-12-31 23:59:59.999999') 
GROUP BY 1 ORDER BY 1 ASC

SELECT `reports`.`id`,`... FROM `reports

My reports table is huge, but the datebased generic views shouldnt be loading the entire reporting table.
Especially not with make_object_list set to false.

[SuperJared]

It should be:

'make_object_list': False

You were passing a string.

[Steven Wagner <stevenwagner@…>]

Fixed the False string. Doesn't change behavior.

Here is the traceback of the crash:

Traceback (most recent call last):

  File "/usr/lib/python2.4/site-packages/django/core/servers/basehttp.py", line 272, in run
    self.result = application(self.environ, self.start_response)

  File "/usr/lib/python2.4/site-packages/django/core/servers/basehttp.py", line 614, in __call__
    return self.application(environ, start_response)

  File "/usr/lib/python2.4/site-packages/django/core/handlers/wsgi.py", line 189, in __call__
    response = self.get_response(request)

  File "/usr/lib/python2.4/site-packages/django/core/handlers/base.py", line 111, in get_response
    return debug.technical_500_response(request, *sys.exc_info())

  File "/usr/lib/python2.4/site-packages/django/views/debug.py", line 131, in technical_500_response
    return HttpResponseServerError(t.render(c), mimetype='text/html')

  File "/usr/lib/python2.4/site-packages/django/template/__init__.py", line 168, in render
    return self.nodelist.render(context)

  File "/usr/lib/python2.4/site-packages/django/template/__init__.py", line 705, in render
    bits.append(self.render_node(node, context))

  File "/usr/lib/python2.4/site-packages/django/template/__init__.py", line 723, in render_node
    result = node.render(context)

  File "/usr/lib/python2.4/site-packages/django/template/defaulttags.py", line 122, in render
    nodelist.append(node.render(context))

  File "/usr/lib/python2.4/site-packages/django/template/defaulttags.py", line 209, in render
    return self.nodelist_true.render(context)

  File "/usr/lib/python2.4/site-packages/django/template/__init__.py", line 705, in render
    bits.append(self.render_node(node, context))

  File "/usr/lib/python2.4/site-packages/django/template/__init__.py", line 733, in render_node
    raise wrapped

TemplateSyntaxError: Caught an exception while rendering: 

Original Traceback (most recent call last):
  File "/usr/lib/python2.4/site-packages/django/template/__init__.py", line 723, in render_node
    result = node.render(context)
  File "/usr/lib/python2.4/site-packages/django/template/defaulttags.py", line 122, in render
    nodelist.append(node.render(context))
  File "/usr/lib/python2.4/site-packages/django/template/__init__.py", line 773, in render
    output = self.filter_expression.resolve(context)
  File "/usr/lib/python2.4/site-packages/django/template/__init__.py", line 579, in resolve
    obj = func(obj, *arg_vals)
  File "/usr/lib/python2.4/site-packages/django/template/defaultfilters.py", line 36, in _dec
    return func(*args, **kwargs)
  File "/usr/lib/python2.4/site-packages/django/template/defaultfilters.py", line 259, in escape
    return escape(value)
  File "/usr/lib/python2.4/site-packages/django/utils/html.py", line 28, in escape
    return html.replace('&', '&amp;').replace('<', '&lt;').replace('>', '&gt;').replace('"', '&quot;').replace("'", '&#39;')
MemoryError

[Malcolm Tredinnick]

A simple look at the code settles this. The archive_year function is clearly not creating object_list if make_object_list is false. It does have to extract all the data to get the dates, though, so this might be the cause of your problem. In any case, this isn't the bug you think it is. You might wish to ask on django-users, describing your model setup and data sizes to get help on managing the memory usage.

You may have also rediscovered a variant of #1453.

[Steven Wagner <stevenwagner@…>]

I am able to use the generic date view on other models and it does not query all of the data from the table. But on my Reports model, even after I cut it down, it still queries all the data in the table.

class Report(models.Model):
    date = models.DateField()
    class Meta:
        db_table = 'reports'

070524 12:29:44   35906 Query       
SELECT CAST(DATE_FORMAT(`reports`.`date`, '%Y-%m-01 00:00:00') AS DATETIME)  FROM `reports` WHERE (`reports`.`date` <= '2007-05-24' 
AND `reports`.`date` BETWEEN '2007-01-01 00:00:00' AND '2007-12-31 23:59:59.999999') 
GROUP BY 1 ORDER BY 1 ASC

35906 Query       
SELECT `reports`.`id`,`reports`.`date` FROM `reports`

Ive looked in the source of the files mentioned in the trace above, but am not sure how to troubleshoot this next.

[Steven Wagner <stevenwagner@…>]

Making my own custom view works without the overhead of the generic date view.

def report_list(request):
    month_list = Report.objects.all().dates('date', 'month', order='DESC')
    return render_to_response('list/report.html', {
        "month_list": month_list,
    })

SELECT CAST(DATE_FORMAT(`reports`.`date`, '%Y-%m-01 00:00:00') AS DATETIME)  
FROM `reports` WHERE `reports`.`date` IS NOT NULL GROUP BY 1 ORDER BY 1 DESC

[shwag < >]

Let me know how I can troubleshoot this, or additional information I can provide.

[shwag < >]

The source of the sql queries was and exception happening that caused the django error page to evaluate the complete set.