Opened 10 years ago

Closed 10 years ago

#4375 closed (wontfix)

generic view archive_year is querying all objects, unexplained.

Reported by: Steven Wagner <stevenwagner@…> Owned by: Jacob
Component: Generic views Version: 0.96
Severity: Keywords:
Cc: Triage Stage: Unreviewed
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: UI/UX:

Description

Here is my urls.py , pretty much identical to djangobook.

info_dict = {
    'queryset': Report.objects.all(),
    'date_field': 'date',
    'make_object_list': 'False',
}

urlpatterns = patterns('django.views.generic.date_based',
   (r'^report/(?P<year>\d{4})/(?P<month>[a-z]{3})/(?P<day>\w{1,2})/(?P<slug>[-\w]+)/$', 'object_detail', dict(info_dict, slug_field='slug')),
   (r'^report/(?P<year>\d{4})/(?P<month>[a-z]{3})/(?P<day>\w{1,2})/$',                  'archive_day',   info_dict),
   (r'^report/(?P<year>\d{4})/(?P<month>[a-z]{3})/$',                                   'archive_month', info_dict),
   (r'^report/(?P<year>\d{4})/$',                                                       'archive_year',  info_dict),
   (r'^report/?$',                                                                     'archive_index', info_dict),
)

Here are the queries I see in my sql log, as the python process climbs in memory usage and 100cpu until django server crashes.

SELECT CAST(DATE_FORMAT(`reports`.`date`, '%Y-%m-01 00:00:00') AS DATETIME)  
FROM `reports` 
WHERE `reports`.`date` IS NOT NULL AND (`reports`.`date` <= '2007-05-23' AND `reports`.`date` BETWEEN '2007-01-01 00:00:00' AND '2007-12-31 23:59:59.999999') 
GROUP BY 1 ORDER BY 1 ASC

SELECT `reports`.`id`,`... FROM `reports

My reports table is huge, but the datebased generic views shouldnt be loading the entire reporting table.
Especially not with make_object_list set to false.

Change History (8)

comment:1 Changed 10 years ago by SuperJared

Resolution: invalid
Status: newclosed

It should be:

'make_object_list': False

You were passing a string.

comment:2 Changed 10 years ago by Steven Wagner <stevenwagner@…>

Resolution: invalid
Status: closedreopened

Fixed the False string. Doesn't change behavior.

Here is the traceback of the crash:

Traceback (most recent call last):

  File "/usr/lib/python2.4/site-packages/django/core/servers/basehttp.py", line 272, in run
    self.result = application(self.environ, self.start_response)

  File "/usr/lib/python2.4/site-packages/django/core/servers/basehttp.py", line 614, in __call__
    return self.application(environ, start_response)

  File "/usr/lib/python2.4/site-packages/django/core/handlers/wsgi.py", line 189, in __call__
    response = self.get_response(request)

  File "/usr/lib/python2.4/site-packages/django/core/handlers/base.py", line 111, in get_response
    return debug.technical_500_response(request, *sys.exc_info())

  File "/usr/lib/python2.4/site-packages/django/views/debug.py", line 131, in technical_500_response
    return HttpResponseServerError(t.render(c), mimetype='text/html')

  File "/usr/lib/python2.4/site-packages/django/template/__init__.py", line 168, in render
    return self.nodelist.render(context)

  File "/usr/lib/python2.4/site-packages/django/template/__init__.py", line 705, in render
    bits.append(self.render_node(node, context))

  File "/usr/lib/python2.4/site-packages/django/template/__init__.py", line 723, in render_node
    result = node.render(context)

  File "/usr/lib/python2.4/site-packages/django/template/defaulttags.py", line 122, in render
    nodelist.append(node.render(context))

  File "/usr/lib/python2.4/site-packages/django/template/defaulttags.py", line 209, in render
    return self.nodelist_true.render(context)

  File "/usr/lib/python2.4/site-packages/django/template/__init__.py", line 705, in render
    bits.append(self.render_node(node, context))

  File "/usr/lib/python2.4/site-packages/django/template/__init__.py", line 733, in render_node
    raise wrapped

TemplateSyntaxError: Caught an exception while rendering: 

Original Traceback (most recent call last):
  File "/usr/lib/python2.4/site-packages/django/template/__init__.py", line 723, in render_node
    result = node.render(context)
  File "/usr/lib/python2.4/site-packages/django/template/defaulttags.py", line 122, in render
    nodelist.append(node.render(context))
  File "/usr/lib/python2.4/site-packages/django/template/__init__.py", line 773, in render
    output = self.filter_expression.resolve(context)
  File "/usr/lib/python2.4/site-packages/django/template/__init__.py", line 579, in resolve
    obj = func(obj, *arg_vals)
  File "/usr/lib/python2.4/site-packages/django/template/defaultfilters.py", line 36, in _dec
    return func(*args, **kwargs)
  File "/usr/lib/python2.4/site-packages/django/template/defaultfilters.py", line 259, in escape
    return escape(value)
  File "/usr/lib/python2.4/site-packages/django/utils/html.py", line 28, in escape
    return html.replace('&', '&amp;').replace('<', '&lt;').replace('>', '&gt;').replace('"', '&quot;').replace("'", '&#39;')
MemoryError

comment:3 Changed 10 years ago by Malcolm Tredinnick

Resolution: invalid
Status: reopenedclosed

A simple look at the code settles this. The archive_year function is clearly not creating object_list if make_object_list is false. It does have to extract all the data to get the dates, though, so this might be the cause of your problem. In any case, this isn't the bug you think it is. You might wish to ask on django-users, describing your model setup and data sizes to get help on managing the memory usage.

You may have also rediscovered a variant of #1453.

comment:4 Changed 10 years ago by Steven Wagner <stevenwagner@…>

I am able to use the generic date view on other models and it does not query all of the data from the table. But on my Reports model, even after I cut it down, it still queries all the data in the table.

class Report(models.Model):
    date = models.DateField()
    class Meta:
        db_table = 'reports'
070524 12:29:44   35906 Query       
SELECT CAST(DATE_FORMAT(`reports`.`date`, '%Y-%m-01 00:00:00') AS DATETIME)  FROM `reports` WHERE (`reports`.`date` <= '2007-05-24' 
AND `reports`.`date` BETWEEN '2007-01-01 00:00:00' AND '2007-12-31 23:59:59.999999') 
GROUP BY 1 ORDER BY 1 ASC

35906 Query       
SELECT `reports`.`id`,`reports`.`date` FROM `reports`

Ive looked in the source of the files mentioned in the trace above, but am not sure how to troubleshoot this next.

comment:5 Changed 10 years ago by Steven Wagner <stevenwagner@…>

Making my own custom view works without the overhead of the generic date view.

def report_list(request):
    month_list = Report.objects.all().dates('date', 'month', order='DESC')
    return render_to_response('list/report.html', {
        "month_list": month_list,
    })

SELECT CAST(DATE_FORMAT(`reports`.`date`, '%Y-%m-01 00:00:00') AS DATETIME)  
FROM `reports` WHERE `reports`.`date` IS NOT NULL GROUP BY 1 ORDER BY 1 DESC

comment:6 Changed 10 years ago by shwag <shwag>

Resolution: invalid
Status: closedreopened

comment:7 Changed 10 years ago by shwag < >

Summary: generic view archive_year is always making and object_list even if false make_object_listgeneric view archive_year is querying all objects, unexplained.

Let me know how I can troubleshoot this, or additional information I can provide.

comment:8 Changed 10 years ago by shwag < >

Resolution: wontfix
Status: reopenedclosed

The source of the sql queries was and exception happening that caused the django error page to evaluate the complete set.

Note: See TracTickets for help on using tickets.
Back to Top