Code

Opened 7 years ago

Closed 7 years ago

#4375 closed (wontfix)

generic view archive_year is querying all objects, unexplained.

Reported by: Steven Wagner <stevenwagner@…> Owned by: jacob
Component: Generic views Version: 0.96
Severity: Keywords:
Cc: Triage Stage: Unreviewed
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: UI/UX:

Description

Here is my urls.py , pretty much identical to djangobook.

info_dict = {
    'queryset': Report.objects.all(),
    'date_field': 'date',
    'make_object_list': 'False',
}

urlpatterns = patterns('django.views.generic.date_based',
   (r'^report/(?P<year>\d{4})/(?P<month>[a-z]{3})/(?P<day>\w{1,2})/(?P<slug>[-\w]+)/$', 'object_detail', dict(info_dict, slug_field='slug')),
   (r'^report/(?P<year>\d{4})/(?P<month>[a-z]{3})/(?P<day>\w{1,2})/$',                  'archive_day',   info_dict),
   (r'^report/(?P<year>\d{4})/(?P<month>[a-z]{3})/$',                                   'archive_month', info_dict),
   (r'^report/(?P<year>\d{4})/$',                                                       'archive_year',  info_dict),
   (r'^report/?$',                                                                     'archive_index', info_dict),
)

Here are the queries I see in my sql log, as the python process climbs in memory usage and 100cpu until django server crashes.

SELECT CAST(DATE_FORMAT(`reports`.`date`, '%Y-%m-01 00:00:00') AS DATETIME)  
FROM `reports` 
WHERE `reports`.`date` IS NOT NULL AND (`reports`.`date` <= '2007-05-23' AND `reports`.`date` BETWEEN '2007-01-01 00:00:00' AND '2007-12-31 23:59:59.999999') 
GROUP BY 1 ORDER BY 1 ASC

SELECT `reports`.`id`,`... FROM `reports

My reports table is huge, but the datebased generic views shouldnt be loading the entire reporting table.
Especially not with make_object_list set to false.

Attachments (0)

Change History (8)

comment:1 Changed 7 years ago by SuperJared

  • Needs documentation unset
  • Needs tests unset
  • Patch needs improvement unset
  • Resolution set to invalid
  • Status changed from new to closed

It should be:

'make_object_list': False

You were passing a string.

comment:2 Changed 7 years ago by Steven Wagner <stevenwagner@…>

  • Resolution invalid deleted
  • Status changed from closed to reopened

Fixed the False string. Doesn't change behavior.

Here is the traceback of the crash:

Traceback (most recent call last):

  File "/usr/lib/python2.4/site-packages/django/core/servers/basehttp.py", line 272, in run
    self.result = application(self.environ, self.start_response)

  File "/usr/lib/python2.4/site-packages/django/core/servers/basehttp.py", line 614, in __call__
    return self.application(environ, start_response)

  File "/usr/lib/python2.4/site-packages/django/core/handlers/wsgi.py", line 189, in __call__
    response = self.get_response(request)

  File "/usr/lib/python2.4/site-packages/django/core/handlers/base.py", line 111, in get_response
    return debug.technical_500_response(request, *sys.exc_info())

  File "/usr/lib/python2.4/site-packages/django/views/debug.py", line 131, in technical_500_response
    return HttpResponseServerError(t.render(c), mimetype='text/html')

  File "/usr/lib/python2.4/site-packages/django/template/__init__.py", line 168, in render
    return self.nodelist.render(context)

  File "/usr/lib/python2.4/site-packages/django/template/__init__.py", line 705, in render
    bits.append(self.render_node(node, context))

  File "/usr/lib/python2.4/site-packages/django/template/__init__.py", line 723, in render_node
    result = node.render(context)

  File "/usr/lib/python2.4/site-packages/django/template/defaulttags.py", line 122, in render
    nodelist.append(node.render(context))

  File "/usr/lib/python2.4/site-packages/django/template/defaulttags.py", line 209, in render
    return self.nodelist_true.render(context)

  File "/usr/lib/python2.4/site-packages/django/template/__init__.py", line 705, in render
    bits.append(self.render_node(node, context))

  File "/usr/lib/python2.4/site-packages/django/template/__init__.py", line 733, in render_node
    raise wrapped

TemplateSyntaxError: Caught an exception while rendering: 

Original Traceback (most recent call last):
  File "/usr/lib/python2.4/site-packages/django/template/__init__.py", line 723, in render_node
    result = node.render(context)
  File "/usr/lib/python2.4/site-packages/django/template/defaulttags.py", line 122, in render
    nodelist.append(node.render(context))
  File "/usr/lib/python2.4/site-packages/django/template/__init__.py", line 773, in render
    output = self.filter_expression.resolve(context)
  File "/usr/lib/python2.4/site-packages/django/template/__init__.py", line 579, in resolve
    obj = func(obj, *arg_vals)
  File "/usr/lib/python2.4/site-packages/django/template/defaultfilters.py", line 36, in _dec
    return func(*args, **kwargs)
  File "/usr/lib/python2.4/site-packages/django/template/defaultfilters.py", line 259, in escape
    return escape(value)
  File "/usr/lib/python2.4/site-packages/django/utils/html.py", line 28, in escape
    return html.replace('&', '&amp;').replace('<', '&lt;').replace('>', '&gt;').replace('"', '&quot;').replace("'", '&#39;')
MemoryError

comment:3 Changed 7 years ago by mtredinnick

  • Resolution set to invalid
  • Status changed from reopened to closed

A simple look at the code settles this. The archive_year function is clearly not creating object_list if make_object_list is false. It does have to extract all the data to get the dates, though, so this might be the cause of your problem. In any case, this isn't the bug you think it is. You might wish to ask on django-users, describing your model setup and data sizes to get help on managing the memory usage.

You may have also rediscovered a variant of #1453.

comment:4 Changed 7 years ago by Steven Wagner <stevenwagner@…>

I am able to use the generic date view on other models and it does not query all of the data from the table. But on my Reports model, even after I cut it down, it still queries all the data in the table.

class Report(models.Model):
    date = models.DateField()
    class Meta:
        db_table = 'reports'
070524 12:29:44   35906 Query       
SELECT CAST(DATE_FORMAT(`reports`.`date`, '%Y-%m-01 00:00:00') AS DATETIME)  FROM `reports` WHERE (`reports`.`date` <= '2007-05-24' 
AND `reports`.`date` BETWEEN '2007-01-01 00:00:00' AND '2007-12-31 23:59:59.999999') 
GROUP BY 1 ORDER BY 1 ASC

35906 Query       
SELECT `reports`.`id`,`reports`.`date` FROM `reports`

Ive looked in the source of the files mentioned in the trace above, but am not sure how to troubleshoot this next.

comment:5 Changed 7 years ago by Steven Wagner <stevenwagner@…>

Making my own custom view works without the overhead of the generic date view.

def report_list(request):
    month_list = Report.objects.all().dates('date', 'month', order='DESC')
    return render_to_response('list/report.html', {
        "month_list": month_list,
    })

SELECT CAST(DATE_FORMAT(`reports`.`date`, '%Y-%m-01 00:00:00') AS DATETIME)  
FROM `reports` WHERE `reports`.`date` IS NOT NULL GROUP BY 1 ORDER BY 1 DESC

comment:6 Changed 7 years ago by shwag <shwag>

  • Resolution invalid deleted
  • Status changed from closed to reopened

comment:7 Changed 7 years ago by shwag < >

  • Summary changed from generic view archive_year is always making and object_list even if false make_object_list to generic view archive_year is querying all objects, unexplained.

Let me know how I can troubleshoot this, or additional information I can provide.

comment:8 Changed 7 years ago by shwag < >

  • Resolution set to wontfix
  • Status changed from reopened to closed

The source of the sql queries was and exception happening that caused the django error page to evaluate the complete set.

Add Comment

Modify Ticket

Change Properties
<Author field>
Action
as closed
as The resolution will be set. Next status will be 'closed'
The resolution will be deleted. Next status will be 'new'
Author


E-mail address and user name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.