Opened 4 years ago

Closed 4 years ago

#25894 closed Bug (fixed)

Evaluation of zero-length slices of queryset.values() fails

Reported by: Marc DEBUREAUX Owned by: Sergey Fedoseev
Component: Database layer (models, ORM) Version: 1.9
Severity: Release blocker Keywords:
Cc: marc@… Triage Stage: Accepted
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

When using a QuerySet returning no values without a whole model behind, the Paginator fails to display data from page.

Example OK:

queryset = Model.objects.filter(my_filter_return_nothing=value)
paginator = Paginator(queryset, 10)
list(paginator.page(1))
>>> []

Example KO:

queryset = Model.objects.values('whatever_field').filter(my_filter_return_nothing=value)
paginator = Paginator(queryset, 10)
list(paginator.page(1))
>>>
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-38-56e5635092e7> in <module>()
----> 1 list(paginator.page(1))

\lib\site-packages\django\core\paginator.py in __len__(self)
    115
    116     def __len__(self):
--> 117         return len(self.object_list)
    118
    119     def __getitem__(self, index):

\lib\site-packages\django\db\models\query.py in __len__(self)
    238
    239     def __len__(self):
--> 240         self._fetch_all()
    241         return len(self._result_cache)
    242

\lib\site-packages\django\db\models\query.py in _fetch_all(self)
   1072     def _fetch_all(self):
   1073         if self._result_cache is None:
-> 1074             self._result_cache = list(self.iterator())
   1075         if self._prefetch_related_lookups and not self._prefetch_done:
   1076             self._prefetch_related_objects()

\lib\site-packages\django\db\models\query.py in __iter__(self)
    110         names = extra_names + field_names + annotation_names
    111
--> 112         for row in compiler.results_iter():
    113             yield dict(zip(names, row))
    114

\lib\site-packages\django\db\models\sql\compiler.py in results_iter(self, results)
    805         if results is None:
    806             results = self.execute_sql(MULTI)
--> 807         fields = [s[0] for s in self.select[0:self.col_count]]
    808         converters = self.get_converters(fields)
    809         for rows in results:

AttributeError: 'SQLCompiler' object has no attribute 'col_count'

Thanks.

Change History (20)

comment:1 Changed 4 years ago by Marc DEBUREAUX

Cc: marc@… added
Severity: NormalRelease blocker

comment:2 Changed 4 years ago by Tim Graham

Is this a regression in Django 1.9? Can you provide a test for tests/pagination/tests.py?

comment:3 Changed 4 years ago by Sergey Fedoseev

I can reproduce it on 1.9 and can't reproduce it on 1.8.7.

comment:4 Changed 4 years ago by Tim Graham

Triage Stage: UnreviewedAccepted

comment:5 Changed 4 years ago by Sergey Fedoseev

Owner: changed from nobody to Sergey Fedoseev
Status: newassigned

comment:6 Changed 4 years ago by Sergey Fedoseev

Bisected to afe0bb7b13bb8dc4370f32225238012c873b0ee3.

Minimal example to reproduce:

from django.contrib.contenttypes.models import ContentType
list(ContentType.objects.values('model')[0:0])

comment:7 Changed 4 years ago by Marc DEBUREAUX

# django monkey patch
# https://code.djangoproject.com/ticket/25894
try:

from django.db.models.sql.compiler import SQLCompiler

def _results_iter(self, results=None):

if not hasattr(self, 'col_count'):

self.col_count = 0

if not self.select:

self.select = []

return self._results_iter(results)

SQLCompiler._results_iter = SQLCompiler.results_iter
SQLCompiler.results_iter = _results_iter

except:

pass

Version 1, edited 4 years ago by Marc DEBUREAUX (previous) (next) (diff)

comment:8 Changed 4 years ago by Sergey Fedoseev

Has patch: set

comment:9 Changed 4 years ago by Marc DEBUREAUX

Component: Core (Other)Database layer (models, ORM)
Summary: Paginator fails with QuerySet returning no valuesEvaluation of zero-length slices of queryset.values() fails

comment:10 Changed 4 years ago by Tim Graham

Triage Stage: AcceptedReady for checkin

comment:11 Changed 4 years ago by Tim Graham <timograham@…>

Resolution: fixed
Status: assignedclosed

In 69b69f6d:

Fixed #25894 -- Fixed evaluation of zero-length slices of QuerySet.values().

comment:12 Changed 4 years ago by Tim Graham <timograham@…>

In 192d1eb:

[1.9.x] Fixed #25894 -- Fixed evaluation of zero-length slices of QuerySet.values().

Backport of 69b69f6d6085d80ccf36a3e999e9e5b98d580786 from master

comment:13 Changed 4 years ago by Tim Graham

Has patch: unset
Resolution: fixed
Status: closednew
Triage Stage: Ready for checkinAccepted

The new test doesn't pass on Oracle.

======================================================================
ERROR: test_zero_length_values_slicing (queries.tests.WeirdQuerysetSlicingTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/tim/code/django/tests/queries/tests.py", line 2435, in test_zero_length_values_slicing
    self.assertQuerysetEqual(Article.objects.values()[n:n], [])
  File "/home/tim/code/django/django/test/testcases.py", line 933, in assertQuerysetEqual
    items = six.moves.map(transform, qs)
  File "/home/tim/code/django/django/db/models/query.py", line 258, in __iter__
    self._fetch_all()
  File "/home/tim/code/django/django/db/models/query.py", line 1072, in _fetch_all
    self._result_cache = list(self.iterator())
  File "/home/tim/code/django/django/db/models/query.py", line 112, in __iter__
    for row in compiler.results_iter():
  File "/home/tim/code/django/django/db/models/sql/compiler.py", line 792, in results_iter
    fields = [s[0] for s in self.select[0:self.col_count]]
AttributeError: 'SQLCompiler' object has no attribute 'col_count'

comment:14 Changed 4 years ago by Sergey Fedoseev

Status: newassigned

comment:15 Changed 4 years ago by Sergey Fedoseev

Has patch: set

On oracle this seems to be broken since 1.8.
PR -- https://github.com/django/django/pull/5834

comment:16 Changed 4 years ago by Tim Graham <timograham@…>

In ed1bcf0:

Refs #25894 -- Fixed evaluation of zero-length slices of QuerySet.values() on Oracle.

comment:17 Changed 4 years ago by Tim Graham <timograham@…>

In 14146f63:

[1.9.x] Refs #25894 -- Fixed evaluation of zero-length slices of QuerySet.values() on Oracle.

Backport of ed1bcf05158acf4bf4e0189d477b6c762bd0133e from master

comment:18 Changed 4 years ago by Tim Graham

Resolution: fixed
Status: assignedclosed

comment:19 Changed 4 years ago by Marc DEBUREAUX

Resolution: fixed
Status: closednew

It doesn't work with the following case:

from django.contrib.contenttypes.models import ContentType
list(ContentType.objects.values('model')[0:0])

Here the patch I use based on the last commit. Does I miss something else?

try:
    def _set_limits(self, low=None, high=None):
        self._set_limits(low, high)
        if self.low_mark == self.high_mark:
            self.set_empty()

    from django.db.models.sql.query import Query
    Query._set_limits = Query.set_limits
    Query.set_limits = _set_limits
except:
    pass

comment:20 Changed 4 years ago by Marc DEBUREAUX

Resolution: fixed
Status: newclosed

Nevermind, I forgot to delete lines from compiler.

Note: See TracTickets for help on using tickets.
Back to Top