Opened 16 years ago

Closed 16 years ago

Last modified 13 years ago

#7759 closed (fixed)

QuerySet count method is wrong while cached results filling

Reported by: Kenneth Arnold Owned by: Malcolm Tredinnick
Component: Database layer (models, ORM) Version: dev
Severity: Keywords: QuerySet count
Cc: Triage Stage: Accepted
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

Example:

>>> Concept.objects.count()
288154L

>>> c=Concept.objects.all()
>>> for x in c:
...     print c.count()
...     break
... 
100

I can't think of any actual usage scenario that would hit this, but I realized it was a problem while reading the QuerySet code.

Change History (12)

comment:1 by Malcolm Tredinnick, 16 years ago

Owner: changed from nobody to Malcolm Tredinnick

Could have sworn I tested this and accounted for it at one point. Clearly it's regressed in one of my moments of madness.

comment:2 by Sung-jin Hong, 16 years ago

Keywords: QuerySet count added
milestone: → 1.0 beta
Triage Stage: Unreviewed → Accepted

Marking as accepted. AND 1.0 beta milestone.

comment:3 by Malcolm Tredinnick, 16 years ago

milestone: 1.0 beta → 1.0

comment:4 by Malcolm Tredinnick, 16 years ago

Resolution: → fixed
Status: new → closed

(In [7938]) Fixed #7759 -- Fixed QuerySet.count() when the results cache was only partially
populated.

comment:5 by Dave Smith <davewsmith@…>, 16 years ago

Resolution: fixed
Status: closed → reopened

On OSX Leopard (Python 2.5.1, SQLite3 3.4.0), r7938 introduces several test failures not present in r7937.

Updated to revision 7938.
Macintosh-8:tests dws$ python runtests.py --settings=test-settings
======================================================================
FAIL: Doctest: regressiontests.queries.models.__test__.API_TESTS
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Library/Python/2.5/site-packages/django/test/_doctest.py", line 2180, in runTest
    raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for regressiontests.queries.models.__test__.API_TESTS
  File "/Users/dws/src/django_dev/tests/regressiontests/queries/models.py", line unknown line number, in API_TESTS

----------------------------------------------------------------------
File "/Users/dws/src/django_dev/tests/regressiontests/queries/models.py", line ?, in regressiontests.queries.models.__test__.API_TESTS
Failed example:
    for i, obj in enumerate(Number.objects.all()):
        obj.save()
        if i > 10: break
Exception raised:
    Traceback (most recent call last):
      File "/Library/Python/2.5/site-packages/django/test/_doctest.py", line 1267, in __run
        compileflags, 1) in test.globs
      File "<doctest regressiontests.queries.models.__test__.API_TESTS[217]>", line 2, in <module>
        obj.save()
      File "/Library/Python/2.5/site-packages/django/db/models/base.py", line 278, in save
        self.save_base()
      File "/Library/Python/2.5/site-packages/django/db/models/base.py", line 347, in save_base
        transaction.commit_unless_managed()
      File "/Library/Python/2.5/site-packages/django/db/transaction.py", line 140, in commit_unless_managed
        connection._commit()
      File "/Library/Python/2.5/site-packages/django/db/backends/__init__.py", line 20, in _commit
        return self.connection.commit()
    OperationalError: SQL logic error or missing database
----------------------------------------------------------------------
File "/Users/dws/src/django_dev/tests/regressiontests/queries/models.py", line ?, in regressiontests.queries.models.__test__.API_TESTS
Failed example:
    for obj in qs:
        qs.count() == count
        break
Expected:
    True
Got:
    False


======================================================================
FAIL: Doctest: regressiontests.select_related_regress.models.__test__.API_TESTS
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Library/Python/2.5/site-packages/django/test/_doctest.py", line 2180, in runTest
    raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for regressiontests.select_related_regress.models.__test__.API_TESTS
  File "/Users/dws/src/django_dev/tests/regressiontests/select_related_regress/models.py", line unknown line number, in API_TESTS

----------------------------------------------------------------------
File "/Users/dws/src/django_dev/tests/regressiontests/select_related_regress/models.py", line ?, in regressiontests.select_related_regress.models.__test__.API_TESTS
Failed example:
    b=Building.objects.create(name='101')
Exception raised:
    Traceback (most recent call last):
      File "/Library/Python/2.5/site-packages/django/test/_doctest.py", line 1267, in __run
        compileflags, 1) in test.globs
      File "<doctest regressiontests.select_related_regress.models.__test__.API_TESTS[0]>", line 1, in <module>
        b=Building.objects.create(name='101')
      File "/Library/Python/2.5/site-packages/django/db/models/manager.py", line 88, in create
        return self.get_query_set().create(**kwargs)
      File "/Library/Python/2.5/site-packages/django/db/models/query.py", line 313, in create
        obj.save()
      File "/Library/Python/2.5/site-packages/django/db/models/base.py", line 278, in save
        self.save_base()
      File "/Library/Python/2.5/site-packages/django/db/models/base.py", line 347, in save_base
        transaction.commit_unless_managed()
      File "/Library/Python/2.5/site-packages/django/db/transaction.py", line 140, in commit_unless_managed
        connection._commit()
      File "/Library/Python/2.5/site-packages/django/db/backends/__init__.py", line 20, in _commit
        return self.connection.commit()
    OperationalError: SQL logic error or missing database
----------------------------------------------------------------------
File "/Users/dws/src/django_dev/tests/regressiontests/select_related_regress/models.py", line ?, in regressiontests.select_related_regress.models.__test__.API_TESTS
Failed example:
    dev1=Device.objects.create(name="router", building=b)
Exception raised:
    Traceback (most recent call last):
      File "/Library/Python/2.5/site-packages/django/test/_doctest.py", line 1267, in __run
        compileflags, 1) in test.globs
      File "<doctest regressiontests.select_related_regress.models.__test__.API_TESTS[1]>", line 1, in <module>
        dev1=Device.objects.create(name="router", building=b)
    NameError: name 'b' is not defined
----------------------------------------------------------------------
File "/Users/dws/src/django_dev/tests/regressiontests/select_related_regress/models.py", line ?, in regressiontests.select_related_regress.models.__test__.API_TESTS
Failed example:
    dev2=Device.objects.create(name="switch", building=b)
Exception raised:
    Traceback (most recent call last):
      File "/Library/Python/2.5/site-packages/django/test/_doctest.py", line 1267, in __run
        compileflags, 1) in test.globs
      File "<doctest regressiontests.select_related_regress.models.__test__.API_TESTS[2]>", line 1, in <module>
        dev2=Device.objects.create(name="switch", building=b)
    NameError: name 'b' is not defined
----------------------------------------------------------------------
File "/Users/dws/src/django_dev/tests/regressiontests/select_related_regress/models.py", line ?, in regressiontests.select_related_regress.models.__test__.API_TESTS
Failed example:
    dev3=Device.objects.create(name="server", building=b)
Exception raised:
    Traceback (most recent call last):
      File "/Library/Python/2.5/site-packages/django/test/_doctest.py", line 1267, in __run
        compileflags, 1) in test.globs
      File "<doctest regressiontests.select_related_regress.models.__test__.API_TESTS[3]>", line 1, in <module>
        dev3=Device.objects.create(name="server", building=b)
    NameError: name 'b' is not defined
----------------------------------------------------------------------
File "/Users/dws/src/django_dev/tests/regressiontests/select_related_regress/models.py", line ?, in regressiontests.select_related_regress.models.__test__.API_TESTS
Failed example:
    port1=Port.objects.create(port_number='4',device=dev1)
Exception raised:
    Traceback (most recent call last):
      File "/Library/Python/2.5/site-packages/django/test/_doctest.py", line 1267, in __run
        compileflags, 1) in test.globs
      File "<doctest regressiontests.select_related_regress.models.__test__.API_TESTS[4]>", line 1, in <module>
        port1=Port.objects.create(port_number='4',device=dev1)
    NameError: name 'dev1' is not defined
----------------------------------------------------------------------
File "/Users/dws/src/django_dev/tests/regressiontests/select_related_regress/models.py", line ?, in regressiontests.select_related_regress.models.__test__.API_TESTS
Failed example:
    port2=Port.objects.create(port_number='7',device=dev2)
Exception raised:
    Traceback (most recent call last):
      File "/Library/Python/2.5/site-packages/django/test/_doctest.py", line 1267, in __run
        compileflags, 1) in test.globs
      File "<doctest regressiontests.select_related_regress.models.__test__.API_TESTS[5]>", line 1, in <module>
        port2=Port.objects.create(port_number='7',device=dev2)
    NameError: name 'dev2' is not defined
----------------------------------------------------------------------
File "/Users/dws/src/django_dev/tests/regressiontests/select_related_regress/models.py", line ?, in regressiontests.select_related_regress.models.__test__.API_TESTS
Failed example:
    port3=Port.objects.create(port_number='1',device=dev3)
Exception raised:
    Traceback (most recent call last):
      File "/Library/Python/2.5/site-packages/django/test/_doctest.py", line 1267, in __run
        compileflags, 1) in test.globs
      File "<doctest regressiontests.select_related_regress.models.__test__.API_TESTS[6]>", line 1, in <module>
        port3=Port.objects.create(port_number='1',device=dev3)
    NameError: name 'dev3' is not defined
----------------------------------------------------------------------
File "/Users/dws/src/django_dev/tests/regressiontests/select_related_regress/models.py", line ?, in regressiontests.select_related_regress.models.__test__.API_TESTS
Failed example:
    c1=Connection.objects.create(start=port1, end=port2)
Exception raised:
    Traceback (most recent call last):
      File "/Library/Python/2.5/site-packages/django/test/_doctest.py", line 1267, in __run
        compileflags, 1) in test.globs
      File "<doctest regressiontests.select_related_regress.models.__test__.API_TESTS[7]>", line 1, in <module>
        c1=Connection.objects.create(start=port1, end=port2)
    NameError: name 'port1' is not defined
----------------------------------------------------------------------
File "/Users/dws/src/django_dev/tests/regressiontests/select_related_regress/models.py", line ?, in regressiontests.select_related_regress.models.__test__.API_TESTS
Failed example:
    c2=Connection.objects.create(start=port2, end=port3)
Exception raised:
    Traceback (most recent call last):
      File "/Library/Python/2.5/site-packages/django/test/_doctest.py", line 1267, in __run
        compileflags, 1) in test.globs
      File "<doctest regressiontests.select_related_regress.models.__test__.API_TESTS[8]>", line 1, in <module>
        c2=Connection.objects.create(start=port2, end=port3)
    NameError: name 'port2' is not defined
----------------------------------------------------------------------
File "/Users/dws/src/django_dev/tests/regressiontests/select_related_regress/models.py", line ?, in regressiontests.select_related_regress.models.__test__.API_TESTS
Failed example:
    connections=Connection.objects.filter(start__device__building=b, end__device__building=b).order_by('id')
Exception raised:
    Traceback (most recent call last):
      File "/Library/Python/2.5/site-packages/django/test/_doctest.py", line 1267, in __run
        compileflags, 1) in test.globs
      File "<doctest regressiontests.select_related_regress.models.__test__.API_TESTS[9]>", line 1, in <module>
        connections=Connection.objects.filter(start__device__building=b, end__device__building=b).order_by('id')
    NameError: name 'b' is not defined
----------------------------------------------------------------------
File "/Users/dws/src/django_dev/tests/regressiontests/select_related_regress/models.py", line ?, in regressiontests.select_related_regress.models.__test__.API_TESTS
Failed example:
    [(c.id, unicode(c.start), unicode(c.end)) for c in connections]
Exception raised:
    Traceback (most recent call last):
      File "/Library/Python/2.5/site-packages/django/test/_doctest.py", line 1267, in __run
        compileflags, 1) in test.globs
      File "<doctest regressiontests.select_related_regress.models.__test__.API_TESTS[10]>", line 1, in <module>
        [(c.id, unicode(c.start), unicode(c.end)) for c in connections]
    NameError: name 'connections' is not defined
----------------------------------------------------------------------
File "/Users/dws/src/django_dev/tests/regressiontests/select_related_regress/models.py", line ?, in regressiontests.select_related_regress.models.__test__.API_TESTS
Failed example:
    connections=Connection.objects.filter(start__device__building=b, end__device__building=b).select_related().order_by('id')
Exception raised:
    Traceback (most recent call last):
      File "/Library/Python/2.5/site-packages/django/test/_doctest.py", line 1267, in __run
        compileflags, 1) in test.globs
      File "<doctest regressiontests.select_related_regress.models.__test__.API_TESTS[11]>", line 1, in <module>
        connections=Connection.objects.filter(start__device__building=b, end__device__building=b).select_related().order_by('id')
    NameError: name 'b' is not defined
----------------------------------------------------------------------
File "/Users/dws/src/django_dev/tests/regressiontests/select_related_regress/models.py", line ?, in regressiontests.select_related_regress.models.__test__.API_TESTS
Failed example:
    [(c.id, unicode(c.start), unicode(c.end)) for c in connections]
Exception raised:
    Traceback (most recent call last):
      File "/Library/Python/2.5/site-packages/django/test/_doctest.py", line 1267, in __run
        compileflags, 1) in test.globs
      File "<doctest regressiontests.select_related_regress.models.__test__.API_TESTS[12]>", line 1, in <module>
        [(c.id, unicode(c.start), unicode(c.end)) for c in connections]
    NameError: name 'connections' is not defined
----------------------------------------------------------------------
File "/Users/dws/src/django_dev/tests/regressiontests/select_related_regress/models.py", line ?, in regressiontests.select_related_regress.models.__test__.API_TESTS
Failed example:
    connections.query.count_active_tables()
Exception raised:
    Traceback (most recent call last):
      File "/Library/Python/2.5/site-packages/django/test/_doctest.py", line 1267, in __run
        compileflags, 1) in test.globs
      File "<doctest regressiontests.select_related_regress.models.__test__.API_TESTS[13]>", line 1, in <module>
        connections.query.count_active_tables()
    NameError: name 'connections' is not defined


----------------------------------------------------------------------
Ran 363 tests in 212.327s

FAILED (failures=2)

This is on top of the test failure I reported against #7411, which also appears in the failures above.

comment:6 by Malcolm Tredinnick, 16 years ago

Can anybody confirm this (I can't)?

in reply to:  6 comment:7 by Karen Tracey <kmtracey@…>, 16 years ago

Replying to mtredinnick:

Can anybody confirm this (I can't)?

I can't but then I don't have a Leopard box, alas. However something looks a little fishy in the failures I think. There's a mixture of locations for Django code in the tracebacks. The tests are running from, for example:

File "/Users/dws/src/django_dev/tests/regressiontests/select_related_regress/models.py", line ?, in regressiontests.select_related_regress.models.__test__.API_TESTS

but the traceback paths are completely different, for example:

File "/Library/Python/2.5/site-packages/django/test/_doctest.py", line 1267, in __run

I'd suspect a missing or incorrect PYTHONPATH setting when the tests were run, resulting in the most recent test suite running against old code?

comment:8 by jkocherhans, 16 years ago

Resolution: → fixed
Status: reopened → closed

I'm pretty sure there's something wrong with your environment, Dave. See Karen's comment. Everything works fine for me on Leopard with [7941]. OS X 10.5.4 Python 2.5.1, SQLite 3.4.0

comment:9 by Dave Smith <davewsmith@…>, 16 years ago

Oh, lovely. Let's assume it's me, then. Probably an old pysqlite2. I'll try updating.

I also have access to a virgin Leopard install, and will try to replicate there later today.

comment:10 by Dave Smith <davewsmith@…>, 16 years ago

Yup. Karen nailed it. After getting things running on a fresh Leopard install, I doubled back and re-read the bug. Ooops.

Sorry for the disruption.

comment:11 by Dave Smith <davewsmith@…>, 16 years ago

By the way, the exact problem was that I had site-packaged/django symlinked to /Users/dws/src/django_src/, but had just checked out a separate ~/src/django_dev to patch in a change to verify. So new tests against (slightly) old code was exactly the problem.

comment:12 by Jacob, 13 years ago

milestone: 1.0

Milestone 1.0 deleted

Note: See TracTickets for help on using tickets.
Back to Top