Django

Code

Ticket #7759 (closed: fixed)

Opened 5 months ago

Last modified 5 months ago

QuerySet count method is wrong while cached results filling

Reported by: kcarnold Assigned to: mtredinnick
Milestone: 1.0 Component: Database layer (models, ORM)
Version: SVN Keywords: QuerySet count
Cc: Triage Stage: Accepted
Has patch: 0 Needs documentation: 0
Needs tests: 0 Patch needs improvement: 0

Description

Example: 

>>> Concept.objects.count()
288154L

>>> c=Concept.objects.all()
>>> for x in c:
...     print c.count()
...     break
... 
100

I can't think of any actual usage scenario that would hit this, but I realized it was a problem while reading the QuerySet code.

Attachments

Change History

07/15/08 11:52:45 changed by mtredinnick

  • owner changed from nobody to mtredinnick.
  • needs_better_patch changed.
  • needs_tests changed.
  • needs_docs changed.

Could have sworn I tested this and accounted for it at one point. Clearly it's regressed in one of my moments of madness.

07/15/08 21:30:05 changed by serialx

  • keywords set to QuerySet count.
  • stage changed from Unreviewed to Accepted.
  • milestone set to 1.0 beta.

Marking as accepted. AND 1.0 beta milestone.

07/16/08 18:03:24 changed by mtredinnick

  • milestone changed from 1.0 beta to 1.0.

07/16/08 18:17:29 changed by mtredinnick

  • status changed from new to closed.
  • resolution set to fixed.

(In [7938]) Fixed #7759 -- Fixed QuerySet?.count() when the results cache was only partially populated.

07/17/08 01:17:24 changed by Dave Smith <davewsmith@gmail.com>

  • status changed from closed to reopened.
  • resolution deleted.

On OSX Leopard (Python 2.5.1, SQLite3 3.4.0), r7938 introduces several test failures not present in r7937. 

Updated to revision 7938.
Macintosh-8:tests dws$ python runtests.py --settings=test-settings
======================================================================
FAIL: Doctest: regressiontests.queries.models.__test__.API_TESTS
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Library/Python/2.5/site-packages/django/test/_doctest.py", line 2180, in runTest
    raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for regressiontests.queries.models.__test__.API_TESTS
  File "/Users/dws/src/django_dev/tests/regressiontests/queries/models.py", line unknown line number, in API_TESTS

----------------------------------------------------------------------
File "/Users/dws/src/django_dev/tests/regressiontests/queries/models.py", line ?, in regressiontests.queries.models.__test__.API_TESTS
Failed example:
    for i, obj in enumerate(Number.objects.all()):
        obj.save()
        if i > 10: break
Exception raised:
    Traceback (most recent call last):
      File "/Library/Python/2.5/site-packages/django/test/_doctest.py", line 1267, in __run
        compileflags, 1) in test.globs
      File "<doctest regressiontests.queries.models.__test__.API_TESTS[217]>", line 2, in <module>
        obj.save()
      File "/Library/Python/2.5/site-packages/django/db/models/base.py", line 278, in save
        self.save_base()
      File "/Library/Python/2.5/site-packages/django/db/models/base.py", line 347, in save_base
        transaction.commit_unless_managed()
      File "/Library/Python/2.5/site-packages/django/db/transaction.py", line 140, in commit_unless_managed
        connection._commit()
      File "/Library/Python/2.5/site-packages/django/db/backends/__init__.py", line 20, in _commit
        return self.connection.commit()
    OperationalError: SQL logic error or missing database
----------------------------------------------------------------------
File "/Users/dws/src/django_dev/tests/regressiontests/queries/models.py", line ?, in regressiontests.queries.models.__test__.API_TESTS
Failed example:
    for obj in qs:
        qs.count() == count
        break
Expected:
    True
Got:
    False


======================================================================
FAIL: Doctest: regressiontests.select_related_regress.models.__test__.API_TESTS
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Library/Python/2.5/site-packages/django/test/_doctest.py", line 2180, in runTest
    raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for regressiontests.select_related_regress.models.__test__.API_TESTS
  File "/Users/dws/src/django_dev/tests/regressiontests/select_related_regress/models.py", line unknown line number, in API_TESTS

----------------------------------------------------------------------
File "/Users/dws/src/django_dev/tests/regressiontests/select_related_regress/models.py", line ?, in regressiontests.select_related_regress.models.__test__.API_TESTS
Failed example:
    b=Building.objects.create(name='101')
Exception raised:
    Traceback (most recent call last):
      File "/Library/Python/2.5/site-packages/django/test/_doctest.py", line 1267, in __run
        compileflags, 1) in test.globs
      File "<doctest regressiontests.select_related_regress.models.__test__.API_TESTS[0]>", line 1, in <module>
        b=Building.objects.create(name='101')
      File "/Library/Python/2.5/site-packages/django/db/models/manager.py", line 88, in create
        return self.get_query_set().create(**kwargs)
      File "/Library/Python/2.5/site-packages/django/db/models/query.py", line 313, in create
        obj.save()
      File "/Library/Python/2.5/site-packages/django/db/models/base.py", line 278, in save
        self.save_base()
      File "/Library/Python/2.5/site-packages/django/db/models/base.py", line 347, in save_base
        transaction.commit_unless_managed()
      File "/Library/Python/2.5/site-packages/django/db/transaction.py", line 140, in commit_unless_managed
        connection._commit()
      File "/Library/Python/2.5/site-packages/django/db/backends/__init__.py", line 20, in _commit
        return self.connection.commit()
    OperationalError: SQL logic error or missing database
----------------------------------------------------------------------
File "/Users/dws/src/django_dev/tests/regressiontests/select_related_regress/models.py", line ?, in regressiontests.select_related_regress.models.__test__.API_TESTS
Failed example:
    dev1=Device.objects.create(name="router", building=b)
Exception raised:
    Traceback (most recent call last):
      File "/Library/Python/2.5/site-packages/django/test/_doctest.py", line 1267, in __run
        compileflags, 1) in test.globs
      File "<doctest regressiontests.select_related_regress.models.__test__.API_TESTS[1]>", line 1, in <module>
        dev1=Device.objects.create(name="router", building=b)
    NameError: name 'b' is not defined
----------------------------------------------------------------------
File "/Users/dws/src/django_dev/tests/regressiontests/select_related_regress/models.py", line ?, in regressiontests.select_related_regress.models.__test__.API_TESTS
Failed example:
    dev2=Device.objects.create(name="switch", building=b)
Exception raised:
    Traceback (most recent call last):
      File "/Library/Python/2.5/site-packages/django/test/_doctest.py", line 1267, in __run
        compileflags, 1) in test.globs
      File "<doctest regressiontests.select_related_regress.models.__test__.API_TESTS[2]>", line 1, in <module>
        dev2=Device.objects.create(name="switch", building=b)
    NameError: name 'b' is not defined
----------------------------------------------------------------------
File "/Users/dws/src/django_dev/tests/regressiontests/select_related_regress/models.py", line ?, in regressiontests.select_related_regress.models.__test__.API_TESTS
Failed example:
    dev3=Device.objects.create(name="server", building=b)
Exception raised:
    Traceback (most recent call last):
      File "/Library/Python/2.5/site-packages/django/test/_doctest.py", line 1267, in __run
        compileflags, 1) in test.globs
      File "<doctest regressiontests.select_related_regress.models.__test__.API_TESTS[3]>", line 1, in <module>
        dev3=Device.objects.create(name="server", building=b)
    NameError: name 'b' is not defined
----------------------------------------------------------------------
File "/Users/dws/src/django_dev/tests/regressiontests/select_related_regress/models.py", line ?, in regressiontests.select_related_regress.models.__test__.API_TESTS
Failed example:
    port1=Port.objects.create(port_number='4',device=dev1)
Exception raised:
    Traceback (most recent call last):
      File "/Library/Python/2.5/site-packages/django/test/_doctest.py", line 1267, in __run
        compileflags, 1) in test.globs
      File "<doctest regressiontests.select_related_regress.models.__test__.API_TESTS[4]>", line 1, in <module>
        port1=Port.objects.create(port_number='4',device=dev1)
    NameError: name 'dev1' is not defined
----------------------------------------------------------------------
File "/Users/dws/src/django_dev/tests/regressiontests/select_related_regress/models.py", line ?, in regressiontests.select_related_regress.models.__test__.API_TESTS
Failed example:
    port2=Port.objects.create(port_number='7',device=dev2)
Exception raised:
    Traceback (most recent call last):
      File "/Library/Python/2.5/site-packages/django/test/_doctest.py", line 1267, in __run
        compileflags, 1) in test.globs
      File "<doctest regressiontests.select_related_regress.models.__test__.API_TESTS[5]>", line 1, in <module>
        port2=Port.objects.create(port_number='7',device=dev2)
    NameError: name 'dev2' is not defined
----------------------------------------------------------------------
File "/Users/dws/src/django_dev/tests/regressiontests/select_related_regress/models.py", line ?, in regressiontests.select_related_regress.models.__test__.API_TESTS
Failed example:
    port3=Port.objects.create(port_number='1',device=dev3)
Exception raised:
    Traceback (most recent call last):
      File "/Library/Python/2.5/site-packages/django/test/_doctest.py", line 1267, in __run
        compileflags, 1) in test.globs
      File "<doctest regressiontests.select_related_regress.models.__test__.API_TESTS[6]>", line 1, in <module>
        port3=Port.objects.create(port_number='1',device=dev3)
    NameError: name 'dev3' is not defined
----------------------------------------------------------------------
File "/Users/dws/src/django_dev/tests/regressiontests/select_related_regress/models.py", line ?, in regressiontests.select_related_regress.models.__test__.API_TESTS
Failed example:
    c1=Connection.objects.create(start=port1, end=port2)
Exception raised:
    Traceback (most recent call last):
      File "/Library/Python/2.5/site-packages/django/test/_doctest.py", line 1267, in __run
        compileflags, 1) in test.globs
      File "<doctest regressiontests.select_related_regress.models.__test__.API_TESTS[7]>", line 1, in <module>
        c1=Connection.objects.create(start=port1, end=port2)
    NameError: name 'port1' is not defined
----------------------------------------------------------------------
File "/Users/dws/src/django_dev/tests/regressiontests/select_related_regress/models.py", line ?, in regressiontests.select_related_regress.models.__test__.API_TESTS
Failed example:
    c2=Connection.objects.create(start=port2, end=port3)
Exception raised:
    Traceback (most recent call last):
      File "/Library/Python/2.5/site-packages/django/test/_doctest.py", line 1267, in __run
        compileflags, 1) in test.globs
      File "<doctest regressiontests.select_related_regress.models.__test__.API_TESTS[8]>", line 1, in <module>
        c2=Connection.objects.create(start=port2, end=port3)
    NameError: name 'port2' is not defined
----------------------------------------------------------------------
File "/Users/dws/src/django_dev/tests/regressiontests/select_related_regress/models.py", line ?, in regressiontests.select_related_regress.models.__test__.API_TESTS
Failed example:
    connections=Connection.objects.filter(start__device__building=b, end__device__building=b).order_by('id')
Exception raised:
    Traceback (most recent call last):
      File "/Library/Python/2.5/site-packages/django/test/_doctest.py", line 1267, in __run
        compileflags, 1) in test.globs
      File "<doctest regressiontests.select_related_regress.models.__test__.API_TESTS[9]>", line 1, in <module>
        connections=Connection.objects.filter(start__device__building=b, end__device__building=b).order_by('id')
    NameError: name 'b' is not defined
----------------------------------------------------------------------
File "/Users/dws/src/django_dev/tests/regressiontests/select_related_regress/models.py", line ?, in regressiontests.select_related_regress.models.__test__.API_TESTS
Failed example:
    [(c.id, unicode(c.start), unicode(c.end)) for c in connections]
Exception raised:
    Traceback (most recent call last):
      File "/Library/Python/2.5/site-packages/django/test/_doctest.py", line 1267, in __run
        compileflags, 1) in test.globs
      File "<doctest regressiontests.select_related_regress.models.__test__.API_TESTS[10]>", line 1, in <module>
        [(c.id, unicode(c.start), unicode(c.end)) for c in connections]
    NameError: name 'connections' is not defined
----------------------------------------------------------------------
File "/Users/dws/src/django_dev/tests/regressiontests/select_related_regress/models.py", line ?, in regressiontests.select_related_regress.models.__test__.API_TESTS
Failed example:
    connections=Connection.objects.filter(start__device__building=b, end__device__building=b).select_related().order_by('id')
Exception raised:
    Traceback (most recent call last):
      File "/Library/Python/2.5/site-packages/django/test/_doctest.py", line 1267, in __run
        compileflags, 1) in test.globs
      File "<doctest regressiontests.select_related_regress.models.__test__.API_TESTS[11]>", line 1, in <module>
        connections=Connection.objects.filter(start__device__building=b, end__device__building=b).select_related().order_by('id')
    NameError: name 'b' is not defined
----------------------------------------------------------------------
File "/Users/dws/src/django_dev/tests/regressiontests/select_related_regress/models.py", line ?, in regressiontests.select_related_regress.models.__test__.API_TESTS
Failed example:
    [(c.id, unicode(c.start), unicode(c.end)) for c in connections]
Exception raised:
    Traceback (most recent call last):
      File "/Library/Python/2.5/site-packages/django/test/_doctest.py", line 1267, in __run
        compileflags, 1) in test.globs
      File "<doctest regressiontests.select_related_regress.models.__test__.API_TESTS[12]>", line 1, in <module>
        [(c.id, unicode(c.start), unicode(c.end)) for c in connections]
    NameError: name 'connections' is not defined
----------------------------------------------------------------------
File "/Users/dws/src/django_dev/tests/regressiontests/select_related_regress/models.py", line ?, in regressiontests.select_related_regress.models.__test__.API_TESTS
Failed example:
    connections.query.count_active_tables()
Exception raised:
    Traceback (most recent call last):
      File "/Library/Python/2.5/site-packages/django/test/_doctest.py", line 1267, in __run
        compileflags, 1) in test.globs
      File "<doctest regressiontests.select_related_regress.models.__test__.API_TESTS[13]>", line 1, in <module>
        connections.query.count_active_tables()
    NameError: name 'connections' is not defined


----------------------------------------------------------------------
Ran 363 tests in 212.327s

FAILED (failures=2)

This is on top of the test failure I reported against #7411, which also appears in the failures above.

(follow-up: ↓ 7 ) 07/17/08 12:11:32 changed by mtredinnick

Can anybody confirm this (I can't)?

(in reply to: ↑ 6 ) 07/17/08 13:21:18 changed by Karen Tracey <kmtracey@gmail.com>

Replying to mtredinnick:

Can anybody confirm this (I can't)?

I can't but then I don't have a Leopard box, alas. However something looks a little fishy in the failures I think. There's a mixture of locations for Django code in the tracebacks. The tests are running from, for example: 

File "/Users/dws/src/django_dev/tests/regressiontests/select_related_regress/models.py", line ?, in regressiontests.select_related_regress.models.__test__.API_TESTS

but the traceback paths are completely different, for example: 

File "/Library/Python/2.5/site-packages/django/test/_doctest.py", line 1267, in __run

I'd suspect a missing or incorrect PYTHONPATH setting when the tests were run, resulting in the most recent test suite running against old code?

07/17/08 13:42:22 changed by jkocherhans

  • status changed from reopened to closed.
  • resolution set to fixed.

I'm pretty sure there's something wrong with your environment, Dave. See Karen's comment. Everything works fine for me on Leopard with [7941]. OS X 10.5.4 Python 2.5.1, SQLite 3.4.0

07/17/08 14:43:10 changed by Dave Smith <davewsmith@gmail.com>

Oh, lovely. Let's assume it's me, then. Probably an old pysqlite2. I'll try updating.

I also have access to a virgin Leopard install, and will try to replicate there later today.

07/17/08 15:22:20 changed by Dave Smith <davewsmith@gmail.com>

Yup. Karen nailed it. After getting things running on a fresh Leopard install, I doubled back and re-read the bug. Ooops.

Sorry for the disruption.

07/17/08 15:26:13 changed by Dave Smith <davewsmith@gmail.com>

By the way, the exact problem was that I had site-packaged/django symlinked to /Users/dws/src/django_src/, but had just checked out a separate ~/src/django_dev to patch in a change to verify. So new tests against (slightly) old code was exactly the problem.

Add/Change #7759 (QuerySet count method is wrong while cached results filling)




Change Properties
Action