queries test failure on Python 2.3

[Karen Tracey <kmtracey@…>]

Running the test suite on Python 2.3, I get a test failure in the queries regression test:

D:\u\kmt\django\trunk\tests>d:\bin\Python2.3.5\python.exe runtests.py --settings=testsettings queries
======================================================================
FAIL: Doctest: regressiontests.queries.models.__test__.API_TESTS
----------------------------------------------------------------------
Traceback (most recent call last):
  File "d:\u\kmt\django\trunk\django\test\_doctest.py", line 2180, in runTest
    raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for regressiontests.queries.models.__test__.API_TESTS
  File "D:\u\kmt\django\trunk\tests\regressiontests\queries\models.py", line unknown line number, in API_TESTS

----------------------------------------------------------------------
File "D:\u\kmt\django\trunk\tests\regressiontests\queries\models.py", line ?, in regressiontests.queries.models.__test__.API_TESTS
Failed example:
    LoopX.objects.all()
Expected:
    Traceback (most recent call last):
    ...
    FieldError: Infinite loop caused by ordering.
Got:
    []
----------------------------------------------------------------------
File "D:\u\kmt\django\trunk\tests\regressiontests\queries\models.py", line ?, in regressiontests.queries.models.__test__.API_TESTS
Failed example:
    LoopZ.objects.all()
Expected:
    Traceback (most recent call last):
    ...
    FieldError: Infinite loop caused by ordering.
Got:
    []


----------------------------------------------------------------------
Ran 1 test in 0.591s

FAILED (failures=1)

I believe it's caused by this Python bug: ​http://bugs.python.org/issue1242657 which caused exceptions raised by __len__ to be silently swallowed. A traceback from when the exception is successfully raised:

>>> LoopX.objects.all()
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "d:\u\kmt\django\trunk\django\db\models\query.py", line 143, in __repr__
    return repr(list(self))
  File "d:\u\kmt\django\trunk\django\db\models\query.py", line 155, in __len__
    self._result_cache.extend(list(self._iter))
  File "d:\u\kmt\django\trunk\django\db\models\query.py", line 268, in iterator
    for row in self.query.results_iter():
  File "d:\u\kmt\django\trunk\django\db\models\sql\query.py", line 204, in results_iter
    for rows in self.execute_sql(MULTI):
  File "d:\u\kmt\django\trunk\django\db\models\sql\query.py", line 1598, in execute_sql
    sql, params = self.as_sql()
  File "d:\u\kmt\django\trunk\django\db\models\sql\query.py", line 251, in as_sql
    ordering = self.get_ordering()
  File "d:\u\kmt\django\trunk\django\db\models\sql\query.py", line 610, in get_ordering
    self.model._meta, default_order=asc):
  File "d:\u\kmt\django\trunk\django\db\models\sql\query.py", line 663, in find_ordering_name
    order, already_seen))
  File "d:\u\kmt\django\trunk\django\db\models\sql\query.py", line 663, in find_ordering_name
    order, already_seen))
  File "d:\u\kmt\django\trunk\django\db\models\sql\query.py", line 657, in find_ordering_name
    raise FieldError('Infinite loop caused by ordering.')
FieldError: Infinite loop caused by ordering.

shows how __len__ swallowing exceptions would cause this test failure.

Don't know that it is worth fixing (apparently you'll just get empty lists instead of exceptions in cases where this is hit) but figured it was at least worth reporting in case anyone else runs across it. (I didn't find any likely match in my tracker search.)

[Malcolm Tredinnick]

Argh! Another __len__-related problem. I'm beginning to really hate its error handling (although, to be fair, if you did something bad in __len__ in, say, Python 1.5, the wheels really fell off the wagon, so the current safe behaviour is probably preferable).

I'll try to fix this. It's actually important.

[Karen Tracey <kmtracey@…>]

Replying to mtredinnick:

I'll try to fix this. It's actually important.

Python 2.3 is still that important, then? I know the answer was a resounding 'yes' when someone asked on one of the lists about 8 months ago...but that was 8 months ago. The problem only occurs on 2.3, not 2.4 or 2.5.

[Malcolm Tredinnick]

Yep. Python 2.3 is fully supported in Django 1.0.

[Malcolm Tredinnick]

Looks like there's not really a lot we can do about this. Arranging to not do anything in __len__ that might cause an exception is something I tried to do months ago for other reasons and the performance effects were terrible. So penalising python 2.4 and 2.5 for what is primarily a diagnostic message in this case isn't really worth it. Django works correctly if you don't make an error in Python 2.3 and I can live with some erroneous cases perform differently than they do in 2.4 and 2.5.

I've adjusted the test suite to not fail on those tests in 2.3, but that's as far as I'm going to go here. Not entirely satisfying, but we're fighting Python bugs and the way list() and friends are implemented internally in Python.

[Malcolm Tredinnick]

(In [7939]) Fixed #7786 -- Removed some tests from running when using Python 2.3.

The problem being "hidden" here is not serious. It won't affect correct code
and only gives a different failure mode for incorrect code. The moral is: don't
write incorrect code.

[Russell Keith-Magee]

Replying to Karen Tracey <kmtracey@gmail.com>:

Replying to mtredinnick:

I'll try to fix this. It's actually important.

Python 2.3 is still that important, then? I know the answer was a resounding 'yes' when someone asked on one of the lists about 8 months ago...but that was 8 months ago. The problem only occurs on 2.3, not 2.4 or 2.5.

Elaborating on Malcolm's comments: Yes, Django 1.0 will support Python 2.3 - for two reasons:

• RHEL4, Debian Sarge and OS X Tiger all provide Python 2.3. These are not entirely uncommon platform amongst conservative hosting providers (or older desktop installs)
• Supporting Python implementations other than CPython (i.e., Jython, PyPy) is on the list for v1.0. Keeping the code compatible with CPython 2.3 is the easiest way to maintain support for these platforms.

[Karen Tracey <kmtracey@…>]

Replying to russellm:

Thanks for the clarification. I asked mainly to make sure I wasn't wasting people's time reporting 2.3 problems; I thought it possible things had changed since the issue was raised on one of the lists more than six months ago, but I guess not. So, I'll keep testing on it and report problems as I find them (this and the other I opened recently are the only ones I hit on the current codebase).

[Karen Tracey <kmtracey@…>]

FYI this Python behavior of swallowing exceptions raised by __len__ has returned in the current (Beta 2) 2.6 release. I haven't tracked down yet whether that is intentional or an oversight on Python's part.

[jcrocholl]

Replying to Karen Tracey <kmtracey@gmail.com>:

FYI this Python behavior of swallowing exceptions raised by __len__ has returned in the current (Beta 2) 2.6 release. I haven't tracked down yet whether that is intentional or an oversight on Python's part.

The problem also exists in Python 2.6beta3.

[Karen Tracey]

Replying to jcrocholl:

The problem also exists in Python 2.6beta3.

Yes, the Python bug cited in the description has been re-opened for 2.6/3.1 (not sure why that's not 3.0, but the bug tracker has 3.1) but not fixed as of yet. Not sure at this point if Django should skip this test when running on 2.6 (as it does for 2.3) or leave things as-is in the hope that Python will re-fix bug #1242657 before 2.6 is actually released.

[Malcolm Tredinnick]

This behaviour isn't showstopper-critical. If that feature of Django stopped working, you wouldn't receive a particular error when you made a mistake. That means only people doing bad things will be harmed and they shouldn't be doing bad things in the first place. So I'm happy to just not run that test on Python 2.6, at least for now.

[Malcolm Tredinnick]

(In [8570]) Hid a few QuerySet regression tests from Python 2.6 due to a bug in the the
Python beta releases. Failures there mean that incorrect code won't raise an
error, but it's otherwise harmless (correct code still runs correctly).

Fixed #7786.

[Jacob]

Milestone 1.0 deleted