admin_tests failures when running Django tests in one natural fashion

[Malcolm Tredinnick]

This may be a PEBCAK issue, but I can't see it.

I run the Django tests as follows

> cd django.git/tests
> ./runtests.py --settings=settings3.py

(etc).

The admin tests introduced in [7876] are showing 11 failures (from 83 tests) now. They are all failing in the same way:

File "/usr/lib/python2.5/site-packages/django/core/management/__init__.py", line 251, in setup_environ
    project_module = __import__(project_name, {}, {}, [''])
ImportError: No module named tests

Note that site-packages/django/ is a symlink to my development codebase, so it is running the right code. I just don't want to have to mess around with PYTHONPATH settings all the time, so I use site-packages as Guido intended.

The PYTHONPATH modification set up in run_test() looks sensible when I print it out right after it's set.

/home/malcolm/BleedingEdge/Django/django.git/tests:/home/malcolm/BleedingEdge/Django/django.git

In fact, the full sys.path setting at the time of the problem looks pretty decent, too (I catch the import exception and dump this to a file, so this is the real value at the time of the crime):

['/home/malcolm/BleedingEdge/Django/django.git/tests/regressiontests', '/home/malcolm/BleedingEdge/Django/django.git/tests', '/home/malcolm/BleedingEdge/Django/django.git', '/usr/lib/python25.zip', '/usr/lib/python2.5', '/usr/lib/python2.5/plat-linux2', '/usr/lib/python2.5/lib-tk', '/usr/lib/python2.5/lib-dynload', '/usr/lib/python2.5/site-packages', '/usr/lib/python2.5/site-packages/Numeric', '/usr/lib/python2.5/site-packages/PIL', '/usr/lib/python2.5/site-packages/gst-0.10', '/usr/lib/python2.5/site-packages/gtk-2.0', '/home/malcolm/BleedingEdge/Django/django.git/tests/..']

This is all with a nice clean tree (nuked the .pyc files and everything not under source control) using [7882]. I am most confused, but don't really have time to debug it now, since this cropped up whilst checking something else. Maybe it's obvious and I'm not seeing it, but it's definitely repeatable.

[Russell Keith-Magee]

I'm not seeing this problem myself, but I can't say I can identify an obvious PEBCAK issue, either.

I have a Django checkout of [7882] in /Users/rkm/svn/django/trunk, symlinked to /Users/rkm/svn/django/live. The 'live' directory is in my PYTHONPATH. Using Python 2.5, and sqlite, I get no test failures. If I clear my PYTHONPATH and use a symlink from my SVN checkout into site-packages, I don't get failures, either. I don't know Karen Tracey's exact setup, but she reported no failures under Ubuntu or Windows.

The buildbots aren't reporting completely clean, but those failures have different causes:

• Sqlite, Python2.5 appears to have an intermittent caching issue with templates
• Sqlite, Python2.4 One platform doesn't have write permission for *.pyc files, so a test condition fails when looking for pyc files.

Postgres on Python 2.5 is building fine.

Some questions to help narrow this down:

• Which 11 tests are failing?
• Are the stack traces you are reporting coming from the test runner, or from the embedded manage.py/django-admin execution?
• Can you provide any more depth in the stack trace than the one line of ImportError?
• Are you seeing these failures as the usual ERROR case on test output, or is the test runner getting interrupted?
• Does the order of test execution matter? Do you see the failures if you run the whole suite, or just when you run the admin_scripts tests? If you run other tests before/after admin_scripts, do you still see failures? (The reason I ask this one is that [7877] was to fix a problem that was test order dependent. I think I've fixed all the ordering problems, but I could still be missing one)

[Malcolm Tredinnick]

I'll attach the full stderr output from running the admin_scripts tests. I see the same results when running the full test suite and when running just that regression test. The tracebacks are coming from the embedded calls to manage.py from the looks of it.

As you'll see, all the results are coming in as normal FAIL results from the test runner, so it isn't exploding. It's just seeing different results on stderr to what it expects.

For reference, I'm running Python 2.5, SQLite in these cases (my settings file contains the absolute minimum to enable runtests.py to run without errors) on an up-to-date Fedora 8 box. No other errors are manifesting themself.

Sorry, Russ, I've got no idea why this isn't working. Been scratching my head on and off for most of this evening. Grab me on IM or IRC (malcolmt on freenode) if you want more realtime information. I don't want to flood you with crap you don't need, but I can repeat this as often as we like to gather information.

[Malcolm Tredinnick]

Output from "./runtests.py --settings=settings admin_scripts 2>stderr-output.txt"

[Russell Keith-Magee]

Found the problem. All the failures are for tests cases which _should_ be catching in manage.py itself - the case where you are running manage.py, but you don't have a settings.py.

If your last comment is accurate (i.e., that your command line reads './runtests.py --settings=settings'), then you *do* have a settings file on your PYTHONPATH - just not the one that the test is testing that you don't have (if that makes any sense).

If you rename /tests/settings.py to /tests/sqlite.py (and delete settings.pyc, if it exists) the tests will pass.

The problem is caused by the fact that /tests is in the PYTHONPATH, so that regressiontests appears as a project directory; however, it may be possible to put /tests/regressiontests in the PYTHONPATH instead. I'll have a poke and see what I can do. Worst case, the solution is to document the fact that you can't use 'settings.py' for your test settings.

[Malcolm Tredinnick]

Instead of asking testers to not use the most obvious name in the world for their settings files, probably better for the tests themselves to ask for something with an unlikely name, e.g. qwertyuiopxxy67.py.

I realise now, too, that the initial report was conflating two different problems. One is the settings.py issue you've discovered. The other is that when I use settings3.py, which I also checked with, I was getting a different error due to something my PostgreSQL install was spitting out to stderr that was confusing the tests. So that one was a side issue that will affect maybe 6 people in the whole world, whereas the SQLite tests were always failing because of the problem you've found.

[Russell Keith-Magee]

Replying to mtredinnick:

Instead of asking testers to not use the most obvious name in the world for their settings files, probably better for the tests themselves to ask for something with an unlikely name, e.g. qwertyuiopxxy67.py.

Problem is, we can't do that. We're testing manage.py, which is a script generated by Django itself, and that script explicitly tries 'import settings'. If we're testing the correct behaviour when settings.py doesn't exist, we need to be able to use the name 'settings.py'.

I suppose one alternative is to drop those 11 tests. ./manage.py fails outright if you don't have settings.py, no matter what you are trying to do. Testing that it does fail is certainly nice for the sake of completeness, but in reality, it's not actually a scenario that is likely to cause much difficulty.

If anything, it highlights a problem with ./manage.py - the fact that * you can't use it if you don't have settings.py defined, even if you provide --settings or DJANGO_SETTINGS_MODULE, and if you happen to have a module on your pythonpath that is called 'settings' but isn't a settings module, you could get some very interesting behaviour.

[Malcolm Tredinnick]

Yeah, you're right about not being able to change it. I'm not too worried about the hard-coded behaviour of manage.py, since it's just a tool for getting started, but not breaking it would be cool. And it's pretty much a given that having something importable named after the settings file you're trying to use will fail. That's just the way Python works.

Kind of annoying to not be able to use settings.py for the test settings (I now have two years or so of muscular memory to unlearn when running tests), but maybe it's unavoidable if we want to test this stuff. C'est la vie. :-(

[Russell Keith-Magee]

(In [7887]) Fixed #7715 -- Modified the admin_scripts tests to do an app level import, rather than a project level import. This removes the potential for conflict between the admin script tests and the settings for the test suite itself. Thanks to Malcolm Tredinnick for his help hunting this one down.