id	summary	reporter	owner	description	type	status	component	version	severity	resolution	keywords	cc	stage	has_patch	needs_docs	needs_tests	needs_better_patch	easy	ui_ux
12574	all tests pass when there broken tests becase they are skipped	CarlFK	nobody	"If a test is broken (like typo in models.py)
and you run the full set of tests: 

{{{django-trunk/tests$ ./runtests.py --settings=sets_sqlite3}}}

the broken test gets skipped
and so you get:
{{{
----------------------------------------------------------------------
Ran 1390 tests in 243.340s
OK
----------------------------------------------------------------------
}}}
Which makes you think ""oh boy!  all my tests passed!"" and go dancing in the street.

But no!  the very test you were most concerned about never got run. 
If you dig around in the output, you will see the error.  
If you see OK, you really have no reason to do that.

Here is why:
(trimmed down version)
{{{
#!python
        try:
            mod = load_app(model_label)
            if mod:
                settings.INSTALLED_APPS.append(model_label)
        except Exception, e:
            sys.stderr.write(""Error while importing %s:"" % model_name + ''.join(traceback.format_exception(*sys.exc_info())[1:]))
}}}

Looking at the change history, it seems this is the result of some refactoring that resulted in this hiding place for bad tests.  The ""continue"" command is unnecessary, given it is at the end of the codeblock.

Seems the fix is to remove the try/except:err.write and let the error get thrown.

btw, the comment for 
{{{
#!python
if not test_labels or model_name in set([label.split('.')[0] for label in test_labels]):
}}}
is screwy.  so is the code.  Neither read well.  if you fully understand it, it would be great if it could be cleaned up so I don't get a headache. 
"		closed	Testing framework	dev		fixed			Accepted	1	0	0	0	0	0