1.6 compatibility checks don't correctly validate TEST_RUNNER

[Russell Keith-Magee]

The 1.6 compatibility checks (executed by the new check management command) don't appear to perform an accurate check.

I can see 5 scenarios that need to be accounted for:

1. A default pre-Django 1.6 project
2. A pre-Django 1.6 project with TEST_RUNNER set to DiscoverRunner
3. A pre-Django 1.6 project with TEST_RUNNER set to something custom
4. A default Django 1.6 project
5. A Django 1.6 project with TEST_RUNNER set to something custom

The management command does the following:

    from django.conf import settings
    new_default = 'django.test.runner.DiscoverRunner'
    test_runner_setting = getattr(settings, 'TEST_RUNNER', new_default)

    if test_runner_setting == new_default:
        ... output warning ...

That is, it retrieves the current setting value of TEST_RUNNER, and if it is DiscoverRunner, it raises a warning. This behavior:

• Is correct for case 1
• Is possibly incorrect for case 3 and 5, as there's no guarantee the custom test runner subclasses DiscoverRunner (and it probably won't)
• Incorrectly raises an error for case 2 and 4.

There are test cases for the current implementation, but they aren't very solid - they're operating inside a test environment that is itself force setting TEST_RUNNER.

[Shai Berger]

Note similar discussion on mailing list: ​Django 1.6 and migrating to the new test runner: note on user experience.
The discussion is not about the "check" command, but essentially asking to incorporate some "check" functionality into the "test" command.

Note in particular that my suggestion to add a settings knob specifying the Django version the project is expecting (which would default to 1.5, with 1.6 in the default project templates) should also allow easy resolution of this bug.