choosing tests with command line among tests provided by custom 'suite' method

[alainD]

Currently the new svn syntax to perform specific test :

./manage.py test animals.AnimalTestCase.testFluffyAnimals

works only if the tests are provided in tests.py or models.py.

The attached patch allow to pick individual tests (unittest classes or method) from the test 'suite' (if provided).

[alainD]

diff file (modified file is django/test/simple.py)

[Russell Keith-Magee]

I'm inclined to mark this 'worksforme' - I have projects with a number of tests that aren't literally defined in models.py or tests.py, but I don't have any difficulty invoking them using the app.TestCase.test notation. How are you defining your tests so that they are found by the suite builder, but aren't addressable by the dot notation?

[alainD2 <http://code.djangoproject.com/>]

My layout is like this :
myapp/tests is full of test files defining one or many TestCase classes ... and the myapp/tests/\_\_init\_\_.py is :

import unittest,os

MOD_LOCATION = 'myapp/tests/'

    
def suite_all():    
    suite = unittest.TestSuite() 
    list_mod = [name[:-2] for name in os.listdir(MOD_LOCATION) if name.startswith('test_') and name.endswith('.py')]
    for mod in list_mod:
        the_mod = __import__(MOD_LOCATION + mod)
        tests = unittest.defaultTestLoader.loadTestsFromModule(the_mod)
        suite.addTests(tests)
    return suite 

def suite():
    return suite_all()

I provide the suite function for django to be able to pick my tests ... but they aren't addressable by the dot notation ...

Am I doing something wrong here ?

[Russell Keith-Magee]

Not wrong - just different, and I didn't parse the ticket title correctly. I haven't been in the habit of using suite(); I just add from test_module import * to the tests/init.py for each of my test module. This puts all the test cases into the myapp.tests module, so the dot notation works.

Marking as 'needs improvement' because of two potential issues that I need to mull over in my mind (feel free to comment on these):

1. _get_all_tests_from_suite() could be used to replace the two getattr calls that are used to extract single tests. This suggests that there could be some refactoring around build_test. (i.e., is build_test even necessary? Can we just build the suite, and extract the tests we need if a dotted name is provided?)
2. _get_all_tests_from_suite() flattens the test namespace. I'm not completely convinced this is a good idea. I'm not convinced it's a bad idea, either - I'm just a little hesitant to remove namespaces unless I can be convinced that name clashes aren't a likely consequence.

Regarding 2: my immediate counter-reaction is that we should be looking to integrate the dot notation for specifying tests with the heirarchical structure of suites. However, I can't think of an obvious way to do this. Alternatively, we can fix this as documentation by recommending (at a documentation level) that suites not be heirarchical. This wouldn't be the ideal solution, but in the absence of a good solution for the general case, it may be the best we can do.

[alainD]

1. that's right, I didn't wanted to refactor simple.py too much, that's why I simply added a function and a small block in build_test

2. I think tests should be defined by their test class and method and that's why I'm not puzzled by the flattening of the test namespace ... besides, tests in tests.py and in models.py are already treated the same way ...

I confess I did not think about adding from test_module import * into tests/init.py ... maybe it is sufficient for most use cases ...

[Russell Keith-Magee]

Replying to alainD:

1. that's right, I didn't wanted to refactor simple.py too much, that's why I simply added a function and a small block in build_test

I understand (and appreciate) the caution, but in this case, it's not really a concern. The only documented API is the run_tests entry point.

2. I think tests should be defined by their test class and method and that's why I'm not puzzled by the flattening of the test namespace ... besides, tests in tests.py and in models.py are already treated the same way ...

The problem is that the test class name isn't a guaranteed unique namespace. Consider the following app layout:

/myapp

models.py

/tests

init.py
anonymous.py
evaluation.py
registered.py

Now, imagine that anonymous.py, evaluation.py and registered.py all contain "ValidationTest(TestCase)". This is a reasonable organization - the three ValidationTests would be in separate modules, but their names would clash if they were referenced using a mytest.ValidationTest naming scheme.

Under the current setup, you only have to worry about clashes between tests in models.py and tests in tests.py; in practice, models.py isn't really the right place for a serious test suite, so the chance of a clash is minimal. Using the 'from test_module import *' approach does increase the chance of a clash, but it requires you to explicitly import tests into a common namespace. The approach you are proposing would replace this explicit namespace clash with an implicit one.

I confess I did not think about adding from test_module import * into tests/init.py ... maybe it is sufficient for most use cases ...

It certainly has been for me. That's not to say that improvements you suggest aren't worthwhile - we just need to work out how to make them fully generic.

[Russell Keith-Magee]

Bah. That example layout should read:

/myapp
   models.py
   /tests
      init.py 
      anonymous.py 
      evaluation.py 
      registered.py

[Ramiro Morales]

#12191 was closed as duplicate, contains a patch.

[Aymeric Augustin]

#19156 was a duplicate with a (much simpler) patch.

[Aymeric Augustin]

It seems to me that #17365 describes the general problem and the way forward.

[Carl Meyer]

Closing this as duplicate of #17365