Opened 16 years ago

Closed 11 years ago

#6712 closed New feature (duplicate)

choosing tests with command line among tests provided by custom 'suite' method

Reported by: alainD Owned by: nobody
Component: Testing framework Version: dev
Severity: Normal Keywords:
Cc: joel@… Triage Stage: Accepted
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: yes
Easy pickings: no UI/UX: no

Description

Currently the new svn syntax to perform specific test :

./manage.py test animals.AnimalTestCase.testFluffyAnimals

works only if the tests are provided in tests.py or models.py.

The attached patch allow to pick individual tests (unittest classes or method) from the test 'suite' (if provided).

Attachments (1)

svn.diff (1.4 KB ) - added by alainD 16 years ago.
diff file (modified file is django/test/simple.py)

Download all attachments as: .zip

Change History (14)

by alainD, 16 years ago

Attachment: svn.diff added

diff file (modified file is django/test/simple.py)

comment:1 by Jacob, 16 years ago

Triage Stage: UnreviewedDesign decision needed

comment:2 by Russell Keith-Magee, 16 years ago

I'm inclined to mark this 'worksforme' - I have projects with a number of tests that aren't literally defined in models.py or tests.py, but I don't have any difficulty invoking them using the app.TestCase.test notation. How are you defining your tests so that they are found by the suite builder, but aren't addressable by the dot notation?

comment:3 by alainD2 <http://code.djangoproject.com/>, 16 years ago

My layout is like this :
myapp/tests is full of test files defining one or many TestCase classes ... and the myapp/tests/\_\_init\_\_.py is :

import unittest,os

MOD_LOCATION = 'myapp/tests/'

    
def suite_all():    
    suite = unittest.TestSuite() 
    list_mod = [name[:-2] for name in os.listdir(MOD_LOCATION) if name.startswith('test_') and name.endswith('.py')]
    for mod in list_mod:
        the_mod = __import__(MOD_LOCATION + mod)
        tests = unittest.defaultTestLoader.loadTestsFromModule(the_mod)
        suite.addTests(tests)
    return suite 

def suite():
    return suite_all()

I provide the suite function for django to be able to pick my tests ... but they aren't addressable by the dot notation ...

Am I doing something wrong here ?

comment:4 by Russell Keith-Magee, 16 years ago

Patch needs improvement: set
Triage Stage: Design decision neededAccepted

Not wrong - just different, and I didn't parse the ticket title correctly. I haven't been in the habit of using suite(); I just add from test_module import * to the tests/init.py for each of my test module. This puts all the test cases into the myapp.tests module, so the dot notation works.

Marking as 'needs improvement' because of two potential issues that I need to mull over in my mind (feel free to comment on these):

  1. _get_all_tests_from_suite() could be used to replace the two getattr calls that are used to extract single tests. This suggests that there could be some refactoring around build_test. (i.e., is build_test even necessary? Can we just build the suite, and extract the tests we need if a dotted name is provided?)
  2. _get_all_tests_from_suite() flattens the test namespace. I'm not completely convinced this is a good idea. I'm not convinced it's a bad idea, either - I'm just a little hesitant to remove namespaces unless I can be convinced that name clashes aren't a likely consequence.

Regarding 2: my immediate counter-reaction is that we should be looking to integrate the dot notation for specifying tests with the heirarchical structure of suites. However, I can't think of an obvious way to do this. Alternatively, we can fix this as documentation by recommending (at a documentation level) that suites not be heirarchical. This wouldn't be the ideal solution, but in the absence of a good solution for the general case, it may be the best we can do.

comment:5 by alainD, 16 years ago

  1. that's right, I didn't wanted to refactor simple.py too much, that's why I simply added a function and a small block in build_test
  1. I think tests should be defined by their test class and method and that's why I'm not puzzled by the flattening of the test namespace ... besides, tests in tests.py and in models.py are already treated the same way ...

I confess I did not think about adding from test_module import * into tests/init.py ... maybe it is sufficient for most use cases ...

in reply to:  5 comment:6 by Russell Keith-Magee, 16 years ago

Replying to alainD:

  1. that's right, I didn't wanted to refactor simple.py too much, that's why I simply added a function and a small block in build_test

I understand (and appreciate) the caution, but in this case, it's not really a concern. The only documented API is the run_tests entry point.

  1. I think tests should be defined by their test class and method and that's why I'm not puzzled by the flattening of the test namespace ... besides, tests in tests.py and in models.py are already treated the same way ...

The problem is that the test class name isn't a guaranteed unique namespace. Consider the following app layout:

/myapp

models.py

/tests

init.py
anonymous.py
evaluation.py
registered.py

Now, imagine that anonymous.py, evaluation.py and registered.py all contain "ValidationTest(TestCase)". This is a reasonable organization - the three ValidationTests would be in separate modules, but their names would clash if they were referenced using a mytest.ValidationTest naming scheme.

Under the current setup, you only have to worry about clashes between tests in models.py and tests in tests.py; in practice, models.py isn't really the right place for a serious test suite, so the chance of a clash is minimal. Using the 'from test_module import *' approach does increase the chance of a clash, but it requires you to explicitly import tests into a common namespace. The approach you are proposing would replace this explicit namespace clash with an implicit one.

I confess I did not think about adding from test_module import * into tests/init.py ... maybe it is sufficient for most use cases ...

It certainly has been for me. That's not to say that improvements you suggest aren't worthwhile - we just need to work out how to make them fully generic.

comment:7 by Russell Keith-Magee, 16 years ago

Bah. That example layout should read:

/myapp
   models.py
   /tests
      init.py 
      anonymous.py 
      evaluation.py 
      registered.py

comment:8 by Julien Phalip, 13 years ago

Severity: Normal
Type: New feature

comment:9 by Ramiro Morales, 13 years ago

Easy pickings: unset
UI/UX: unset

#12191 was closed as duplicate, contains a patch.

comment:10 by Joel Cross, 12 years ago

Cc: joel@… added

comment:11 by Aymeric Augustin, 11 years ago

#19156 was a duplicate with a (much simpler) patch.

comment:12 by Aymeric Augustin, 11 years ago

It seems to me that #17365 describes the general problem and the way forward.

comment:13 by Carl Meyer, 11 years ago

Resolution: duplicate
Status: newclosed

Closing this as duplicate of #17365

Note: See TracTickets for help on using tickets.
Back to Top