Opened 7 years ago

Closed 2 years ago

#6712 closed New feature (duplicate)

choosing tests with command line among tests provided by custom 'suite' method

Reported by: alainD Owned by: nobody
Component: Testing framework Version: master
Severity: Normal Keywords:
Cc: joel@… Triage Stage: Accepted
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: yes
Easy pickings: no UI/UX: no

Description

Currently the new svn syntax to perform specific test :

./manage.py test animals.AnimalTestCase.testFluffyAnimals

works only if the tests are provided in tests.py or models.py.

The attached patch allow to pick individual tests (unittest classes or method) from the test 'suite' (if provided).

Attachments (1)

svn.diff (1.4 KB) - added by alainD 7 years ago.
diff file (modified file is django/test/simple.py)

Download all attachments as: .zip

Change History (14)

Changed 7 years ago by alainD

diff file (modified file is django/test/simple.py)

comment:1 Changed 7 years ago by jacob

  • Needs documentation unset
  • Needs tests unset
  • Patch needs improvement unset
  • Triage Stage changed from Unreviewed to Design decision needed

comment:2 Changed 7 years ago by russellm

I'm inclined to mark this 'worksforme' - I have projects with a number of tests that aren't literally defined in models.py or tests.py, but I don't have any difficulty invoking them using the app.TestCase.test notation. How are you defining your tests so that they are found by the suite builder, but aren't addressable by the dot notation?

comment:3 Changed 7 years ago by alainD2 <http://code.djangoproject.com/>

My layout is like this :
myapp/tests is full of test files defining one or many TestCase classes ... and the myapp/tests/\_\_init\_\_.py is :

import unittest,os

MOD_LOCATION = 'myapp/tests/'

    
def suite_all():    
    suite = unittest.TestSuite() 
    list_mod = [name[:-2] for name in os.listdir(MOD_LOCATION) if name.startswith('test_') and name.endswith('.py')]
    for mod in list_mod:
        the_mod = __import__(MOD_LOCATION + mod)
        tests = unittest.defaultTestLoader.loadTestsFromModule(the_mod)
        suite.addTests(tests)
    return suite 

def suite():
    return suite_all()

I provide the suite function for django to be able to pick my tests ... but they aren't addressable by the dot notation ...

Am I doing something wrong here ?

comment:4 Changed 7 years ago by russellm

  • Patch needs improvement set
  • Triage Stage changed from Design decision needed to Accepted

Not wrong - just different, and I didn't parse the ticket title correctly. I haven't been in the habit of using suite(); I just add from test_module import * to the tests/init.py for each of my test module. This puts all the test cases into the myapp.tests module, so the dot notation works.

Marking as 'needs improvement' because of two potential issues that I need to mull over in my mind (feel free to comment on these):

  1. _get_all_tests_from_suite() could be used to replace the two getattr calls that are used to extract single tests. This suggests that there could be some refactoring around build_test. (i.e., is build_test even necessary? Can we just build the suite, and extract the tests we need if a dotted name is provided?)
  2. _get_all_tests_from_suite() flattens the test namespace. I'm not completely convinced this is a good idea. I'm not convinced it's a bad idea, either - I'm just a little hesitant to remove namespaces unless I can be convinced that name clashes aren't a likely consequence.

Regarding 2: my immediate counter-reaction is that we should be looking to integrate the dot notation for specifying tests with the heirarchical structure of suites. However, I can't think of an obvious way to do this. Alternatively, we can fix this as documentation by recommending (at a documentation level) that suites not be heirarchical. This wouldn't be the ideal solution, but in the absence of a good solution for the general case, it may be the best we can do.

comment:5 follow-up: Changed 7 years ago by alainD

  1. that's right, I didn't wanted to refactor simple.py too much, that's why I simply added a function and a small block in build_test
  1. I think tests should be defined by their test class and method and that's why I'm not puzzled by the flattening of the test namespace ... besides, tests in tests.py and in models.py are already treated the same way ...

I confess I did not think about adding from test_module import * into tests/init.py ... maybe it is sufficient for most use cases ...

comment:6 in reply to: ↑ 5 Changed 7 years ago by russellm

Replying to alainD:

  1. that's right, I didn't wanted to refactor simple.py too much, that's why I simply added a function and a small block in build_test

I understand (and appreciate) the caution, but in this case, it's not really a concern. The only documented API is the run_tests entry point.

  1. I think tests should be defined by their test class and method and that's why I'm not puzzled by the flattening of the test namespace ... besides, tests in tests.py and in models.py are already treated the same way ...

The problem is that the test class name isn't a guaranteed unique namespace. Consider the following app layout:

/myapp

models.py

/tests

init.py
anonymous.py
evaluation.py
registered.py

Now, imagine that anonymous.py, evaluation.py and registered.py all contain "ValidationTest(TestCase)". This is a reasonable organization - the three ValidationTests would be in separate modules, but their names would clash if they were referenced using a mytest.ValidationTest naming scheme.

Under the current setup, you only have to worry about clashes between tests in models.py and tests in tests.py; in practice, models.py isn't really the right place for a serious test suite, so the chance of a clash is minimal. Using the 'from test_module import *' approach does increase the chance of a clash, but it requires you to explicitly import tests into a common namespace. The approach you are proposing would replace this explicit namespace clash with an implicit one.

I confess I did not think about adding from test_module import * into tests/init.py ... maybe it is sufficient for most use cases ...

It certainly has been for me. That's not to say that improvements you suggest aren't worthwhile - we just need to work out how to make them fully generic.

comment:7 Changed 7 years ago by russellm

Bah. That example layout should read:

/myapp
   models.py
   /tests
      init.py 
      anonymous.py 
      evaluation.py 
      registered.py

comment:8 Changed 4 years ago by julien

  • Severity set to Normal
  • Type set to New feature

comment:9 Changed 4 years ago by ramiro

  • Easy pickings unset
  • UI/UX unset

#12191 was closed as duplicate, contains a patch.

comment:10 Changed 3 years ago by ukch

  • Cc joel@… added

comment:11 Changed 2 years ago by aaugustin

#19156 was a duplicate with a (much simpler) patch.

comment:12 Changed 2 years ago by aaugustin

It seems to me that #17365 describes the general problem and the way forward.

comment:13 Changed 2 years ago by carljm

  • Resolution set to duplicate
  • Status changed from new to closed

Closing this as duplicate of #17365

Note: See TracTickets for help on using tickets.
Back to Top