SimpleTestCase should prevent execution of database queries

[Simon Charette]

SimpleTestCase is a simple subclass of unittest.TestCase with added assertion methods to deal with Django related stuff such as the TestClient.

While it's ​documented that SimpleTestCase shouldn't be used to test anything related to the ORM, nothing prevent users from doing it in unexpected ways. This can result in hard to detect test isolation issues as we have experienced with the Django test suite.

I suggest to prevent the usage of the ORM during the execution of a SimpleTestCase by raising an exception pointing to the usage of TestCase (or TransactionTestCase) on query execution attempts.

This will also be a good safenet for our internal usage, we should feel confident asking contributors to use SimpleTestCase where appropriate to avoid useless transaction wrapping.

[Anssi Kääriäinen]

I'm not sure full prevention of database queries is a good idea. There are cases where you do want to run outside transaction control but prevent database flushing. The most important reasons for the latter part are that you want to preserve some of your dataset between test runs, and performance of database flushing.

Maybe we could go with an approach where SimpleTestCase disables database queries unless the test class sets attribute allow_database_queries to True. Then the exception could tell why it isn't a good idea to allow database queries, but also allow users to execute queries if they want to do so.

Another approach could be to support TransactionTestCase with "disable_db_flush" attribute. If you use that class, then your database isn't flushed pre or post run. This one is more explicit in opting in to the unsafe behavior. The hard part might be that all flushing must be disabled until all TransactionTestCases with disable_db_flush set to True are ran.

[Simon Charette]

Maybe we could go with an approach where SimpleTestCase disables database queries unless the test class sets attribute allow_database_queries to True. Then the exception could tell why it isn't a good idea to allow database queries, but also allow users to execute queries if they want to do so.

This is what ​the actual implementation does. I'll alter the message error to suggest setting the allow_database_queries attribute to True with a big warning.

[Simon Charette <charette.s@…>]

In 3db21c35:

Refs #24652 -- Enforced test isolation in file_storage tests.

[Simon Charette <charette.s@…>]

In 8bf1449:

Refs #24652 -- Converted a template test to avoid executing queries.

[Simon Charette <charette.s@…>]

In ead36e8:

Refs #24652 -- Made sure template backend tests call their super setUpClass.

[Simon Charette <charette.s@…>]

In c15b0c2:

Fixed #24652 -- Disallowed query execution in SimpleTestCase subclasses.

Thanks to Tim and Anssi for the review.

[Simon Charette <charette.s@…>]

In e846ea06:

[1.8.x] Refs #24652 -- Enforced test isolation in file_storage tests.

Backport of 3db21c351b9b1108954c388799d35c8dad7dfc19 from master

[Simon Charette <charette.s@…>]

In 3ab3be4b:

[1.8.x] Refs #24652 -- Converted a template test to avoid executing queries.

Backport of 8bf1449edb4139451643e3823b194b4c02ca7633 from master

[Simon Charette <charette.s@…>]

In 2b2a2157:

[1.8.x] Refs #24652 -- Made sure template backend tests call their super setUpClass.

Backport of ead36e8a471389a6032d825c8245245ebb89ea5d from master

[Simon Charette <charette.s@…>]

In be67400b:

Refs #24652 -- Used SimpleTestCase where appropriate.

[Simon Charette <charette.s@…>]

In 4ccfc44:

Refs #24652 -- Fixed a test failure in file_uploads tests on Windows.

Thanks to Tim Graham for the report.