Make Default Test Output More Verbose (And Formatted Prettier)

[Rich Jones]

This ticket is part of the Django Test Output Beautification initiative: https://code.djangoproject.com/ticket/29101

The current default test output provides no information about what test is currently running. This makes the test output pretty useless if you're actually trying to determine which test is taking so long, or failing, or misbehaving, etc.

The current test output is a wall of unformatted text - bleh! With --verbosity=3, it includes information about which test is running, and the description of the test (yay), but the formatting is still horrible.

test_bad_file_grouping (data_refinery_workers.downloaders.test_transcriptome_index.DownloadTranscriptomeIndexTestCase)
Raises exception if both files don't have the same download_url. ... 2018-02-02 15:31:50,587 local/MainProcess data_refinery_workers.downloaders.transcriptome_index ERROR [downloader_job: 12]: A Batch's file doesn't have the same download URL as the other batch's file.
ok
test_good_file_grouping (data_refinery_workers.downloaders.test_transcriptome_index.DownloadTranscriptomeIndexTestCase)
Raises exception if both files don't have the same download_url. ...
ok

We should make the _default_ test output to be something like:

========================
Testing 'test_bad_file_grouping'
(data_refinery_workers.downloaders.test_transcriptome_index.DownloadTranscriptomeIndexTestCase)
_Raises exception if both files don't have the same download_url._
-----------------------------------------------------
...
ERROR [downloader_job: 12]: A Batch's file doesn't have the same download URL as the other batch's file.
Ok!

========================
Testing 'test_good_file_grouping'
(data_refinery_workers.downloaders.test_transcriptome_index.DownloadTranscriptomeIndexTestCase)
_Makes sure that proper file group runs the pipeline._
-----------------------------------------------------
...
Ok!

Where test title, test description and Ok/Failure are also colorized appropriately.

I'm not totally certain what else the 'verbosity' argument is controlling, (perhaps log level?), but either way I think the way to improve this experience is clear: Move to a more structured, human-friendly text formatting and away from barfing out a wall of unformatted text.

Related tickets:
https://code.djangoproject.com/ticket/29101
https://code.djangoproject.com/ticket/22449

[Chris Jerdonek]

I like that the default output prints just a single . for each test. For example, if you have 1000 tests, you wouldn't want 5 lines per test, or even one line.

However, I agree the format can be improved when the verbosity is increased. I do think one of the available verbosity levels should show the "minimal" information needed to know what test is being run -- so without any divider lines, test description, etc. It can be as simple as a single line that looks something like:

Testing: <test name>

or perhaps prefixed with the number of the test:

1. Testing: <test name>

Higher verbosity levels can show more like the description, divider lines, etc.

[Tim Graham]

Django is delegating to Python's testing output here. I'm not immediately convinced that we should implement our own test output. As I mentioned in the other tickets, there are a number of third-party testing tools that offer other outputs. Do we need to spend our time doing something similar? Feel free to start a thread on the DevelopersMailingList to determine if there's a consensus.