Use sqlparse to format SQL in tests failures with debug flag

[Paolo Melchiorre]

Running test with --debug flag that generate errors print unformatted SQL code that need to be formatted to read it properly and figuring out what's not working.

sqlparse is already a dependency so can be very useful to format that SQL code in the traceback with it.

Example of test failure that generate unformatted SQL code:

$ ./runtests.py --settings=postgres --timing --parallel=1 aggregation.tests.AggregateTestCase.test_aggregation_exists_multivalued_outeref --debug-sql
Testing against Django installed in '/home/paulox/Projects/django/django'
Found 1 test(s).
Creating test database for alias 'default'...
System check identified no issues (0 silenced).
E
======================================================================
ERROR: test_aggregation_exists_multivalued_outeref (aggregation.tests.AggregateTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/paulox/Projects/django/django/db/backends/utils.py", line 89, in _execute
    return self.cursor.execute(sql, params)
  File "/home/paulox/.virtualenvs/django/lib/python3.10/site-packages/psycopg/cursor.py", line 725, in execute
    raise ex.with_traceback(None)
psycopg.errors.GroupingError: subquery uses ungrouped column "aggregation_book.publisher_id" from outer query
LINE 1: ... "aggregation_book" U0 WHERE U0."publisher_id" = ("aggregati...
                                                             ^

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/paulox/Projects/django/tests/aggregation/tests.py", line 1667, in test_aggregation_exists_multivalued_outeref
    self.assertCountEqual(
  File "/usr/lib/python3.10/unittest/case.py", line 1188, in assertCountEqual
    first_seq, second_seq = list(first), list(second)
  File "/home/paulox/Projects/django/django/db/models/query.py", line 394, in __iter__
    self._fetch_all()
  File "/home/paulox/Projects/django/django/db/models/query.py", line 1876, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
  File "/home/paulox/Projects/django/django/db/models/query.py", line 87, in __iter__
    results = compiler.execute_sql(
  File "/home/paulox/Projects/django/django/db/models/sql/compiler.py", line 1518, in execute_sql
    cursor.execute(sql, params)
  File "/home/paulox/Projects/django/django/db/backends/utils.py", line 103, in execute
    return super().execute(sql, params)
  File "/home/paulox/Projects/django/django/db/backends/utils.py", line 67, in execute
    return self._execute_with_wrappers(
  File "/home/paulox/Projects/django/django/db/backends/utils.py", line 80, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/home/paulox/Projects/django/django/db/backends/utils.py", line 84, in _execute
    with self.db.wrap_database_errors:
  File "/home/paulox/Projects/django/django/db/utils.py", line 91, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/home/paulox/Projects/django/django/db/backends/utils.py", line 89, in _execute
    return self.cursor.execute(sql, params)
  File "/home/paulox/.virtualenvs/django/lib/python3.10/site-packages/psycopg/cursor.py", line 725, in execute
    raise ex.with_traceback(None)
django.db.utils.ProgrammingError: subquery uses ungrouped column "aggregation_book.publisher_id" from outer query
LINE 1: ... "aggregation_book" U0 WHERE U0."publisher_id" = ("aggregati...
                                                             ^

----------------------------------------------------------------------
(0.000) SELECT "aggregation_publisher"."id", "aggregation_publisher"."name", "aggregation_publisher"."num_awards", "aggregation_publisher"."duration", EXISTS(SELECT 1 AS "a" FROM "aggregation_book" U0 WHERE U0."publisher_id" = ("aggregation_book"."publisher_id") LIMIT 1) AS "books_exists", COUNT("aggregation_book"."id") AS "books_count" FROM "aggregation_publisher" LEFT OUTER JOIN "aggregation_book" ON ("aggregation_publisher"."id" = "aggregation_book"."publisher_id") GROUP BY "aggregation_publisher"."id", EXISTS(SELECT 1 AS "a" FROM "aggregation_book" U0 WHERE U0."publisher_id" = ("aggregation_book"."publisher_id") LIMIT 1); args=(1, 1); alias=default

----------------------------------------------------------------------
Ran 1 test in 0.040s

FAILED (errors=1)
Destroying test database for alias 'default'...
Total database setup took 1.077s
  Creating 'default' took 1.077s
Total database teardown took 0.073s
Total run took 1.303s

[Mariusz Felisiak]

Sounds good, thanks.

[Giebisch]

Sounds good, I'll get on it

[Giebisch]

PR ​https://github.com/django/django/pull/16217 has been created

[Simon Charette]

Thanks for the patch Giebisch!

Left a comment for improvement to the tests, please uncheck patch needs improvement when they are addressed so the patch appears in the review queue.

[Giebisch]

Replying to Simon Charette:

Thanks for the patch Giebisch!

Left a comment for improvement to the tests, please uncheck patch needs improvement when they are addressed so the patch appears in the review queue.

Thanks for the fast feedback! You are actually right. I'll update the code this weekend :)

[Giebisch]

Replying to Simon Charette:

Thanks for the patch Giebisch!

Left a comment for improvement to the tests, please uncheck patch needs improvement when they are addressed so the patch appears in the review queue.

I've just updated the code, hope this is what you had in mind. Hope to contribute more in the near future!

[Simon Charette]

Thanks Giebisch, the patch looks great!

Do you think you could also add a mention of the new feature in ​the Tests section of the 4.2 release notes?

[Giebisch]

Replying to Simon Charette:

Thanks Giebisch, the patch looks great!

Do you think you could also add a mention of the new feature in ​the Tests section of the 4.2 release notes?

Sure thing, done!

[Mariusz Felisiak <felisiak.mariusz@…>]

In 3283120c:

Fixed #34111 -- Made test runner with --debug-sql format SQL queries.