Cloning test databases should reraise errors on MySQL.

[Stephen Finucane]

I've been seeing the following error message when attempting to run tests in parallel using a MySQL backend:

py36-django22 run-test-pre: PYTHONHASHSEED='1711093702'
py36-django22 run-test: commands[0] | python /home/patchwork/patchwork/manage.py test --noinput --parallel -- patchwork
mysqldump: Couldn't execute 'SELECT COLUMN_NAME,                       JSON_EXTRACT(HISTOGRAM, '$."number-of-buckets-specified"')                FROM information_schema.COLUMN_STATISTICS                WHERE SCHEMA_NAME = 'test_patchwork' AND TABLE_NAME = 'auth_group';': Unknown table 'COLUMN_STATISTICS' in information_schema (1109)
mysqldump: Couldn't execute 'SELECT COLUMN_NAME,                       JSON_EXTRACT(HISTOGRAM, '$."number-of-buckets-specified"')                FROM information_schema.COLUMN_STATISTICS                WHERE SCHEMA_NAME = 'test_patchwork' AND TABLE_NAME = 'auth_group';': Unknown table 'COLUMN_STATISTICS' in information_schema (1109)
System check identified no issues (0 silenced).
Creating test database for alias 'default'...
Cloning test database for alias 'default'...
Cloning test database for alias 'default'...
...

This is followed by a lot of test failures due to missing databases.

In my instance, I traced this back to the fact that I was using mariadb-server with mysql-client rather than mariadb-server. Switching the client packages resolved the issue. However, posts on StackOverflow suggest the same issue can occur when using older MySQL servers (5.7?) that don't provide these statistics with newer clients so it's not as clear cut as this.

I'm not sure there's much to be done here from a Django perspective. We could arguably add the -column-statistics=0 flag to the mysqldump call (in django.db.backends.mysql.creation.DatabaseCreation._clone_db), but this is not compatible with the version of this tool provided by mariadb-client. We might also want to make this a more serious error and fail the test run rather than attempt to proceed, obscuring the failure in the process (this could conceivably get flaky if you had a mix of DB and non-DB tests running in different runner instances). Perhaps the best thing to do here is close this and leave it here as a breadcrumb for others that run into the issue.

[Mariusz Felisiak]

Thanks for this ticket, however Django 2.2 doesn't officially support MariaDB. Moreover Django 2.2 is in extended support so it doesn't receive bugfixes anymore (except security issues). Please reopen this ticket if you can reproduce the issue on Django 4.0 (or the current main branch). I couldn't reproduce it with MySQL 8+ or MariaDB.

[Stephen Finucane]

Reopening per feedback on #34083

[Stephen Finucane]

Reproducing from #34083.

I'm attempting to run tests in parallel using MariaDB. When running tests, all access to the additional databases fail. Scrolling back to the top, I see the following error messages:

❯ tox -e py38-django41 -- patchwork.tests.api.test_user patchwork.tests.api.test_event
py38-django41 run-test-pre: PYTHONHASHSEED='631429049'
py38-django41 run-test: commands[0] | python /home/patchwork/patchwork/manage.py test --noinput --parallel -v 2 --timing -- patchwork.tests.api.test_user patchwork.tests.api.test_event
Found 25 test(s).
Creating test database for alias 'default' ('test_patchwork')...
Operations to perform:
  Synchronize unmigrated apps: django_filters, humanize, messages, rest_framework, staticfiles
  Apply all migrations: admin, auth, authtoken, contenttypes, patchwork, sessions, sites
Synchronizing apps without migrations:
  Creating tables...
    Running deferred SQL...
Running migrations:
  Applying contenttypes.0001_initial... OK
  ... {skipped} ...
  Applying sites.0002_alter_domain_unique... OK
Cloning test database for alias 'default' ('test_patchwork')...
mysqldump: Couldn't execute 'SELECT COLUMN_NAME,                       JSON_EXTRACT(HISTOGRAM, '$."number-of-buckets-specified"')                FROM information_schema.COLUMN_STATISTICS                WHERE SCHEMA_NAME = 'test_patchwork' AND TABLE_NAME = 'auth_group';': Unknown table 'COLUMN_STATISTICS' in information_schema (1109)
Cloning test database for alias 'default' ('test_patchwork')...
mysqldump: Couldn't execute 'SELECT COLUMN_NAME,                       JSON_EXTRACT(HISTOGRAM, '$."number-of-buckets-specified"')                FROM information_schema.COLUMN_STATISTICS                WHERE SCHEMA_NAME = 'test_patchwork' AND TABLE_NAME = 'auth_group';': Unknown table 'COLUMN_STATISTICS' in information_schema (1109)
System check identified no issues (0 silenced).
...

The same issue happens on multiple Django versions. With a bit of hacking on the code in the tox venv, I was able to inspect the command that Django is actually executing:

Dumping! cmd = mysqldump --user=patchwork --host=localhost --routines --events test_patchwork

Executing this locally, I see the same issue:

❯ MYSQL_PWD=password mysqldump --user=patchwork --host=localhost --routines --events patchwork
-- MySQL dump 10.13  Distrib 8.0.30, for Linux (x86_64)
--
-- Host: localhost    Database: patchwork
-- ------------------------------------------------------
-- Server version       5.5.5-10.5.16-MariaDB

/*!40101 SET @OLD_CHARACTER_SET_CLIENT=@@CHARACTER_SET_CLIENT */;
/*!40101 SET @OLD_CHARACTER_SET_RESULTS=@@CHARACTER_SET_RESULTS */;
/*!40101 SET @OLD_COLLATION_CONNECTION=@@COLLATION_CONNECTION */;
/*!50503 SET NAMES utf8mb4 */;
/*!40103 SET @OLD_TIME_ZONE=@@TIME_ZONE */;
/*!40103 SET TIME_ZONE='+00:00' */;
/*!40014 SET @OLD_UNIQUE_CHECKS=@@UNIQUE_CHECKS, UNIQUE_CHECKS=0 */;
/*!40014 SET @OLD_FOREIGN_KEY_CHECKS=@@FOREIGN_KEY_CHECKS, FOREIGN_KEY_CHECKS=0 */;
/*!40101 SET @OLD_SQL_MODE=@@SQL_MODE, SQL_MODE='NO_AUTO_VALUE_ON_ZERO' */;
/*!40111 SET @OLD_SQL_NOTES=@@SQL_NOTES, SQL_NOTES=0 */;

--
-- Table structure for table `auth_group`
--

DROP TABLE IF EXISTS `auth_group`;
/*!40101 SET @saved_cs_client     = @@character_set_client */;
/*!50503 SET character_set_client = utf8mb4 */;
CREATE TABLE `auth_group` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `name` varchar(150) NOT NULL,
  PRIMARY KEY (`id`),
  UNIQUE KEY `name` (`name`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1;
/*!40101 SET character_set_client = @saved_cs_client */;

--
-- Dumping data for table `auth_group`
--

LOCK TABLES `auth_group` WRITE;
/*!40000 ALTER TABLE `auth_group` DISABLE KEYS */;
/*!40000 ALTER TABLE `auth_group` ENABLE KEYS */;
UNLOCK TABLES;
mysqldump: Couldn't execute 'SELECT COLUMN_NAME,                       JSON_EXTRACT(HISTOGRAM, '$."number-of-buckets-specified"')                FROM information_schema.COLUMN_STATISTICS                WHERE SCHEMA_NAME = 'patchwork' AND TABLE_NAME = 'auth_group';': Unknown table 'COLUMN_STATISTICS' in information_schema (1109)

❯ echo $?
2

I'm using Fedora 36 with the default packages:

❯ cat /etc/system-release
Fedora release 36 (Thirty Six)

❯ sudo dnf list installed | grep mariadb
mariadb-backup.x86_64                                  3:10.5.16-1.fc36                    @updates
mariadb-common.x86_64                                  3:10.5.16-1.fc36                    @updates
mariadb-connector-c.x86_64                             3.2.7-1.fc36                        @updates
mariadb-connector-c-config.noarch                      3.2.7-1.fc36                        @updates
mariadb-connector-c-devel.x86_64                       3.2.7-1.fc36                        @updates
mariadb-connector-c-doc.noarch                         3.2.7-1.fc36                        @updates
mariadb-cracklib-password-check.x86_64                 3:10.5.16-1.fc36                    @updates
mariadb-devel.x86_64                                   3:10.5.16-1.fc36                    @updates
mariadb-errmsg.x86_64                                  3:10.5.16-1.fc36                    @updates
mariadb-gssapi-server.x86_64                           3:10.5.16-1.fc36                    @updates
mariadb-server.x86_64                                  3:10.5.16-1.fc36                    @updates
mariadb-server-utils.x86_64                            3:10.5.16-1.fc36                    @updates

❯ sudo dnf list installed | grep mysql
community-mysql.x86_64                                 8.0.30-2.fc36                       @updates
community-mysql-common.x86_64                          8.0.30-2.fc36                       @updates
mysql-selinux.noarch                                   1.0.5-1.fc36                        @updates

I had previously reported a similar issue using a combination of mysql-client with an older version of mysql-server or mariadb-server. This was reported as #33537. However, I'm now seeing the issue with the latest versions of both the server and client. As noted in the issue, I can work around this on the command line by adding the --column-statistics=0 argument to mysqldump, however, there doesn't appear to be any mechanism to do this in Django. Alternatively, I can add the following to my.cnf (or equivalent):

/etc/my.cnf.d/mysqldump.cnf
[mysqldump]
column-statistics=0

There's a chance that someone has packaged something wrong in Fedora land and that I'm using an incompatible client and server combo (MySQL-derived client, MariaDB server). However, it's surprising that the tests proceeded to run despite the DB clone failing. As you can see above, mysqldump returned a non-zero exit code so there doesn't appear to be any reason Django couldn't have stopped immediately. I think it would make sense to fail fast at a minimum. If we wanted, we could also add the --column-statistics=0 argument to our call to mysqldump (or provide a mechanism for users to do so), however, this isn't compatible with the 5.7.x version of mysqldump or the version provided by e.g. the mariadb-client package on Ubuntu) so this would have to be a version-specific check.

[Mariusz Felisiak]

Your issue looks like a misconfiguration and such it is still invalid.

Do you want to propose adding check=True to subprocess calls and change this ticket to a cleanup?

[Stephen Finucane]

Replying to Mariusz Felisiak:

Your issue looks like a misconfiguration and such it is still invalid.

Do you want to propose adding check=True to subprocess calls and change this ticket to a cleanup?

Yes, exactly (though not with check=True as I think that's only supported by subprocess.run?). I've submitted a patch that should address this.

[Mariusz Felisiak]

​PR

[Sage Abdullah]

There is a new PR for this, and looks like it hasn't been reviewed. Please correct me if I'm wrong. Thanks!

​PR