Opened 3 years ago
Last modified 3 months ago
#33537 assigned Cleanup/optimization
Cloning test databases should reraise errors on MySQL.
Reported by: | Stephen Finucane | Owned by: | Faakhir30 |
---|---|---|---|
Component: | Database layer (models, ORM) | Version: | 4.1 |
Severity: | Normal | Keywords: | |
Cc: | Sage Abdullah | Triage Stage: | Accepted |
Has patch: | yes | Needs documentation: | no |
Needs tests: | no | Patch needs improvement: | yes |
Easy pickings: | no | UI/UX: | no |
Description
I've been seeing the following error message when attempting to run tests in parallel using a MySQL backend:
py36-django22 run-test-pre: PYTHONHASHSEED='1711093702' py36-django22 run-test: commands[0] | python /home/patchwork/patchwork/manage.py test --noinput --parallel -- patchwork mysqldump: Couldn't execute 'SELECT COLUMN_NAME, JSON_EXTRACT(HISTOGRAM, '$."number-of-buckets-specified"') FROM information_schema.COLUMN_STATISTICS WHERE SCHEMA_NAME = 'test_patchwork' AND TABLE_NAME = 'auth_group';': Unknown table 'COLUMN_STATISTICS' in information_schema (1109) mysqldump: Couldn't execute 'SELECT COLUMN_NAME, JSON_EXTRACT(HISTOGRAM, '$."number-of-buckets-specified"') FROM information_schema.COLUMN_STATISTICS WHERE SCHEMA_NAME = 'test_patchwork' AND TABLE_NAME = 'auth_group';': Unknown table 'COLUMN_STATISTICS' in information_schema (1109) System check identified no issues (0 silenced). Creating test database for alias 'default'... Cloning test database for alias 'default'... Cloning test database for alias 'default'... ...
This is followed by a lot of test failures due to missing databases.
In my instance, I traced this back to the fact that I was using mariadb-server
with mysql-client
rather than mariadb-server
. Switching the client packages resolved the issue. However, posts on StackOverflow suggest the same issue can occur when using older MySQL servers (5.7?) that don't provide these statistics with newer clients so it's not as clear cut as this.
I'm not sure there's much to be done here from a Django perspective. We could arguably add the -column-statistics=0
flag to the mysqldump
call (in django.db.backends.mysql.creation.DatabaseCreation._clone_db
), but this is not compatible with the version of this tool provided by mariadb-client
. We might also want to make this a more serious error and fail the test run rather than attempt to proceed, obscuring the failure in the process (this could conceivably get flaky if you had a mix of DB and non-DB tests running in different runner instances). Perhaps the best thing to do here is close this and leave it here as a breadcrumb for others that run into the issue.
Change History (13)
comment:1 by , 3 years ago
Resolution: | → invalid |
---|---|
Status: | new → closed |
comment:2 by , 2 years ago
Resolution: | invalid |
---|---|
Status: | closed → new |
Reopening per feedback on #34083
comment:3 by , 2 years ago
Version: | 3.2 → 4.1 |
---|
comment:4 by , 2 years ago
Reproducing from #34083.
I'm attempting to run tests in parallel using MariaDB. When running tests, all access to the additional databases fail. Scrolling back to the top, I see the following error messages:
❯ tox -e py38-django41 -- patchwork.tests.api.test_user patchwork.tests.api.test_event py38-django41 run-test-pre: PYTHONHASHSEED='631429049' py38-django41 run-test: commands[0] | python /home/patchwork/patchwork/manage.py test --noinput --parallel -v 2 --timing -- patchwork.tests.api.test_user patchwork.tests.api.test_event Found 25 test(s). Creating test database for alias 'default' ('test_patchwork')... Operations to perform: Synchronize unmigrated apps: django_filters, humanize, messages, rest_framework, staticfiles Apply all migrations: admin, auth, authtoken, contenttypes, patchwork, sessions, sites Synchronizing apps without migrations: Creating tables... Running deferred SQL... Running migrations: Applying contenttypes.0001_initial... OK ... {skipped} ... Applying sites.0002_alter_domain_unique... OK Cloning test database for alias 'default' ('test_patchwork')... mysqldump: Couldn't execute 'SELECT COLUMN_NAME, JSON_EXTRACT(HISTOGRAM, '$."number-of-buckets-specified"') FROM information_schema.COLUMN_STATISTICS WHERE SCHEMA_NAME = 'test_patchwork' AND TABLE_NAME = 'auth_group';': Unknown table 'COLUMN_STATISTICS' in information_schema (1109) Cloning test database for alias 'default' ('test_patchwork')... mysqldump: Couldn't execute 'SELECT COLUMN_NAME, JSON_EXTRACT(HISTOGRAM, '$."number-of-buckets-specified"') FROM information_schema.COLUMN_STATISTICS WHERE SCHEMA_NAME = 'test_patchwork' AND TABLE_NAME = 'auth_group';': Unknown table 'COLUMN_STATISTICS' in information_schema (1109) System check identified no issues (0 silenced). ...
The same issue happens on multiple Django versions. With a bit of hacking on the code in the tox venv, I was able to inspect the command that Django is actually executing:
Dumping! cmd = mysqldump --user=patchwork --host=localhost --routines --events test_patchwork
Executing this locally, I see the same issue:
❯ MYSQL_PWD=password mysqldump --user=patchwork --host=localhost --routines --events patchwork -- MySQL dump 10.13 Distrib 8.0.30, for Linux (x86_64) -- -- Host: localhost Database: patchwork -- ------------------------------------------------------ -- Server version 5.5.5-10.5.16-MariaDB /*!40101 SET @OLD_CHARACTER_SET_CLIENT=@@CHARACTER_SET_CLIENT */; /*!40101 SET @OLD_CHARACTER_SET_RESULTS=@@CHARACTER_SET_RESULTS */; /*!40101 SET @OLD_COLLATION_CONNECTION=@@COLLATION_CONNECTION */; /*!50503 SET NAMES utf8mb4 */; /*!40103 SET @OLD_TIME_ZONE=@@TIME_ZONE */; /*!40103 SET TIME_ZONE='+00:00' */; /*!40014 SET @OLD_UNIQUE_CHECKS=@@UNIQUE_CHECKS, UNIQUE_CHECKS=0 */; /*!40014 SET @OLD_FOREIGN_KEY_CHECKS=@@FOREIGN_KEY_CHECKS, FOREIGN_KEY_CHECKS=0 */; /*!40101 SET @OLD_SQL_MODE=@@SQL_MODE, SQL_MODE='NO_AUTO_VALUE_ON_ZERO' */; /*!40111 SET @OLD_SQL_NOTES=@@SQL_NOTES, SQL_NOTES=0 */; -- -- Table structure for table `auth_group` -- DROP TABLE IF EXISTS `auth_group`; /*!40101 SET @saved_cs_client = @@character_set_client */; /*!50503 SET character_set_client = utf8mb4 */; CREATE TABLE `auth_group` ( `id` int(11) NOT NULL AUTO_INCREMENT, `name` varchar(150) NOT NULL, PRIMARY KEY (`id`), UNIQUE KEY `name` (`name`) ) ENGINE=InnoDB DEFAULT CHARSET=latin1; /*!40101 SET character_set_client = @saved_cs_client */; -- -- Dumping data for table `auth_group` -- LOCK TABLES `auth_group` WRITE; /*!40000 ALTER TABLE `auth_group` DISABLE KEYS */; /*!40000 ALTER TABLE `auth_group` ENABLE KEYS */; UNLOCK TABLES; mysqldump: Couldn't execute 'SELECT COLUMN_NAME, JSON_EXTRACT(HISTOGRAM, '$."number-of-buckets-specified"') FROM information_schema.COLUMN_STATISTICS WHERE SCHEMA_NAME = 'patchwork' AND TABLE_NAME = 'auth_group';': Unknown table 'COLUMN_STATISTICS' in information_schema (1109) ❯ echo $? 2
I'm using Fedora 36 with the default packages:
❯ cat /etc/system-release Fedora release 36 (Thirty Six) ❯ sudo dnf list installed | grep mariadb mariadb-backup.x86_64 3:10.5.16-1.fc36 @updates mariadb-common.x86_64 3:10.5.16-1.fc36 @updates mariadb-connector-c.x86_64 3.2.7-1.fc36 @updates mariadb-connector-c-config.noarch 3.2.7-1.fc36 @updates mariadb-connector-c-devel.x86_64 3.2.7-1.fc36 @updates mariadb-connector-c-doc.noarch 3.2.7-1.fc36 @updates mariadb-cracklib-password-check.x86_64 3:10.5.16-1.fc36 @updates mariadb-devel.x86_64 3:10.5.16-1.fc36 @updates mariadb-errmsg.x86_64 3:10.5.16-1.fc36 @updates mariadb-gssapi-server.x86_64 3:10.5.16-1.fc36 @updates mariadb-server.x86_64 3:10.5.16-1.fc36 @updates mariadb-server-utils.x86_64 3:10.5.16-1.fc36 @updates ❯ sudo dnf list installed | grep mysql community-mysql.x86_64 8.0.30-2.fc36 @updates community-mysql-common.x86_64 8.0.30-2.fc36 @updates mysql-selinux.noarch 1.0.5-1.fc36 @updates
I had previously reported a similar issue using a combination of mysql-client
with an older version of mysql-server
or mariadb-server
. This was reported as #33537. However, I'm now seeing the issue with the latest versions of both the server and client. As noted in the issue, I can work around this on the command line by adding the --column-statistics=0
argument to mysqldump
, however, there doesn't appear to be any mechanism to do this in Django. Alternatively, I can add the following to my.cnf
(or equivalent):
/etc/my.cnf.d/mysqldump.cnf [mysqldump] column-statistics=0
There's a chance that someone has packaged something wrong in Fedora land and that I'm using an incompatible client and server combo (MySQL-derived client, MariaDB server). However, it's surprising that the tests proceeded to run despite the DB clone failing. As you can see above, mysqldump
returned a non-zero exit code so there doesn't appear to be any reason Django couldn't have stopped immediately. I think it would make sense to fail fast at a minimum. If we wanted, we could also add the --column-statistics=0
argument to our call to mysqldump
(or provide a mechanism for users to do so), however, this isn't compatible with the 5.7.x version of mysqldump
or the version provided by e.g. the mariadb-client
package on Ubuntu) so this would have to be a version-specific check.
follow-up: 6 comment:5 by , 2 years ago
Your issue looks like a misconfiguration and such it is still invalid.
Do you want to propose adding check=True
to subprocess calls and change this ticket to a cleanup?
comment:6 by , 2 years ago
Replying to Mariusz Felisiak:
Your issue looks like a misconfiguration and such it is still invalid.
Do you want to propose adding
check=True
to subprocess calls and change this ticket to a cleanup?
Yes, exactly (though not with check=True
as I think that's only supported by subprocess.run
?). I've submitted a patch that should address this.
comment:7 by , 2 years ago
Summary: | Cloning test database fails with mysql-client 8.x and older mysql-server or mariadb-server → Cloning test databases should reraise errors on MySQL. |
---|---|
Triage Stage: | Unreviewed → Accepted |
Type: | Bug → Cleanup/optimization |
comment:8 by , 22 months ago
Has patch: | set |
---|---|
Owner: | changed from | to
Patch needs improvement: | set |
Status: | new → assigned |
comment:9 by , 16 months ago
Owner: | removed |
---|---|
Status: | assigned → new |
comment:10 by , 3 months ago
Cc: | added |
---|
comment:11 by , 3 months ago
Patch needs improvement: | unset |
---|
There is a new PR for this, and looks like it hasn't been reviewed. Please correct me if I'm wrong. Thanks!
comment:12 by , 3 months ago
Owner: | set to |
---|---|
Status: | new → assigned |
comment:13 by , 3 months ago
Patch needs improvement: | set |
---|
Thanks for this ticket, however Django 2.2 doesn't officially support MariaDB. Moreover Django 2.2 is in extended support so it doesn't receive bugfixes anymore (except security issues). Please reopen this ticket if you can reproduce the issue on Django 4.0 (or the current
main
branch). I couldn't reproduce it with MySQL 8+ or MariaDB.