Opened 3 years ago

Closed 3 years ago

Last modified 3 years ago

#20025 closed Bug (fixed)

Do something for MySQL under Python 3

Reported by: aaugustin Owned by: aaugustin
Component: Python 3 Version: master
Severity: Release blocker Keywords:
Cc: Triage Stage: Accepted
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description (last modified by aaugustin)

If MySQLdb isn't ported to Python 3 by the time we release 1.6 alpha, we should either recommend an alternative if there's a solid one, or document that Django cannot be used with MySQL on Python 3.

Change History (18)

comment:1 Changed 3 years ago by aaugustin

  • Description modified (diff)

comment:2 Changed 3 years ago by sleroux

At this time, the official MYSQLdb development has stalled.

There are several forks available on git-hub in order to port that library for Python 3:

At first sight, those ports seem to work with django-1.6 (master dev) -- with the minor modification of removing the use_unicode connection param. This modification was already available there:
however some sources state that there is a bunch of issues [...] in non-ascii environment:

As of myself, I've done some (very basic) preliminary tests using clelland/MySQL-for-Python-3 alone: Inserting and retrieving data containing non-ASCII characters (é, à, ç, è) appears to work. By examining the output of mysqldump, strings have been correctly encoded at DB level. I performed my tests against MySQL5.1 using both utf8 and latin1 as DB charset. Maybe there is some problems with the integration of both "Django 1.6" and "MySQL for Python 3"?

So, if someone could post here some data/link to the possible issues with MySQL-for-Python-3 & Django, that would help things in order not to drop MySQL support in Python 3 environment.

Last edited 3 years ago by sleroux (previous) (diff)

comment:3 Changed 3 years ago by aaugustin

  • Has patch set
  • Owner changed from nobody to aaugustin
  • Status changed from new to assigned
  • Triage Stage changed from Accepted to Ready for checkin

comment:4 Changed 3 years ago by aaugustin

Before merging this, we should install the library we recommend on and make sure the tests pass.

comment:5 Changed 3 years ago by aaugustin

  • Patch needs improvement set
  • Triage Stage changed from Ready for checkin to Accepted

It turns out that the tests don't pass...

I made a few fixes and updated the pull request. The only remaining problem pertains to BinaryField.

ERROR: test_set_and_retrieve (model_fields.tests.BinaryFieldTests)
Traceback (most recent call last):
  File "/Users/myk/Documents/dev/django/tests/model_fields/", line 450, in test_set_and_retrieve
    self.assertEqual(bytes(, bytes(bdata))
TypeError: string argument without an encoding

(There's also three failures in the serializers_regress tests but they have the same root cause.)

pdb shows that is "b'\\x00F\\xfe'" instead of b'\x00F\xfe'. In other words, str() or force_str() was incorrectly applied to the value.

comment:6 Changed 3 years ago by aaugustin

This change helps a bit (thanks Claude for spotting the bug):

--- a/MySQLdb/
+++ b/MySQLdb/
@@ -73,7 +73,7 @@ def test_DBAPISet_set_inequality_membership():
 def Binary(x):
-    return str(x)
+    return bytes(x)
 def Connect(*args, **kwargs):
     """Factory function for connections.Connection."""

But then another exceptions occurs because MySQL-for-Python-3 assumes that all bytestrings can and should be utf-8 decoded (the point of the str refactor in Python 3 is to forbid this):

ERROR: test_set_and_retrieve (model_fields.tests.BinaryFieldTests)
Traceback (most recent call last):
  File "/Users/myk/Documents/dev/django/tests/model_fields/", line 448, in test_set_and_retrieve
  File "/Users/myk/Documents/dev/django/django/db/models/", line 548, in save
    force_update=force_update, update_fields=update_fields)
  File "/Users/myk/Documents/dev/django/django/db/models/", line 576, in save_base
    updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)
  File "/Users/myk/Documents/dev/django/django/db/models/", line 663, in _save_table
    result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)
  File "/Users/myk/Documents/dev/django/django/db/models/", line 682, in _do_insert
    using=using, raw=raw)
  File "/Users/myk/Documents/dev/django/django/db/models/", line 226, in _insert
    return insert_query(self.model, objs, fields, **kwargs)
  File "/Users/myk/Documents/dev/django/django/db/models/", line 1580, in insert_query
    return query.get_compiler(using=using).execute_sql(return_id)
  File "/Users/myk/Documents/dev/django/django/db/models/sql/", line 884, in execute_sql
    cursor.execute(sql, params)
  File "/Users/myk/Documents/dev/django/django/db/", line 105, in inner
    return func(*args, **kwargs)
  File "/Users/myk/Documents/dev/django/django/db/backends/mysql/", line 124, in execute
    return self.cursor.execute(query, args)
  File "/Users/myk/Documents/dev/MySQL-for-Python-3/mysql-py3/lib/python3.3/site-packages/MySQL_python-1.2.3-py3.3-macosx-10.8-x86_64.egg/MySQLdb/", line 160, in execute
    query = query % db.literal(args)
  File "/Users/myk/Documents/dev/MySQL-for-Python-3/mysql-py3/lib/python3.3/site-packages/MySQL_python-1.2.3-py3.3-macosx-10.8-x86_64.egg/MySQLdb/", line 241, in literal
    return self.escape(o, self.encoders)
  File "/Users/myk/Documents/dev/MySQL-for-Python-3/mysql-py3/lib/python3.3/site-packages/MySQL_python-1.2.3-py3.3-macosx-10.8-x86_64.egg/MySQLdb/", line 190, in bytes_literal
    return db.literal(u.decode(bytes_literal.charset))
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xfe in position 2: invalid start byte

I'm starting to doubt the quality of this port. Either no one's using MySQL on Python 3 besides toy projects, or I haven't found the library they're using :/

comment:7 Changed 3 years ago by aaugustin

  • Patch needs improvement unset

I've updated the pull request with a commit that marks the affected tests as expected failures.

comment:8 Changed 3 years ago by Aymeric Augustin <aymeric.augustin@…>

  • Resolution set to fixed
  • Status changed from assigned to closed

In e81e319f15f448d550d7e30b204d9490a9999b99:

Fixed #20025 -- Pointed to a MySQLdb fork for Python 3.

Made a few minor compatibility adjustments.

comment:9 Changed 3 years ago by Aymeric Augustin <aymeric.augustin@…>

In 86b4ac665afe793a457ae84dfa1dfbbbb7e3c2bf:

[py3] Stopped iterating on exceptions. Refs #20025.

comment:10 Changed 3 years ago by aaugustin

Follow-up ticket for the problem with BinaryFields: #20377.

comment:11 Changed 3 years ago by travis.jensen@…

Is there any chance of this getting into 1.5.2? I realize I can play with it on master (and I'll probably start there), but I will have an easier time experimenting with it at work if I can say I'm playing with an official release. More experimenting from me means the test coverage for Django (ok, so maybe not enough to be worth it, but it doesn't hurt to ask :).

comment:12 Changed 3 years ago by Aymeric Augustin <aymeric.augustin@…>

In b5d6a5b21a2af4f0b4c3e8e3ca4b44e77812d996:

[1.5.x] [py3] Stopped iterating on exceptions. Refs #20025.

Backport of 86b4ac66 from master.

comment:13 Changed 3 years ago by aaugustin

I've backported the two commits that really matter.

Other commits are just ignoring tests that fail because of bugs in the MySQLdb port.

Last edited 3 years ago by aaugustin (previous) (diff)

comment:14 Changed 3 years ago by Aymeric Augustin <aymeric.augustin@…>

In 6d3d6081e83295b4d0af8d4dbcf388bebc33ba41:

[1.5.x] Fixed #20025 -- Pointed to a MySQLdb fork for Python 3.

Made a few minor compatibility adjustments.

Backport of e81e319f from master.

comment:15 Changed 3 years ago by benjaoming

What about OurSQL? It has a PyPi package, and the django-connector is already written (also with a PyPi package)!

I found it in a mailing thread where SQLAlchemy people were saying that the Oracle connector was causing a lot of issues, but OurSQL was the best one they'd found. @aaugustin Can you do a similar testing to OurSQL and django-oursql and see if it's running smoothly? I think it looks very promising, the django-connector even has some limited GIS support.

comment:16 Changed 3 years ago by aaugustin

Sorry, I don't have any interest in this, besides making it possible to release Django 1.6.

comment:17 Changed 3 years ago by BrianO

Sorry if I'm being dumb, but this thread leaves me confused:

(*) There's a fork of MySQLdb that works with Python3 and Django 1.5.x? What's its URL?
() Does this fork of MySQLdb work with released Django 1.5.4? how about with released 1.5.1?

Thanks in advance.

Note: See TracTickets for help on using tickets.
Back to Top