Do something for MySQL under Python 3

[Aymeric Augustin]

If MySQLdb isn't ported to Python 3 by the time we release 1.6 alpha, we should either recommend an alternative if there's a solid one, or document that Django cannot be used with MySQL on Python 3.

[Sylvain Leroux]

At this time, the official MYSQLdb development has stalled.

There are several forks available on git-hub in order to port that library for Python 3:

• ​https://github.com/davispuh/MySQL-for-Python-3
• ​https://github.com/clelland/MySQL-for-Python-3
• ...

At first sight, those ports seem to work with django-1.6 (master dev) -- with the minor modification of removing the use_unicode connection param. This modification was already available there: ​https://github.com/vsajip/django/blob/django3/django/db/backends/mysql/base.py#L386
however some sources state that there is a bunch of issues [...] in non-ascii environment:
​http://python.6.n6.nabble.com/MySQL-Django-1-6-Python3-tt5009591.html#a5009904

As of myself, I've done some (very basic) preliminary tests using clelland/MySQL-for-Python-3 alone: Inserting and retrieving data containing non-ASCII characters (é, à, ç, è) appears to work. By examining the output of mysqldump, strings have been correctly encoded at DB level. I performed my tests against MySQL5.1 using both utf8 and latin1 as DB charset. Maybe there is some problems with the integration of both "Django 1.6" and "MySQL for Python 3"?

So, if someone could post here some data/link to the possible issues with MySQL-for-Python-3 & Django, that would help things in order not to drop MySQL support in Python 3 environment.

[Aymeric Augustin]

Mailing-list thread: ​https://groups.google.com/d/msg/django-developers/8r_RVmUe5ys/WKWFy1owlyUJ

Pull request: ​https://github.com/django/django/pull/1044

I'll wait a few days for feedback.

[Aymeric Augustin]

Before merging this, we should install the library we recommend on ci.djangoproject.com and make sure the tests pass.

[Aymeric Augustin]

It turns out that the tests don't pass...

I made a few fixes and updated the pull request. The only remaining problem pertains to BinaryField.

======================================================================
ERROR: test_set_and_retrieve (model_fields.tests.BinaryFieldTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/myk/Documents/dev/django/tests/model_fields/tests.py", line 450, in test_set_and_retrieve
    self.assertEqual(bytes(dm.data), bytes(bdata))
TypeError: string argument without an encoding

(There's also three failures in the serializers_regress tests but they have the same root cause.)

pdb shows that dm.data is "b'\\x00F\\xfe'" instead of b'\x00F\xfe'. In other words, str() or force_str() was incorrectly applied to the value.

[Aymeric Augustin]

This change helps a bit (thanks Claude for spotting the bug):

--- a/MySQLdb/__init__.py
+++ b/MySQLdb/__init__.py
@@ -73,7 +73,7 @@ def test_DBAPISet_set_inequality_membership():
     assert FIELD_TYPE.DATE != STRING
 
 def Binary(x):
-    return str(x)
+    return bytes(x)
 
 def Connect(*args, **kwargs):
     """Factory function for connections.Connection."""

But then another exceptions occurs because MySQL-for-Python-3 assumes that all bytestrings can and should be utf-8 decoded (the point of the str refactor in Python 3 is to forbid this):

======================================================================
ERROR: test_set_and_retrieve (model_fields.tests.BinaryFieldTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/myk/Documents/dev/django/tests/model_fields/tests.py", line 448, in test_set_and_retrieve
    dm.save()
  File "/Users/myk/Documents/dev/django/django/db/models/base.py", line 548, in save
    force_update=force_update, update_fields=update_fields)
  File "/Users/myk/Documents/dev/django/django/db/models/base.py", line 576, in save_base
    updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)
  File "/Users/myk/Documents/dev/django/django/db/models/base.py", line 663, in _save_table
    result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)
  File "/Users/myk/Documents/dev/django/django/db/models/base.py", line 682, in _do_insert
    using=using, raw=raw)
  File "/Users/myk/Documents/dev/django/django/db/models/manager.py", line 226, in _insert
    return insert_query(self.model, objs, fields, **kwargs)
  File "/Users/myk/Documents/dev/django/django/db/models/query.py", line 1580, in insert_query
    return query.get_compiler(using=using).execute_sql(return_id)
  File "/Users/myk/Documents/dev/django/django/db/models/sql/compiler.py", line 884, in execute_sql
    cursor.execute(sql, params)
  File "/Users/myk/Documents/dev/django/django/db/utils.py", line 105, in inner
    return func(*args, **kwargs)
  File "/Users/myk/Documents/dev/django/django/db/backends/mysql/base.py", line 124, in execute
    return self.cursor.execute(query, args)
  File "/Users/myk/Documents/dev/MySQL-for-Python-3/mysql-py3/lib/python3.3/site-packages/MySQL_python-1.2.3-py3.3-macosx-10.8-x86_64.egg/MySQLdb/cursors.py", line 160, in execute
    query = query % db.literal(args)
  File "/Users/myk/Documents/dev/MySQL-for-Python-3/mysql-py3/lib/python3.3/site-packages/MySQL_python-1.2.3-py3.3-macosx-10.8-x86_64.egg/MySQLdb/connections.py", line 241, in literal
    return self.escape(o, self.encoders)
  File "/Users/myk/Documents/dev/MySQL-for-Python-3/mysql-py3/lib/python3.3/site-packages/MySQL_python-1.2.3-py3.3-macosx-10.8-x86_64.egg/MySQLdb/connections.py", line 190, in bytes_literal
    return db.literal(u.decode(bytes_literal.charset))
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xfe in position 2: invalid start byte

I'm starting to doubt the quality of this port. Either no one's using MySQL on Python 3 besides toy projects, or I haven't found the library they're using :/

[Aymeric Augustin]

I've updated the pull request with a commit that marks the affected tests as expected failures.

[Aymeric Augustin <aymeric.augustin@…>]

In e81e319f15f448d550d7e30b204d9490a9999b99:

Fixed #20025 -- Pointed to a MySQLdb fork for Python 3.

Made a few minor compatibility adjustments.

[Aymeric Augustin <aymeric.augustin@…>]

In 86b4ac665afe793a457ae84dfa1dfbbbb7e3c2bf:

[py3] Stopped iterating on exceptions. Refs #20025.

[Aymeric Augustin]

Follow-up ticket for the problem with BinaryFields: #20377.

[travis.jensen@…]

Is there any chance of this getting into 1.5.2? I realize I can play with it on master (and I'll probably start there), but I will have an easier time experimenting with it at work if I can say I'm playing with an official release. More experimenting from me means the test coverage for Django (ok, so maybe not enough to be worth it, but it doesn't hurt to ask :).

[Aymeric Augustin <aymeric.augustin@…>]

In b5d6a5b21a2af4f0b4c3e8e3ca4b44e77812d996:

[1.5.x] [py3] Stopped iterating on exceptions. Refs #20025.

Backport of 86b4ac66 from master.

[Aymeric Augustin]

I've backported the two commits that really matter.

Other commits are just ignoring tests that fail because of bugs in the MySQLdb port.

[Aymeric Augustin <aymeric.augustin@…>]

In 6d3d6081e83295b4d0af8d4dbcf388bebc33ba41:

[1.5.x] Fixed #20025 -- Pointed to a MySQLdb fork for Python 3.

Made a few minor compatibility adjustments.

Backport of e81e319f from master.

[benjaoming]

What about OurSQL? It has a PyPi package, and the django-connector is already written (also with a PyPi package)!

​http://pythonhosted.org/oursql/
​https://github.com/dcramer/django-oursql

I found it in a mailing thread where SQLAlchemy people were saying that the Oracle connector was causing a lot of issues, but OurSQL was the best one they'd found. @aaugustin Can you do a similar testing to OurSQL and django-oursql and see if it's running smoothly? I think it looks very promising, the django-connector even has some limited GIS support.

[Aymeric Augustin]

Sorry, I don't have any interest in this, besides making it possible to release Django 1.6.

[BrianO]

Sorry if I'm being dumb, but this thread leaves me confused:

(*) There's a fork of MySQLdb that works with Python3 and Django 1.5.x? What's its URL?
() Does this fork of MySQLdb work with released Django 1.5.4? how about with released 1.5.1?

Thanks in advance.

[Aymeric Augustin]

See ​https://docs.djangoproject.com/en/1.5/ref/databases/#python-3.