Opened 11 years ago

Closed 11 years ago

Last modified 10 years ago

#20025 closed Bug (fixed)

Do something for MySQL under Python 3

Reported by: Aymeric Augustin Owned by: Aymeric Augustin
Component: Python 3 Version: dev
Severity: Release blocker Keywords:
Cc: Triage Stage: Accepted
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description (last modified by Aymeric Augustin)

If MySQLdb isn't ported to Python 3 by the time we release 1.6 alpha, we should either recommend an alternative if there's a solid one, or document that Django cannot be used with MySQL on Python 3.

Change History (18)

comment:1 by Aymeric Augustin, 11 years ago

Description: modified (diff)

comment:2 by Sylvain Leroux, 11 years ago

At this time, the official MYSQLdb development has stalled.

There are several forks available on git-hub in order to port that library for Python 3:

At first sight, those ports seem to work with django-1.6 (master dev) -- with the minor modification of removing the use_unicode connection param. This modification was already available there: https://github.com/vsajip/django/blob/django3/django/db/backends/mysql/base.py#L386
however some sources state that there is a bunch of issues [...] in non-ascii environment:
http://python.6.n6.nabble.com/MySQL-Django-1-6-Python3-tt5009591.html#a5009904

As of myself, I've done some (very basic) preliminary tests using clelland/MySQL-for-Python-3 alone: Inserting and retrieving data containing non-ASCII characters (é, à, ç, è) appears to work. By examining the output of mysqldump, strings have been correctly encoded at DB level. I performed my tests against MySQL5.1 using both utf8 and latin1 as DB charset. Maybe there is some problems with the integration of both "Django 1.6" and "MySQL for Python 3"?

So, if someone could post here some data/link to the possible issues with MySQL-for-Python-3 & Django, that would help things in order not to drop MySQL support in Python 3 environment.

Last edited 11 years ago by Sylvain Leroux (previous) (diff)

comment:3 by Aymeric Augustin, 11 years ago

Has patch: set
Owner: changed from nobody to Aymeric Augustin
Status: newassigned
Triage Stage: AcceptedReady for checkin

comment:4 by Aymeric Augustin, 11 years ago

Before merging this, we should install the library we recommend on ci.djangoproject.com and make sure the tests pass.

comment:5 by Aymeric Augustin, 11 years ago

Patch needs improvement: set
Triage Stage: Ready for checkinAccepted

It turns out that the tests don't pass...

I made a few fixes and updated the pull request. The only remaining problem pertains to BinaryField.

======================================================================
ERROR: test_set_and_retrieve (model_fields.tests.BinaryFieldTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/myk/Documents/dev/django/tests/model_fields/tests.py", line 450, in test_set_and_retrieve
    self.assertEqual(bytes(dm.data), bytes(bdata))
TypeError: string argument without an encoding

(There's also three failures in the serializers_regress tests but they have the same root cause.)

pdb shows that dm.data is "b'\\x00F\\xfe'" instead of b'\x00F\xfe'. In other words, str() or force_str() was incorrectly applied to the value.

comment:6 by Aymeric Augustin, 11 years ago

This change helps a bit (thanks Claude for spotting the bug):

--- a/MySQLdb/__init__.py
+++ b/MySQLdb/__init__.py
@@ -73,7 +73,7 @@ def test_DBAPISet_set_inequality_membership():
     assert FIELD_TYPE.DATE != STRING
 
 def Binary(x):
-    return str(x)
+    return bytes(x)
 
 def Connect(*args, **kwargs):
     """Factory function for connections.Connection."""

But then another exceptions occurs because MySQL-for-Python-3 assumes that all bytestrings can and should be utf-8 decoded (the point of the str refactor in Python 3 is to forbid this):

======================================================================
ERROR: test_set_and_retrieve (model_fields.tests.BinaryFieldTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/myk/Documents/dev/django/tests/model_fields/tests.py", line 448, in test_set_and_retrieve
    dm.save()
  File "/Users/myk/Documents/dev/django/django/db/models/base.py", line 548, in save
    force_update=force_update, update_fields=update_fields)
  File "/Users/myk/Documents/dev/django/django/db/models/base.py", line 576, in save_base
    updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)
  File "/Users/myk/Documents/dev/django/django/db/models/base.py", line 663, in _save_table
    result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)
  File "/Users/myk/Documents/dev/django/django/db/models/base.py", line 682, in _do_insert
    using=using, raw=raw)
  File "/Users/myk/Documents/dev/django/django/db/models/manager.py", line 226, in _insert
    return insert_query(self.model, objs, fields, **kwargs)
  File "/Users/myk/Documents/dev/django/django/db/models/query.py", line 1580, in insert_query
    return query.get_compiler(using=using).execute_sql(return_id)
  File "/Users/myk/Documents/dev/django/django/db/models/sql/compiler.py", line 884, in execute_sql
    cursor.execute(sql, params)
  File "/Users/myk/Documents/dev/django/django/db/utils.py", line 105, in inner
    return func(*args, **kwargs)
  File "/Users/myk/Documents/dev/django/django/db/backends/mysql/base.py", line 124, in execute
    return self.cursor.execute(query, args)
  File "/Users/myk/Documents/dev/MySQL-for-Python-3/mysql-py3/lib/python3.3/site-packages/MySQL_python-1.2.3-py3.3-macosx-10.8-x86_64.egg/MySQLdb/cursors.py", line 160, in execute
    query = query % db.literal(args)
  File "/Users/myk/Documents/dev/MySQL-for-Python-3/mysql-py3/lib/python3.3/site-packages/MySQL_python-1.2.3-py3.3-macosx-10.8-x86_64.egg/MySQLdb/connections.py", line 241, in literal
    return self.escape(o, self.encoders)
  File "/Users/myk/Documents/dev/MySQL-for-Python-3/mysql-py3/lib/python3.3/site-packages/MySQL_python-1.2.3-py3.3-macosx-10.8-x86_64.egg/MySQLdb/connections.py", line 190, in bytes_literal
    return db.literal(u.decode(bytes_literal.charset))
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xfe in position 2: invalid start byte

I'm starting to doubt the quality of this port. Either no one's using MySQL on Python 3 besides toy projects, or I haven't found the library they're using :/

comment:7 by Aymeric Augustin, 11 years ago

Patch needs improvement: unset

I've updated the pull request with a commit that marks the affected tests as expected failures.

comment:8 by Aymeric Augustin <aymeric.augustin@…>, 11 years ago

Resolution: fixed
Status: assignedclosed

In e81e319f15f448d550d7e30b204d9490a9999b99:

Fixed #20025 -- Pointed to a MySQLdb fork for Python 3.

Made a few minor compatibility adjustments.

comment:9 by Aymeric Augustin <aymeric.augustin@…>, 11 years ago

In 86b4ac665afe793a457ae84dfa1dfbbbb7e3c2bf:

[py3] Stopped iterating on exceptions. Refs #20025.

comment:10 by Aymeric Augustin, 11 years ago

Follow-up ticket for the problem with BinaryFields: #20377.

comment:11 by travis.jensen@…, 11 years ago

Is there any chance of this getting into 1.5.2? I realize I can play with it on master (and I'll probably start there), but I will have an easier time experimenting with it at work if I can say I'm playing with an official release. More experimenting from me means the test coverage for Django (ok, so maybe not enough to be worth it, but it doesn't hurt to ask :).

comment:12 by Aymeric Augustin <aymeric.augustin@…>, 11 years ago

In b5d6a5b21a2af4f0b4c3e8e3ca4b44e77812d996:

[1.5.x] [py3] Stopped iterating on exceptions. Refs #20025.

Backport of 86b4ac66 from master.

comment:13 by Aymeric Augustin, 11 years ago

I've backported the two commits that really matter.

Other commits are just ignoring tests that fail because of bugs in the MySQLdb port.

Last edited 11 years ago by Aymeric Augustin (previous) (diff)

comment:14 by Aymeric Augustin <aymeric.augustin@…>, 11 years ago

In 6d3d6081e83295b4d0af8d4dbcf388bebc33ba41:

[1.5.x] Fixed #20025 -- Pointed to a MySQLdb fork for Python 3.

Made a few minor compatibility adjustments.

Backport of e81e319f from master.

comment:15 by benjaoming, 11 years ago

What about OurSQL? It has a PyPi package, and the django-connector is already written (also with a PyPi package)!

http://pythonhosted.org/oursql/
https://github.com/dcramer/django-oursql

I found it in a mailing thread where SQLAlchemy people were saying that the Oracle connector was causing a lot of issues, but OurSQL was the best one they'd found. @aaugustin Can you do a similar testing to OurSQL and django-oursql and see if it's running smoothly? I think it looks very promising, the django-connector even has some limited GIS support.

comment:16 by Aymeric Augustin, 11 years ago

Sorry, I don't have any interest in this, besides making it possible to release Django 1.6.

comment:17 by BrianO, 10 years ago

Sorry if I'm being dumb, but this thread leaves me confused:

(*) There's a fork of MySQLdb that works with Python3 and Django 1.5.x? What's its URL?
() Does this fork of MySQLdb work with released Django 1.5.4? how about with released 1.5.1?

Thanks in advance.

Note: See TracTickets for help on using tickets.
Back to Top