Code

Opened 16 months ago

Closed 14 months ago

Last modified 10 months ago

#20025 closed Bug (fixed)

Do something for MySQL under Python 3

Reported by: aaugustin Owned by: aaugustin
Component: Python 3 Version: master
Severity: Release blocker Keywords:
Cc: Triage Stage: Accepted
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description (last modified by aaugustin)

If MySQLdb isn't ported to Python 3 by the time we release 1.6 alpha, we should either recommend an alternative if there's a solid one, or document that Django cannot be used with MySQL on Python 3.

Attachments (0)

Change History (18)

comment:1 Changed 16 months ago by aaugustin

  • Description modified (diff)

comment:2 Changed 15 months ago by sleroux

At this time, the official MYSQLdb development has stalled.

There are several forks available on git-hub in order to port that library for Python 3:

At first sight, those ports seem to work with django-1.6 (master dev) -- with the minor modification of removing the use_unicode connection param. This modification was already available there: https://github.com/vsajip/django/blob/django3/django/db/backends/mysql/base.py#L386
however some sources state that there is a bunch of issues [...] in non-ascii environment:
http://python.6.n6.nabble.com/MySQL-Django-1-6-Python3-tt5009591.html#a5009904

As of myself, I've done some (very basic) preliminary tests using clelland/MySQL-for-Python-3 alone: Inserting and retrieving data containing non-ASCII characters (é, à, ç, è) appears to work. By examining the output of mysqldump, strings have been correctly encoded at DB level. I performed my tests against MySQL5.1 using both utf8 and latin1 as DB charset. Maybe there is some problems with the integration of both "Django 1.6" and "MySQL for Python 3"?

So, if someone could post here some data/link to the possible issues with MySQL-for-Python-3 & Django, that would help things in order not to drop MySQL support in Python 3 environment.

Last edited 15 months ago by sleroux (previous) (diff)

comment:3 Changed 14 months ago by aaugustin

  • Has patch set
  • Owner changed from nobody to aaugustin
  • Status changed from new to assigned
  • Triage Stage changed from Accepted to Ready for checkin

comment:4 Changed 14 months ago by aaugustin

Before merging this, we should install the library we recommend on ci.djangoproject.com and make sure the tests pass.

comment:5 Changed 14 months ago by aaugustin

  • Patch needs improvement set
  • Triage Stage changed from Ready for checkin to Accepted

It turns out that the tests don't pass...

I made a few fixes and updated the pull request. The only remaining problem pertains to BinaryField.

======================================================================
ERROR: test_set_and_retrieve (model_fields.tests.BinaryFieldTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/myk/Documents/dev/django/tests/model_fields/tests.py", line 450, in test_set_and_retrieve
    self.assertEqual(bytes(dm.data), bytes(bdata))
TypeError: string argument without an encoding

(There's also three failures in the serializers_regress tests but they have the same root cause.)

pdb shows that dm.data is "b'\\x00F\\xfe'" instead of b'\x00F\xfe'. In other words, str() or force_str() was incorrectly applied to the value.

comment:6 Changed 14 months ago by aaugustin

This change helps a bit (thanks Claude for spotting the bug):

--- a/MySQLdb/__init__.py
+++ b/MySQLdb/__init__.py
@@ -73,7 +73,7 @@ def test_DBAPISet_set_inequality_membership():
     assert FIELD_TYPE.DATE != STRING
 
 def Binary(x):
-    return str(x)
+    return bytes(x)
 
 def Connect(*args, **kwargs):
     """Factory function for connections.Connection."""

But then another exceptions occurs because MySQL-for-Python-3 assumes that all bytestrings can and should be utf-8 decoded (the point of the str refactor in Python 3 is to forbid this):

======================================================================
ERROR: test_set_and_retrieve (model_fields.tests.BinaryFieldTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/myk/Documents/dev/django/tests/model_fields/tests.py", line 448, in test_set_and_retrieve
    dm.save()
  File "/Users/myk/Documents/dev/django/django/db/models/base.py", line 548, in save
    force_update=force_update, update_fields=update_fields)
  File "/Users/myk/Documents/dev/django/django/db/models/base.py", line 576, in save_base
    updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)
  File "/Users/myk/Documents/dev/django/django/db/models/base.py", line 663, in _save_table
    result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)
  File "/Users/myk/Documents/dev/django/django/db/models/base.py", line 682, in _do_insert
    using=using, raw=raw)
  File "/Users/myk/Documents/dev/django/django/db/models/manager.py", line 226, in _insert
    return insert_query(self.model, objs, fields, **kwargs)
  File "/Users/myk/Documents/dev/django/django/db/models/query.py", line 1580, in insert_query
    return query.get_compiler(using=using).execute_sql(return_id)
  File "/Users/myk/Documents/dev/django/django/db/models/sql/compiler.py", line 884, in execute_sql
    cursor.execute(sql, params)
  File "/Users/myk/Documents/dev/django/django/db/utils.py", line 105, in inner
    return func(*args, **kwargs)
  File "/Users/myk/Documents/dev/django/django/db/backends/mysql/base.py", line 124, in execute
    return self.cursor.execute(query, args)
  File "/Users/myk/Documents/dev/MySQL-for-Python-3/mysql-py3/lib/python3.3/site-packages/MySQL_python-1.2.3-py3.3-macosx-10.8-x86_64.egg/MySQLdb/cursors.py", line 160, in execute
    query = query % db.literal(args)
  File "/Users/myk/Documents/dev/MySQL-for-Python-3/mysql-py3/lib/python3.3/site-packages/MySQL_python-1.2.3-py3.3-macosx-10.8-x86_64.egg/MySQLdb/connections.py", line 241, in literal
    return self.escape(o, self.encoders)
  File "/Users/myk/Documents/dev/MySQL-for-Python-3/mysql-py3/lib/python3.3/site-packages/MySQL_python-1.2.3-py3.3-macosx-10.8-x86_64.egg/MySQLdb/connections.py", line 190, in bytes_literal
    return db.literal(u.decode(bytes_literal.charset))
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xfe in position 2: invalid start byte

I'm starting to doubt the quality of this port. Either no one's using MySQL on Python 3 besides toy projects, or I haven't found the library they're using :/

comment:7 Changed 14 months ago by aaugustin

  • Patch needs improvement unset

I've updated the pull request with a commit that marks the affected tests as expected failures.

comment:8 Changed 14 months ago by Aymeric Augustin <aymeric.augustin@…>

  • Resolution set to fixed
  • Status changed from assigned to closed

In e81e319f15f448d550d7e30b204d9490a9999b99:

Fixed #20025 -- Pointed to a MySQLdb fork for Python 3.

Made a few minor compatibility adjustments.

comment:9 Changed 14 months ago by Aymeric Augustin <aymeric.augustin@…>

In 86b4ac665afe793a457ae84dfa1dfbbbb7e3c2bf:

[py3] Stopped iterating on exceptions. Refs #20025.

comment:10 Changed 14 months ago by aaugustin

Follow-up ticket for the problem with BinaryFields: #20377.

comment:11 Changed 14 months ago by travis.jensen@…

Is there any chance of this getting into 1.5.2? I realize I can play with it on master (and I'll probably start there), but I will have an easier time experimenting with it at work if I can say I'm playing with an official release. More experimenting from me means the test coverage for Django (ok, so maybe not enough to be worth it, but it doesn't hurt to ask :).

comment:12 Changed 14 months ago by Aymeric Augustin <aymeric.augustin@…>

In b5d6a5b21a2af4f0b4c3e8e3ca4b44e77812d996:

[1.5.x] [py3] Stopped iterating on exceptions. Refs #20025.

Backport of 86b4ac66 from master.

comment:13 Changed 14 months ago by aaugustin

I've backported the two commits that really matter.

Other commits are just ignoring tests that fail because of bugs in the MySQLdb port.

Last edited 14 months ago by aaugustin (previous) (diff)

comment:14 Changed 14 months ago by Aymeric Augustin <aymeric.augustin@…>

In 6d3d6081e83295b4d0af8d4dbcf388bebc33ba41:

[1.5.x] Fixed #20025 -- Pointed to a MySQLdb fork for Python 3.

Made a few minor compatibility adjustments.

Backport of e81e319f from master.

comment:15 Changed 12 months ago by benjaoming

What about OurSQL? It has a PyPi package, and the django-connector is already written (also with a PyPi package)!

http://pythonhosted.org/oursql/
https://github.com/dcramer/django-oursql

I found it in a mailing thread where SQLAlchemy people were saying that the Oracle connector was causing a lot of issues, but OurSQL was the best one they'd found. @aaugustin Can you do a similar testing to OurSQL and django-oursql and see if it's running smoothly? I think it looks very promising, the django-connector even has some limited GIS support.

comment:16 Changed 12 months ago by aaugustin

Sorry, I don't have any interest in this, besides making it possible to release Django 1.6.

comment:17 Changed 10 months ago by BrianO

Sorry if I'm being dumb, but this thread leaves me confused:

(*) There's a fork of MySQLdb that works with Python3 and Django 1.5.x? What's its URL?
() Does this fork of MySQLdb work with released Django 1.5.4? how about with released 1.5.1?

Thanks in advance.

Add Comment

Modify Ticket

Change Properties
<Author field>
Action
as closed
as The resolution will be set. Next status will be 'closed'
The resolution will be deleted. Next status will be 'new'
Author


E-mail address and user name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.