In some cases MySQLdb returns array.array which causes an error in a2b_base64()

[alexander.pugachev@…]

settings.py

INSTALLED_APPS = (
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.sites',
    'mysite.polls',
    'django.contrib.admin'
)

urls.py

urlpatterns = patterns('',
    # Example:
    # (r'^mysite/', include('mysite.apps.foo.urls.foo')),

    # Uncomment this for admin:
     (r'^admin/', include('django.contrib.admin.urls')),
)

db synced, server restarted, got admin area login form at /admin/ local url
logged in as superuser, got error:

TypeError at /admin/
a2b_base64() argument 1 must be string or read-only character buffer, not array.array
Request Method: 	GET
Request URL: 	http://127.0.0.1:8000/admin/
Exception Type: 	TypeError
Exception Value: 	a2b_base64() argument 1 must be string or read-only character buffer, not array.array
Exception Location: 	c:\python24\lib\base64.py in decodestring, line 319

[anonymous]

indeed it was this project

[Chris Beaven]

Thanks for the ticket Alexander, but I'm pretty sure that we'd have more reports of this in 7 months if it was still a problem.
If you can provide some more insight, feel free to reopen.

[anonymous]

I (different anonymous) also get this problem. I am using mysql5.0.33, apache 2.2.4, python 2.4.4, django 0.95.1 on freebsd 6.2. I am trying to figure it out. I tried with firefox and also with lynx, using the apache server and also the native django server. That exception handler is amazing though! Here's some more output:

/usr/local/lib/python2.4/site-packages/Django-0.95.1-py2.4.egg/django/contrib/sessions/models.py in get_decoded

60. def get_decoded(self):
61. encoded_data = base64.decodestring(self.session_data) ...

Local vars

Variable Value
self <Session: Session object>

/usr/local/lib/python2.4/base64.py in decodestring

319. def decodestring(s):
320. """Decode a string."""
321. return binascii.a2b_base64(s)

Local vars

Variable Value
s array('c', 'KGRwMQpTJ3Rlc3Rjb29raWUnCnAyClMnd29ya2VkJwpwMwpzLmU1ZjhhN2QwYzQxYWJlY2RjMWYx\nNDQ3MzUzZjY4YzQw\n')

If anyone wants to inspect my installation, please email me at rbancroft at gmail dot com and I can show you. Also, I did not follow the tutorial 100% as written so there could be something I am missing.

[anonymous]

so I put everything in a new database letting django create all of the database tables with syncdb and it worked. one difference is perhaps that my own databases uses the utf-bin charset, whereas when django created it, it used latin_swedish_ci?

[Chris Beaven]

Thanks for the report, anonymous. If that's the case then it's a very different ticket to this one.

Perhaps someone could investigate and open a new ticket?

[anonymous]

closed, reopened, invalid, whatever. just an anonymous comment regarding the error:

mysql driver has some strange habit of returning byte arrays as array.array, causing lots of different headaches in various places. patching the mysql driver to return str (as django seems to expext, possibly utf8 encoded) fixes all of that.

[James Bennett]

It appears from #3424 that this is still happening; reopening here and marking that one as a dupe since the comment above has a theory on how to fix it.

[LightLan@…]

I just tried with a newer svn(4750) and the error still happens.
I initialized the database from scratch using ./manage.py syncdb.
It's the same error, as before.

[Simon G. <dev@…>]

To recap:

From the anonymous comment above, it has been suggested that the solution is to get the second tuple value from any arrays returned by the MySQL backend.

It would be nice to work out when MySQLdb decides to do this - it looks like this is a charset issue (moving to latin-1 from utf-8 fixed this).

[django@…]

Index: models.py
===================================================================
--- models.py   (revision 4734)
+++ models.py   (working copy)
@@ -58,7 +58,7 @@
         verbose_name_plural = _('sessions')

     def get_decoded(self):
-        encoded_data = base64.decodestring(self.session_data)
+        encoded_data = base64.decodestring(self.session_data.tostring())
         pickled, tamper_check = encoded_data[:-32], encoded_data[-32:]
         if md5.new(pickled + settings.SECRET_KEY).hexdigest() != tamper_check:
             from django.core.exceptions import SuspiciousOperation

Sorry, but I'am not found anything for send path.

[Malcolm Tredinnick]

The patch fragment in the previous comment can't be correct: it assumes that self.session_data is always going to be something that has a tostring() method. However, the reason that this bug is only happening for some people is because this value is _usually_ a string, not an array. So we cannot do any kind of universal conversion here.

[Simon G. <dev@…>]

This is also affecting the second tutorial (#3783), so we should try to fix this ASAP.

This is a rather ugly bruteforce way of doing it:

if hasattr(self.session_data, 'tostring):
   self.session_data = self.session_data.tostring()
encoded_data = base64.decodestring(self.session_data)

but I'll try to dig through MySQLdb tomorrow and see when it decides to return an array (it appears to be charset contingent)

[Simon G. <dev@…>]

What I think is happening is that one of the converters is returning an array when we don't want it to. I don't have the bug (I'm using UTF-8 everywhere), and my converters look like this:

>>> from django.conf import settings as s
>>> import MySQLdb
>>> db = MySQLdb.connect(host=s.DATABASE_HOST, user=s.DATABASE_USER, passwd=s.DATABASE_PASSWORD, db=s.DATABASE_NAME)
>>> db.converter

{0: <class 'decimal.Decimal'>,
 1: <type 'int'>,
 2: <type 'int'>,
 3: <type 'long'>,
 4: <type 'float'>,
 5: <type 'float'>,
 7: <function mysql_timestamp_converter at 0x265bb0>,
 8: <type 'long'>,
 9: <type 'int'>,
 10: <function Date_or_None at 0x2654b0>,
 11: <function TimeDelta_or_None at 0x277570>,
 12: <function DateTime_or_None at 0x2773b0>,
 13: <type 'int'>,
 15: [(128, <type 'str'>)],
 246: <class 'decimal.Decimal'>,
 248: <function Str2Set at 0x265b70>,
 252: [(128, <type 'str'>)],
 253: [(128, <type 'str'>)],
 254: [(128, <type 'str'>)]}

Can someone who is having this problem see if they see anything different?

[Evren Esat Özkan]

I have same problem with python-mysqldb (1.2.1-p2-4ubuntu2) and this what I get from db.converter:

{0: <class 'decimal.Decimal'>,
 1: <type 'int'>,
 2: <type 'int'>,
 3: <type 'long'>,
 4: <type 'float'>,
 5: <type 'float'>,
 7: <function mysql_timestamp_converter at 0xb7d594fc>,
 8: <type 'long'>,
 9: <type 'int'>,
 10: <function Date_or_None at 0xb7d59454>,
 11: <function TimeDelta_or_None at 0xb7d593e4>,
 12: <function DateTime_or_None at 0xb7d593ac>,
 13: <type 'int'>,
 246: <class 'decimal.Decimal'>,
 248: <function Str2Set at 0xb7d59684>,
 252: [(128, <function char_array at 0xb7d59844>), (None, None)],
 253: [(2048, <function Str2Set at 0xb7d59684>), (None, None)],
 254: [(2048, <function Str2Set at 0xb7d59684>), (None, None)]}

If I upgrade to MySQL_python-1.2.2-py2.4-linux-i686.egg I get this error?!

File "build/bdist.linux-i686/egg/MySQLdb/connections.py", line 282, in set_character_set  SystemError: NULL object passed to Py_BuildValue 

The odd thing is I don't get this error at home with mysqldb 1.2.2

[Simon G. <dev@…>]

Ok - if I'm understanding MySQLdb properly, we need to get rid of that char_array converter for "252" (i.e. BLOB fields). I'm not sure what the best converter is, but Thing2Str looks like it will work.

So, try this and see if it helps:

>>> from django.conf import settings as s
>>> from django.contrib.sessions.models import Session
>>> import MySQLdb
>>> db = MySQLdb.connect(host=s.DATABASE_HOST, user=s.DATABASE_USER, passwd=s.DATABASE_PASSWORD, db=s.DATABASE_NAME)
>>>
>>> # THIS SHOULD FAIL WITH THE a2b_base64 ERROR
>>> Session.objects.all()[0].session_data

>> # UPDATE THE CONVERTERS AND TRY AGAIN
>>> db.converter.update({ 252: MySQLdb.converters.Thing2Str, })
>>> Session.objects.all()[0].session_data

[Brian Rosner <brosner@…>]

I ran into this problem today. I am running RHEL 4 with MySQL 4.1.20, Python 2.4.4, MySQLdb-1.2.1_p2. I switched my database to utf8_bin and that is when this problem started. I upgraded MySQLdb to 1.2.2 and the problem was fixed. Looks like the issue was with MySQLdb, but still seems like there might be an issue here, but I could be wrong.

[anonymous]

I add the same problem after changing the charset/collection of MySql.

[alexander.pugachev@…]

Fixing asap takes very long time.
This bug occur when you have MySQL database in utf-8, and your collation is utf8_bin.
I get rid from this bug usually with keeping my utf-8 databases in collation utf8_general_ci.

Patch from django@… can not fix this bug, but you can add small check for type at the same place.
Say, you can replace

encoded_data = base64.decodestring(self.session_data)

with

if type(self.session_data) == list:
    encoded_data = base64.decodestring("".join(self.session_data))
else:
    encoded_data = base64.decodestring(self.session_data)

I am not insist this code runs without errors, but idea is clear. This is ugly workaround, but failing admin area is more ugly probably, rspecially for newcomers running tutorial. This kind of check is used now when uploading files, so having such a fragment is not a big deal. If this error is caused by MySQLdb of some versions, some people anyway will have that version, so better have it fixed here than wait when MySQLdb will be fixed and all people will update their MySQLdb.

[Malcolm Tredinnick]

Alexander: this ticket is waiting for somebody who can repeat the problem to give feedback on Simon's suggested fix in comment 14. If that works, it is the correct fix, since it is applied at the right point (initial conversion). So instead of complaining, how about testing the suggestion and seeing if it works?

[Lars Yencken <lars.yencken@…>]

I also encountered this bug when I changed my collation from utf8_general_ci to utf8_bin, which I required for my application. In response to comment 14, I found no failure when I tried his examples, but an identical array was returned both times.

In any case, testing with hasattr seemed to work fine and fixed the problem for me:

if hasattr(self.session_data, 'tostring):
    encoded_data = base64.decodestring(self.session_data.tostring())
else:
    encoded_data = base64.decodestring(self.session_data) 

[anonymous]

same here - update to mysqldb 1.2.2-1000 (fink) fixed it

[Thomas Steinacher <tom@…>]

I can confirm this problem on Debian using python-mysqldb 1.2.1-p2-4. I have a TextField in my model. When I enter a value in the admin (e.g. "mycontent") and save, the contents of the field becomes "array('c', 'mycontent')", which is definitely wrong. This happens with utf8_bin, not with utf8_general_ci.

[Thomas Steinacher <tom@…>]

It looks like this was fixed in mysqldb 1.2.2.
See ​http://sourceforge.net/tracker/index.php?func=detail&aid=1495765&group_id=22307&atid=374932

I cannot confirm that 1.2.2 fixes it, because my Debian Etch doesn't have 1.2.2 yet.

[Lars Yencken <lars.yencken@…>]

Updating to MySQLdb 1.2.2 fixes it for me also.

[mrts]

Can we close this by requiring MySQLdb >= 1.2.2?

[Malcolm Tredinnick]

Based on all the investigations here and the MySQLdb bug that was pointed to, I think we can close this with an addition to the documentation. It only affects TextField fields when using utf8_bin with MySQLdb-1.2.1p2.

I'm about to commit said documentation change and close this off. Would be nice to fix it generally, but the only way to do that requires making a large break of internal abstraction boundaries just for one particular configuration of MySQL, in one version of MySQLdb for one Django field.

[Malcolm Tredinnick]

(In [8568]) Added documentation to explain the gains and losses when using utf8_bin
collation in MySQL. This should help people to make a reasonably informed
decision. Usually, leaving the MySQL collation alone will be the best solution,
but if you must change it, this gives a start to the information you need and
pointers to the appropriate place in the MySQL docs.

There's a small chance I also got all the necessary Sphinx markup correct, too
(it builds without errors, but I may have missed some chances for glory and
linkage).

Fixed #2335, #8506.

[Jacob]

Milestone 1.0 deleted