Opened 9 years ago

Closed 7 years ago

Last modified 4 years ago

#2335 closed defect (fixed)

In some cases MySQLdb returns array.array which causes an error in a2b_base64()

Reported by: alexander.pugachev@… Owned by: nobody
Component: Database layer (models, ORM) Version: master
Severity: normal Keywords: a2b_base64()
Cc: dev@…, tom@… Triage Stage: Accepted
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: yes
Easy pickings: UI/UX:

Description

settings.py

INSTALLED_APPS = (
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.sites',
    'mysite.polls',
    'django.contrib.admin'
)

urls.py

urlpatterns = patterns('',
    # Example:
    # (r'^mysite/', include('mysite.apps.foo.urls.foo')),

    # Uncomment this for admin:
     (r'^admin/', include('django.contrib.admin.urls')),
)

db synced, server restarted, got admin area login form at /admin/ local url
logged in as superuser, got error:

TypeError at /admin/
a2b_base64() argument 1 must be string or read-only character buffer, not array.array
Request Method: 	GET
Request URL: 	http://127.0.0.1:8000/admin/
Exception Type: 	TypeError
Exception Value: 	a2b_base64() argument 1 must be string or read-only character buffer, not array.array
Exception Location: 	c:\python24\lib\base64.py in decodestring, line 319

Attachments (1)

lancer_beacon.zip (5.4 KB) - added by anonymous 9 years ago.
indeed it was this project

Download all attachments as: .zip

Change History (29)

Changed 9 years ago by anonymous

indeed it was this project

comment:1 Changed 9 years ago by SmileyChris

  • Resolution set to invalid
  • Status changed from new to closed

Thanks for the ticket Alexander, but I'm pretty sure that we'd have more reports of this in 7 months if it was still a problem.
If you can provide some more insight, feel free to reopen.

comment:2 Changed 9 years ago by anonymous

  • Resolution invalid deleted
  • Status changed from closed to reopened

I (different anonymous) also get this problem. I am using mysql5.0.33, apache 2.2.4, python 2.4.4, django 0.95.1 on freebsd 6.2. I am trying to figure it out. I tried with firefox and also with lynx, using the apache server and also the native django server. That exception handler is amazing though! Here's some more output:

/usr/local/lib/python2.4/site-packages/Django-0.95.1-py2.4.egg/django/contrib/sessions/models.py in get_decoded

  1. def get_decoded(self):
  2. encoded_data = base64.decodestring(self.session_data) ...

Local vars

Variable Value
self <Session: Session object>

/usr/local/lib/python2.4/base64.py in decodestring

  1. def decodestring(s):
  2. """Decode a string."""
  3. return binascii.a2b_base64(s)

Local vars

Variable Value
s array('c', 'KGRwMQpTJ3Rlc3Rjb29raWUnCnAyClMnd29ya2VkJwpwMwpzLmU1ZjhhN2QwYzQxYWJlY2RjMWYx\nNDQ3MzUzZjY4YzQw\n')

If anyone wants to inspect my installation, please email me at rbancroft at gmail dot com and I can show you. Also, I did not follow the tutorial 100% as written so there could be something I am missing.

comment:3 Changed 9 years ago by anonymous

so I put everything in a new database letting django create all of the database tables with syncdb and it worked. one difference is perhaps that my own databases uses the utf-bin charset, whereas when django created it, it used latin_swedish_ci?

comment:4 Changed 9 years ago by SmileyChris

  • Resolution set to invalid
  • Status changed from reopened to closed

Thanks for the report, anonymous. If that's the case then it's a very different ticket to this one.

Perhaps someone could investigate and open a new ticket?

comment:5 Changed 9 years ago by anonymous

closed, reopened, invalid, whatever. just an anonymous comment regarding the error:

mysql driver has some strange habit of returning byte arrays as array.array, causing lots of different headaches in various places. patching the mysql driver to return str (as django seems to expext, possibly utf8 encoded) fixes all of that.

comment:6 Changed 8 years ago by ubernostrum

  • Resolution invalid deleted
  • Status changed from closed to reopened

It appears from #3424 that this is still happening; reopening here and marking that one as a dupe since the comment above has a theory on how to fix it.

comment:7 Changed 8 years ago by LightLan@…

I just tried with a newer svn(4750) and the error still happens.
I initialized the database from scratch using ./manage.py syncdb.
It's the same error, as before.

comment:8 Changed 8 years ago by Simon G. <dev@…>

  • Keywords a2b_base64() added
  • Summary changed from Admin area doesn't work at win32 python2.4.3 when followed instructions from Tutorial part 2. code revision 3336 (HEAD for now) to In some cases MySQLdb returns array.array which causes an error in a2b_base64()
  • Triage Stage changed from Unreviewed to Accepted

To recap:

From the anonymous comment above, it has been suggested that the solution is to get the second tuple value from any arrays returned by the MySQL backend.

It would be nice to work out when MySQLdb decides to do this - it looks like this is a charset issue (moving to latin-1 from utf-8 fixed this).

comment:9 Changed 8 years ago by django@…

  • Has patch set
  • Needs tests set
Index: models.py
===================================================================
--- models.py   (revision 4734)
+++ models.py   (working copy)
@@ -58,7 +58,7 @@
         verbose_name_plural = _('sessions')

     def get_decoded(self):
-        encoded_data = base64.decodestring(self.session_data)
+        encoded_data = base64.decodestring(self.session_data.tostring())
         pickled, tamper_check = encoded_data[:-32], encoded_data[-32:]
         if md5.new(pickled + settings.SECRET_KEY).hexdigest() != tamper_check:
             from django.core.exceptions import SuspiciousOperation

Sorry, but I'am not found anything for send path.

comment:10 Changed 8 years ago by mtredinnick

  • Has patch unset
  • Needs tests unset

The patch fragment in the previous comment can't be correct: it assumes that self.session_data is always going to be something that has a tostring() method. However, the reason that this bug is only happening for some people is because this value is _usually_ a string, not an array. So we cannot do any kind of universal conversion here.

comment:11 Changed 8 years ago by Simon G. <dev@…>

This is also affecting the second tutorial (#3783), so we should try to fix this ASAP.

This is a rather ugly bruteforce way of doing it:

if hasattr(self.session_data, 'tostring):
   self.session_data = self.session_data.tostring()
encoded_data = base64.decodestring(self.session_data)

but I'll try to dig through MySQLdb tomorrow and see when it decides to return an array (it appears to be charset contingent)

comment:12 Changed 8 years ago by Simon G. <dev@…>

What I think is happening is that one of the converters is returning an array when we don't want it to. I don't have the bug (I'm using UTF-8 everywhere), and my converters look like this:

>>> from django.conf import settings as s
>>> import MySQLdb
>>> db = MySQLdb.connect(host=s.DATABASE_HOST, user=s.DATABASE_USER, passwd=s.DATABASE_PASSWORD, db=s.DATABASE_NAME)
>>> db.converter

{0: <class 'decimal.Decimal'>,
 1: <type 'int'>,
 2: <type 'int'>,
 3: <type 'long'>,
 4: <type 'float'>,
 5: <type 'float'>,
 7: <function mysql_timestamp_converter at 0x265bb0>,
 8: <type 'long'>,
 9: <type 'int'>,
 10: <function Date_or_None at 0x2654b0>,
 11: <function TimeDelta_or_None at 0x277570>,
 12: <function DateTime_or_None at 0x2773b0>,
 13: <type 'int'>,
 15: [(128, <type 'str'>)],
 246: <class 'decimal.Decimal'>,
 248: <function Str2Set at 0x265b70>,
 252: [(128, <type 'str'>)],
 253: [(128, <type 'str'>)],
 254: [(128, <type 'str'>)]}

Can someone who is having this problem see if they see anything different?

comment:13 Changed 8 years ago by Evren Esat Özkan

I have same problem with python-mysqldb (1.2.1-p2-4ubuntu2) and this what I get from db.converter:

{0: <class 'decimal.Decimal'>,
 1: <type 'int'>,
 2: <type 'int'>,
 3: <type 'long'>,
 4: <type 'float'>,
 5: <type 'float'>,
 7: <function mysql_timestamp_converter at 0xb7d594fc>,
 8: <type 'long'>,
 9: <type 'int'>,
 10: <function Date_or_None at 0xb7d59454>,
 11: <function TimeDelta_or_None at 0xb7d593e4>,
 12: <function DateTime_or_None at 0xb7d593ac>,
 13: <type 'int'>,
 246: <class 'decimal.Decimal'>,
 248: <function Str2Set at 0xb7d59684>,
 252: [(128, <function char_array at 0xb7d59844>), (None, None)],
 253: [(2048, <function Str2Set at 0xb7d59684>), (None, None)],
 254: [(2048, <function Str2Set at 0xb7d59684>), (None, None)]}

If I upgrade to MySQL_python-1.2.2-py2.4-linux-i686.egg I get this error?!

File "build/bdist.linux-i686/egg/MySQLdb/connections.py", line 282, in set_character_set  SystemError: NULL object passed to Py_BuildValue 

The odd thing is I don't get this error at home with mysqldb 1.2.2

comment:14 Changed 8 years ago by Simon G. <dev@…>

  • Cc dev@… added

Ok - if I'm understanding MySQLdb properly, we need to get rid of that char_array converter for "252" (i.e. BLOB fields). I'm not sure what the best converter is, but Thing2Str looks like it will work.

So, try this and see if it helps:

>>> from django.conf import settings as s
>>> from django.contrib.sessions.models import Session
>>> import MySQLdb
>>> db = MySQLdb.connect(host=s.DATABASE_HOST, user=s.DATABASE_USER, passwd=s.DATABASE_PASSWORD, db=s.DATABASE_NAME)
>>>
>>> # THIS SHOULD FAIL WITH THE a2b_base64 ERROR
>>> Session.objects.all()[0].session_data

>> # UPDATE THE CONVERTERS AND TRY AGAIN
>>> db.converter.update({ 252: MySQLdb.converters.Thing2Str, })
>>> Session.objects.all()[0].session_data

comment:15 follow-up: Changed 8 years ago by Brian Rosner <brosner@…>

I ran into this problem today. I am running RHEL 4 with MySQL 4.1.20, Python 2.4.4, MySQLdb-1.2.1_p2. I switched my database to utf8_bin and that is when this problem started. I upgraded MySQLdb to 1.2.2 and the problem was fixed. Looks like the issue was with MySQLdb, but still seems like there might be an issue here, but I could be wrong.

comment:16 in reply to: ↑ 15 Changed 8 years ago by anonymous

I add the same problem after changing the charset/collection of MySql.

comment:17 Changed 8 years ago by alexander.pugachev@…

  • Patch needs improvement set

Fixing asap takes very long time.
This bug occur when you have MySQL database in utf-8, and your collation is utf8_bin.
I get rid from this bug usually with keeping my utf-8 databases in collation utf8_general_ci.

Patch from django@… can not fix this bug, but you can add small check for type at the same place.
Say, you can replace

encoded_data = base64.decodestring(self.session_data)

with

if type(self.session_data) == list:
    encoded_data = base64.decodestring("".join(self.session_data))
else:
    encoded_data = base64.decodestring(self.session_data)

I am not insist this code runs without errors, but idea is clear. This is ugly workaround, but failing admin area is more ugly probably, rspecially for newcomers running tutorial. This kind of check is used now when uploading files, so having such a fragment is not a big deal. If this error is caused by MySQLdb of some versions, some people anyway will have that version, so better have it fixed here than wait when MySQLdb will be fixed and all people will update their MySQLdb.

comment:18 Changed 8 years ago by mtredinnick

Alexander: this ticket is waiting for somebody who can repeat the problem to give feedback on Simon's suggested fix in comment 14. If that works, it is the correct fix, since it is applied at the right point (initial conversion). So instead of complaining, how about testing the suggestion and seeing if it works?

comment:19 Changed 8 years ago by Lars Yencken <lars.yencken@…>

I also encountered this bug when I changed my collation from utf8_general_ci to utf8_bin, which I required for my application. In response to comment 14, I found no failure when I tried his examples, but an identical array was returned both times.

In any case, testing with hasattr seemed to work fine and fixed the problem for me:

if hasattr(self.session_data, 'tostring):
    encoded_data = base64.decodestring(self.session_data.tostring())
else:
    encoded_data = base64.decodestring(self.session_data) 

comment:20 Changed 8 years ago by anonymous

same here - update to mysqldb 1.2.2-1000 (fink) fixed it

comment:21 Changed 8 years ago by Thomas Steinacher <tom@…>

  • Cc tom@… added

I can confirm this problem on Debian using python-mysqldb 1.2.1-p2-4. I have a TextField in my model. When I enter a value in the admin (e.g. "mycontent") and save, the contents of the field becomes "array('c', 'mycontent')", which is definitely wrong. This happens with utf8_bin, not with utf8_general_ci.

comment:22 follow-up: Changed 8 years ago by Thomas Steinacher <tom@…>

It looks like this was fixed in mysqldb 1.2.2.
See http://sourceforge.net/tracker/index.php?func=detail&aid=1495765&group_id=22307&atid=374932

I cannot confirm that 1.2.2 fixes it, because my Debian Etch doesn't have 1.2.2 yet.

comment:23 in reply to: ↑ 22 Changed 8 years ago by Lars Yencken <lars.yencken@…>

Updating to MySQLdb 1.2.2 fixes it for me also.

comment:24 Changed 7 years ago by mrts

  • milestone set to 1.0

Can we close this by requiring MySQLdb >= 1.2.2?

comment:25 Changed 7 years ago by brosner

  • Component changed from Admin interface to Database wrapper

comment:26 Changed 7 years ago by mtredinnick

Based on all the investigations here and the MySQLdb bug that was pointed to, I think we can close this with an addition to the documentation. It only affects TextField fields when using utf8_bin with MySQLdb-1.2.1p2.

I'm about to commit said documentation change and close this off. Would be nice to fix it generally, but the only way to do that requires making a large break of internal abstraction boundaries just for one particular configuration of MySQL, in one version of MySQLdb for one Django field.

comment:27 Changed 7 years ago by mtredinnick

  • Resolution set to fixed
  • Status changed from reopened to closed

(In [8568]) Added documentation to explain the gains and losses when using utf8_bin
collation in MySQL. This should help people to make a reasonably informed
decision. Usually, leaving the MySQL collation alone will be the best solution,
but if you must change it, this gives a start to the information you need and
pointers to the appropriate place in the MySQL docs.

There's a small chance I also got all the necessary Sphinx markup correct, too
(it builds without errors, but I may have missed some chances for glory and
linkage).

Fixed #2335, #8506.

comment:3 Changed 4 years ago by jacob

  • milestone 1.0 deleted

Milestone 1.0 deleted

Note: See TracTickets for help on using tickets.
Back to Top