Opened 7 years ago

Closed 7 years ago

Last modified 7 years ago

#18340 closed Bug (fixed)

formtools.utils.form_hmac is not consistent with unicode input

Reported by: Claude Paroz Owned by: nobody
Component: contrib.formtools Version: master
Severity: Normal Keywords:
Cc: Triage Stage: Accepted
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no


This problem can only be reproduced with cPickle. When feeding the django.contrib.formtools.utils.form_hmac function with Unicode values, stripping the leading/ending spaces produces different pickled results than the unstripped identical values. Here is an output of the two form_hmac passes in the FormHmacTests.test_textfield_hash when you convert form values to Unicode:

(Pdb) p data, pickled
([('name', u'joe'), ('bio', u'Nothing notable.')], '\x80\x02]q\x01(U\x04nameq\x02X\x03\x00\x00\x00joeq\x03\x86q\x04U\x03bioq\x05X\x10\x00\x00\x00Nothing notable.q\x06\x86q\x07e.')

(Pdb) p data, pickled
([('name', u'joe'), ('bio', u'Nothing notable.')], '\x80\x02]q\x01(U\x04nameq\x02X\x03\x00\x00\x00joe\x86q\x03U\x03bioq\x04X\x10\x00\x00\x00Nothing notable.q\x05\x86q\x06e.')

Possible workarounds:

  • use the (slower) pickle module only, as I don't think the performance penalty will be noticeable in this part of the code
  • encode all unicode values to 'utf-8' (smart_str) before feeding them to pickle.dumps
  • feed repr(data) to pickle.dumps

Attachments (1)

18340.diff (1.4 KB) - added by Claude Paroz 7 years ago.
Do not use cPickle

Download all attachments as: .zip

Change History (6)

Changed 7 years ago by Claude Paroz

Attachment: 18340.diff added

Do not use cPickle

comment:1 Changed 7 years ago by Claude Paroz

Has patch: set

comment:2 Changed 7 years ago by Claude Paroz <claude@…>

Resolution: fixed
Status: newclosed

In [1a66f53f9413583d1da266356bb35e5f82868641]:

Fixed #18340 -- Fixed formtools form_hmac with Unicode input

Using cPickle, two apparently identical Unicode strings could
generate different pickled results depending on previous operations
on those strings.

comment:3 Changed 7 years ago by Claude Paroz <claude@…>

In [078ea51b1c7504e3f941bc1d96b0499ace29bdb0]:

Fixed test failures after commit 1a66f53. Refs #18340

comment:4 Changed 7 years ago by anonymous

It's worth noting that while pickle currently seems to be idempotent, it's not guaranteed to anywhere in the spec. Relying on pickling python objects might not be the ideal solution here.

comment:5 Changed 7 years ago by Claude Paroz <claude@…>

In [b109ff8062f4bb225181ec462d69c9dd79339567]:

Replaced pickle by json in form_hmac calculation

Refs #18340

Note: See TracTickets for help on using tickets.
Back to Top