Some contrib.formtools tests fail when Python hash value randomization is enabled

[Matt McDonald]

The following tests in the contrib formtools application can fail when hash value randomization is enabled in the Python interpreter (2.7+). As written, these test cases currently rely on the implicit order of items in a dictionary, which is unsupported.

django.contrib.formtools.tests.WizardTests.test_9473

This test attempts to perform a regex to extract name/value pairs from an HTML string representing a list of <input> fields. The order of the attributes in the HTML is dependent on the sort order of the dictionary that originally represented them. Thus, a regex that expects the 'name' attribute to always directly precede the 'value' attribute will fail.

django.contrib.formtools.tests.wizard.cookiestorage.TestCookieStorage.test_reset_cookie

This test attempts to compare the result of JSON serialized dict to a literal string representation of that expected JSON. Since the order of the elements cannot be predicted, this comparison is prone to failure.

(For more background on hash randomization: ​http://bugs.python.org/issue13703)

Here's an example command line that produces the failures (note -R flag to enable hash randomization):

$ python -R ./runtests.py --settings test_sqlite \
  formtools.TestCookieStorage.test_reset_cookie \
  formtools.WizardTests.test_9473
Creating test database for alias 'default'...
Creating test database for alias 'other'...
FE
======================================================================
ERROR: test_9473 (django.contrib.formtools.tests.WizardTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/django/django/contrib/formtools/tests/__init__.py", line 442, in test_9473
    response = self.check_wizard_step(response, step_no)
  File "/tmp/django/django/contrib/formtools/tests/__init__.py", line 437, in check_wizard_step
    return self.client.post('/wizard2/', data)
  File "/tmp/django/django/test/client.py", line 429, in post
    response = super(Client, self).post(path, data=data, content_type=content_type, **extra)
  File "/tmp/django/django/test/client.py", line 263, in post
    return self.request(**r)
  File "/tmp/django/django/test/client.py", line 390, in request
    six.reraise(*exc_info)
  File "/tmp/django/django/core/handlers/base.py", line 115, in get_response
    response = callback(request, *callback_args, **callback_kwargs)
  File "/tmp/django/django/utils/decorators.py", line 25, in _wrapper
    return bound_func(*args, **kwargs)
  File "/tmp/django/django/utils/decorators.py", line 91, in _wrapped_view
    response = view_func(request, *args, **kwargs)
  File "/tmp/django/django/utils/decorators.py", line 21, in bound_func
    return func(self, *args2, **kwargs2)
  File "/tmp/django/django/contrib/formtools/wizard/legacy.py", line 101, in __call__
    request, f):
  File "/tmp/django/django/contrib/formtools/wizard/legacy.py", line 62, in _check_security_hash
    expected = self.security_hash(request, form)
  File "/tmp/django/django/contrib/formtools/wizard/legacy.py", line 177, in security_hash
    return form_hmac(form)
  File "/tmp/django/django/contrib/formtools/utils.py", line 21, in form_hmac
    value = bf.field.clean(bf.data) or ''
  File "/tmp/django/django/forms/fields.py", line 155, in clean
    self.validate(value)
  File "/tmp/django/django/forms/fields.py", line 127, in validate
    raise ValidationError(self.error_messages['required'])
ValidationError: [u'This field is required.']

======================================================================
FAIL: test_reset_cookie (django.contrib.formtools.tests.wizard.cookiestorage.TestCookieStorage)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/django/django/contrib/formtools/tests/wizard/cookiestorage.py", line 44, in test_reset_cookie
    self.assertEqual(unsigned_cookie_data, '{"step_files":{},"step":null,"extra_data":{},"step_data":{}}')
AssertionError: u'{"step_files":{},"extra_data":{},"step_data":{},"step":null}' != '{"step_files":{},"step":null,"extra_data":{},"step_data":{}}'

----------------------------------------------------------------------
Ran 2 tests in 0.015s

FAILED (failures=1, errors=1)
Destroying test database for alias 'default'...
Destroying test database for alias 'other'...

[Claude Paroz]

This is a part of what has already been reported in #17758. I will combine your patch with some code found on that ticket to fix this one.

[Claude Paroz <claude@…>]

In 1f84b042f1c3fab0f806de814d668917dd86236a:

Fixed #19020 -- Do not depend on dict order in formtools tests

Thanks metzen for the report and initial patch, and Łukasz Rekucki
for an inspirational patch on his randomhash_fixes branch.