Opened 13 years ago

Closed 13 years ago

#16393 closed Bug (fixed)

CookieWizard incompatible with simlejson 2.0.9

Reported by: Karen Tracey Owned by: Karen Tracey
Component: contrib.formtools Version: dev
Severity: Normal Keywords:
Cc: Triage Stage: Accepted
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

On an Ubuntu 10.04 machine, the formtools tests produce 5 errors (listed below). I eventually tracked this down to having the Ubuntu-repository version of python-simplejson installed. The version there is 2.0.9. If I instead pip install the current (2.1.16) one the tests pass. (They also pass on a machine with no simplejson installed, so falling back to the 2.0.7 level bundled with Django.) The difference seems to be in how QueryDicts are serialized: with the 2.0.9 level all the key values stored by the wizard are wrapped in lists; this list-wrapping does not seem to happen with the other levels.

It seems like a bug in the 2.0.9 level (at least...I did not try anything besides 2.0.7, 2.0.9, and 2.1.16, so the problem may appear in other in-between levels, I don't know). The reason I'm opening a ticket here is Ubuntu 10.04 is the current Unbuntu long-term service release, and its system-install level of this package is one that doesn't seem to work properly, so I fear a fair number of people may run with it. One possibility to "fix" it would be to document a recommended level for this package if it is installed (we kind of do that for sqlite: ​https://docs.djangoproject.com/en/dev/ref/databases/#sqlite-3-3-6-or-newer-strongly-recommended). We don't currently seem to have any good place to put such info for simplejson though.

Alternatively maybe something could be changed in QueryDict code to get them to correctly serialize even on this 2.0.9 level of simplejson...though I have no idea if that is feasible.

These are the test errors; I'm assuming they reflect problems that would be seen in real code and they are not just test setup artifacts:

--> ./runtests.py --settings=testdb.sqlite -v2 formtools
Importing application formtools
Creating test database for alias 'default' (':memory:')...
Creating tables ...
Creating table django_content_type
Creating table auth_permission
Creating table auth_group_permissions
Creating table auth_group
Creating table auth_user_user_permissions
Creating table auth_user_groups
Creating table auth_user
Creating table django_site
Creating table django_flatpage_sites
Creating table django_flatpage
Creating table django_redirect
Creating table django_session
Creating table django_comments
Creating table django_comment_flags
Creating table django_admin_log
Installing custom SQL ...
Installing indexes ...
No fixtures found.
Creating test database for alias 'other' ('other_db')...
Destroying old test database 'other'...
Creating tables ...
Creating table django_content_type
Creating table auth_permission
Creating table auth_group_permissions
Creating table auth_group
Creating table auth_user_user_permissions
Creating table auth_user_groups
Creating table auth_user
Creating table django_site
Creating table django_flatpage_sites
Creating table django_flatpage
Creating table django_redirect
Creating table django_session
Creating table django_comments
Creating table django_comment_flags
Creating table django_admin_log
Installing custom SQL ...
Installing indexes ...
No fixtures found.
test_init (django.contrib.formtools.wizard.tests.formtests.CookieFormTests) ... ok
test_cleaned_data (django.contrib.formtools.wizard.tests.wizardtests.tests.CookieWizardTests) ... ERROR
test_form_finish (django.contrib.formtools.wizard.tests.wizardtests.tests.CookieWizardTests) ... ERROR
test_form_post_error (django.contrib.formtools.wizard.tests.wizardtests.tests.CookieWizardTests) ... ok
test_form_post_success (django.contrib.formtools.wizard.tests.wizardtests.tests.CookieWizardTests) ... ok
test_form_refresh (django.contrib.formtools.wizard.tests.wizardtests.tests.CookieWizardTests) ... ERROR
test_form_stepback (django.contrib.formtools.wizard.tests.wizardtests.tests.CookieWizardTests) ... ok
test_initial_call (django.contrib.formtools.wizard.tests.wizardtests.tests.CookieWizardTests) ... ok
test_manipulated_data (django.contrib.formtools.wizard.tests.wizardtests.tests.CookieWizardTests) ... ok
test_template_context (django.contrib.formtools.wizard.tests.wizardtests.tests.CookieWizardTests) ... ok
test_done (django.contrib.formtools.wizard.tests.formtests.FormTests) ... ok
test_first_step (django.contrib.formtools.wizard.tests.formtests.FormTests) ... ok
test_form_condition (django.contrib.formtools.wizard.tests.formtests.FormTests) ... ok
test_form_init (django.contrib.formtools.wizard.tests.formtests.FormTests) ... ok
test_form_initial (django.contrib.formtools.wizard.tests.formtests.FormTests) ... ok
test_form_instance (django.contrib.formtools.wizard.tests.formtests.FormTests) ... ok
test_form_kwargs (django.contrib.formtools.wizard.tests.formtests.FormTests) ... ok
test_form_prefix (django.contrib.formtools.wizard.tests.formtests.FormTests) ... ok
test_formset_instance (django.contrib.formtools.wizard.tests.formtests.FormTests) ... ok
test_persistence (django.contrib.formtools.wizard.tests.formtests.FormTests) ... ok
test_revalidation (django.contrib.formtools.wizard.tests.formtests.FormTests) ... ok
test_revalidation (django.contrib.formtools.wizard.tests.namedwizardtests.tests.NamedCookieFormTests) ... ok
test_cleaned_data (django.contrib.formtools.wizard.tests.namedwizardtests.tests.NamedCookieWizardTests) ... ERROR
test_form_finish (django.contrib.formtools.wizard.tests.namedwizardtests.tests.NamedCookieWizardTests) ... ERROR
test_form_jump (django.contrib.formtools.wizard.tests.namedwizardtests.tests.NamedCookieWizardTests) ... ok
test_form_post_error (django.contrib.formtools.wizard.tests.namedwizardtests.tests.NamedCookieWizardTests) ... ok
test_form_post_success (django.contrib.formtools.wizard.tests.namedwizardtests.tests.NamedCookieWizardTests) ... ok
test_form_reset (django.contrib.formtools.wizard.tests.namedwizardtests.tests.NamedCookieWizardTests) ... ok
test_form_stepback (django.contrib.formtools.wizard.tests.namedwizardtests.tests.NamedCookieWizardTests) ... ok
test_initial_call (django.contrib.formtools.wizard.tests.namedwizardtests.tests.NamedCookieWizardTests) ... ok
test_initial_call_with_params (django.contrib.formtools.wizard.tests.namedwizardtests.tests.NamedCookieWizardTests) ... ok
test_manipulated_data (django.contrib.formtools.wizard.tests.namedwizardtests.tests.NamedCookieWizardTests) ... ok
test_revalidation (django.contrib.formtools.wizard.tests.namedwizardtests.tests.NamedSessionFormTests) ... ok
test_cleaned_data (django.contrib.formtools.wizard.tests.namedwizardtests.tests.NamedSessionWizardTests) ... ok
test_form_finish (django.contrib.formtools.wizard.tests.namedwizardtests.tests.NamedSessionWizardTests) ... ok
test_form_jump (django.contrib.formtools.wizard.tests.namedwizardtests.tests.NamedSessionWizardTests) ... ok
test_form_post_error (django.contrib.formtools.wizard.tests.namedwizardtests.tests.NamedSessionWizardTests) ... ok
test_form_post_success (django.contrib.formtools.wizard.tests.namedwizardtests.tests.NamedSessionWizardTests) ... ok
test_form_reset (django.contrib.formtools.wizard.tests.namedwizardtests.tests.NamedSessionWizardTests) ... ok
test_form_stepback (django.contrib.formtools.wizard.tests.namedwizardtests.tests.NamedSessionWizardTests) ... ok
test_initial_call (django.contrib.formtools.wizard.tests.namedwizardtests.tests.NamedSessionWizardTests) ... ok
test_initial_call_with_params (django.contrib.formtools.wizard.tests.namedwizardtests.tests.NamedSessionWizardTests) ... ok
test_manipulated_data (django.contrib.formtools.wizard.tests.namedwizardtests.tests.NamedSessionWizardTests) ... ok
test_bool_submit (django.contrib.formtools.tests.PreviewTests) ... ok
test_form_get (django.contrib.formtools.tests.PreviewTests) ... ok
test_form_preview (django.contrib.formtools.tests.PreviewTests) ... ok
test_form_submit (django.contrib.formtools.tests.PreviewTests) ... ok
test_form_submit_bad_hash (django.contrib.formtools.tests.PreviewTests) ... ok
test_form_submit_good_hash (django.contrib.formtools.tests.PreviewTests) ... ok
test_unused_name (django.contrib.formtools.tests.PreviewTests) ... ok
test_init (django.contrib.formtools.wizard.tests.formtests.SessionFormTests) ... ok
test_cleaned_data (django.contrib.formtools.wizard.tests.wizardtests.tests.SessionWizardTests) ... ok
test_form_finish (django.contrib.formtools.wizard.tests.wizardtests.tests.SessionWizardTests) ... ok
test_form_post_error (django.contrib.formtools.wizard.tests.wizardtests.tests.SessionWizardTests) ... ok
test_form_post_success (django.contrib.formtools.wizard.tests.wizardtests.tests.SessionWizardTests) ... ok
test_form_refresh (django.contrib.formtools.wizard.tests.wizardtests.tests.SessionWizardTests) ... ok
test_form_stepback (django.contrib.formtools.wizard.tests.wizardtests.tests.SessionWizardTests) ... ok
test_initial_call (django.contrib.formtools.wizard.tests.wizardtests.tests.SessionWizardTests) ... ok
test_manipulated_data (django.contrib.formtools.wizard.tests.wizardtests.tests.SessionWizardTests) ... ok
test_template_context (django.contrib.formtools.wizard.tests.wizardtests.tests.SessionWizardTests) ... ok
test_current_step (django.contrib.formtools.wizard.tests.cookiestoragetests.TestCookieStorage) ... ok
test_extra_context (django.contrib.formtools.wizard.tests.cookiestoragetests.TestCookieStorage) ... ok
test_manipulated_cookie (django.contrib.formtools.wizard.tests.cookiestoragetests.TestCookieStorage) ... ok
test_reset_cookie (django.contrib.formtools.wizard.tests.cookiestoragetests.TestCookieStorage) ... ok
test_step_data (django.contrib.formtools.wizard.tests.cookiestoragetests.TestCookieStorage) ... ok
test_load_storage (django.contrib.formtools.wizard.tests.loadstoragetests.TestLoadStorage) ... ok
test_missing_class (django.contrib.formtools.wizard.tests.loadstoragetests.TestLoadStorage) ... ok
test_missing_module (django.contrib.formtools.wizard.tests.loadstoragetests.TestLoadStorage) ... ok
test_current_step (django.contrib.formtools.wizard.tests.sessionstoragetests.TestSessionStorage) ... ok
test_extra_context (django.contrib.formtools.wizard.tests.sessionstoragetests.TestSessionStorage) ... ok
test_step_data (django.contrib.formtools.wizard.tests.sessionstoragetests.TestSessionStorage) ... ok
test_template (django.contrib.formtools.wizard.tests.wizardtests.tests.WizardTestKwargs) ... ok
test_11726 (django.contrib.formtools.tests.WizardTests) ... ok
test_14498 (django.contrib.formtools.tests.WizardTests) ... ok
test_14576 (django.contrib.formtools.tests.WizardTests) ... ok
test_15075 (django.contrib.formtools.tests.WizardTests) ... ok
test_9473 (django.contrib.formtools.tests.WizardTests) ... ok
test_bad_hash (django.contrib.formtools.tests.WizardTests) ... ok
test_good_hash (django.contrib.formtools.tests.WizardTests) ... ok
test_step_increments (django.contrib.formtools.tests.WizardTests) ... ok
test_step_starts_at_zero (django.contrib.formtools.tests.WizardTests) ... ok
test_empty_permitted (django.contrib.formtools.tests.FormHmacTests) ... ok
test_textfield_hash (django.contrib.formtools.tests.FormHmacTests) ... ok
test_empty_permitted (django.contrib.formtools.tests.SecurityHashTests) ... ok
test_textfield_hash (django.contrib.formtools.tests.SecurityHashTests) ... ok

======================================================================
ERROR: test_cleaned_data (django.contrib.formtools.wizard.tests.wizardtests.tests.CookieWizardTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/kmtracey/django/trunk/django/contrib/formtools/wizard/tests/wizardtests/tests.py", line 118, in test_cleaned_data
    response = self.client.post(self.wizard_url, self.wizard_step_data[3])
  File "/home/kmtracey/django/trunk/django/test/client.py", line 455, in post
    response = super(Client, self).post(path, data=data, content_type=content_type, **extra)
  File "/home/kmtracey/django/trunk/django/test/client.py", line 256, in post
    return self.request(**r)
  File "/home/kmtracey/django/trunk/django/core/handlers/base.py", line 111, in get_response
    response = callback(request, *callback_args, **callback_kwargs)
  File "/home/kmtracey/django/trunk/django/views/generic/base.py", line 47, in view
    return self.dispatch(request, *args, **kwargs)
  File "/home/kmtracey/django/trunk/django/contrib/formtools/wizard/views.py", line 219, in dispatch
    response = super(WizardView, self).dispatch(request, *args, **kwargs)
  File "/home/kmtracey/django/trunk/django/views/generic/base.py", line 68, in dispatch
    return handler(request, *args, **kwargs)
  File "/home/kmtracey/django/trunk/django/contrib/formtools/wizard/views.py", line 282, in post
    return self.render_done(form, **kwargs)
  File "/home/kmtracey/django/trunk/django/contrib/formtools/wizard/views.py", line 317, in render_done
    if not form_obj.is_valid():
  File "/home/kmtracey/django/trunk/django/forms/forms.py", line 121, in is_valid
    return self.is_bound and not bool(self.errors)
  File "/home/kmtracey/django/trunk/django/forms/forms.py", line 112, in _get_errors
    self.full_clean()
  File "/home/kmtracey/django/trunk/django/forms/forms.py", line 267, in full_clean
    self._clean_fields()
  File "/home/kmtracey/django/trunk/django/forms/forms.py", line 284, in _clean_fields
    value = field.clean(value)
  File "/home/kmtracey/django/trunk/django/forms/fields.py", line 151, in clean
    value = self.to_python(value)
  File "/home/kmtracey/django/trunk/django/forms/models.py", line 976, in to_python
    value = self.queryset.get(**{key: value})
  File "/home/kmtracey/django/trunk/django/db/models/query.py", line 341, in get
    clone = self.filter(*args, **kwargs)
  File "/home/kmtracey/django/trunk/django/db/models/query.py", line 550, in filter
    return self._filter_or_exclude(False, *args, **kwargs)
  File "/home/kmtracey/django/trunk/django/db/models/query.py", line 568, in _filter_or_exclude
    clone.query.add_q(Q(*args, **kwargs))
  File "/home/kmtracey/django/trunk/django/db/models/sql/query.py", line 1185, in add_q
    can_reuse=used_aliases, force_having=force_having)
  File "/home/kmtracey/django/trunk/django/db/models/sql/query.py", line 1120, in add_filter
    connector)
  File "/home/kmtracey/django/trunk/django/db/models/sql/where.py", line 67, in add
    value = obj.prepare(lookup_type, value)
  File "/home/kmtracey/django/trunk/django/db/models/sql/where.py", line 316, in prepare
    return self.field.get_prep_lookup(lookup_type, value)
  File "/home/kmtracey/django/trunk/django/db/models/fields/__init__.py", line 291, in get_prep_lookup
    return self.get_prep_value(value)
  File "/home/kmtracey/django/trunk/django/db/models/fields/__init__.py", line 488, in get_prep_value
    return int(value)
TypeError: int() argument must be a string or a number, not 'list'

======================================================================
ERROR: test_form_finish (django.contrib.formtools.wizard.tests.wizardtests.tests.CookieWizardTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/kmtracey/django/trunk/django/contrib/formtools/wizard/tests/wizardtests/tests.py", line 90, in test_form_finish
    response = self.client.post(self.wizard_url, self.wizard_step_data[3])
  File "/home/kmtracey/django/trunk/django/test/client.py", line 455, in post
    response = super(Client, self).post(path, data=data, content_type=content_type, **extra)
  File "/home/kmtracey/django/trunk/django/test/client.py", line 256, in post
    return self.request(**r)
  File "/home/kmtracey/django/trunk/django/core/handlers/base.py", line 111, in get_response
    response = callback(request, *callback_args, **callback_kwargs)
  File "/home/kmtracey/django/trunk/django/views/generic/base.py", line 47, in view
    return self.dispatch(request, *args, **kwargs)
  File "/home/kmtracey/django/trunk/django/contrib/formtools/wizard/views.py", line 219, in dispatch
    response = super(WizardView, self).dispatch(request, *args, **kwargs)
  File "/home/kmtracey/django/trunk/django/views/generic/base.py", line 68, in dispatch
    return handler(request, *args, **kwargs)
  File "/home/kmtracey/django/trunk/django/contrib/formtools/wizard/views.py", line 282, in post
    return self.render_done(form, **kwargs)
  File "/home/kmtracey/django/trunk/django/contrib/formtools/wizard/views.py", line 317, in render_done
    if not form_obj.is_valid():
  File "/home/kmtracey/django/trunk/django/forms/forms.py", line 121, in is_valid
    return self.is_bound and not bool(self.errors)
  File "/home/kmtracey/django/trunk/django/forms/forms.py", line 112, in _get_errors
    self.full_clean()
  File "/home/kmtracey/django/trunk/django/forms/forms.py", line 267, in full_clean
    self._clean_fields()
  File "/home/kmtracey/django/trunk/django/forms/forms.py", line 284, in _clean_fields
    value = field.clean(value)
  File "/home/kmtracey/django/trunk/django/forms/fields.py", line 151, in clean
    value = self.to_python(value)
  File "/home/kmtracey/django/trunk/django/forms/models.py", line 976, in to_python
    value = self.queryset.get(**{key: value})
  File "/home/kmtracey/django/trunk/django/db/models/query.py", line 341, in get
    clone = self.filter(*args, **kwargs)
  File "/home/kmtracey/django/trunk/django/db/models/query.py", line 550, in filter
    return self._filter_or_exclude(False, *args, **kwargs)
  File "/home/kmtracey/django/trunk/django/db/models/query.py", line 568, in _filter_or_exclude
    clone.query.add_q(Q(*args, **kwargs))
  File "/home/kmtracey/django/trunk/django/db/models/sql/query.py", line 1185, in add_q
    can_reuse=used_aliases, force_having=force_having)
  File "/home/kmtracey/django/trunk/django/db/models/sql/query.py", line 1120, in add_filter
    connector)
  File "/home/kmtracey/django/trunk/django/db/models/sql/where.py", line 67, in add
    value = obj.prepare(lookup_type, value)
  File "/home/kmtracey/django/trunk/django/db/models/sql/where.py", line 316, in prepare
    return self.field.get_prep_lookup(lookup_type, value)
  File "/home/kmtracey/django/trunk/django/db/models/fields/__init__.py", line 291, in get_prep_lookup
    return self.get_prep_value(value)
  File "/home/kmtracey/django/trunk/django/db/models/fields/__init__.py", line 488, in get_prep_value
    return int(value)
TypeError: int() argument must be a string or a number, not 'list'

======================================================================
ERROR: test_form_refresh (django.contrib.formtools.wizard.tests.wizardtests.tests.CookieWizardTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/kmtracey/django/trunk/django/contrib/formtools/wizard/tests/wizardtests/tests.py", line 179, in test_form_refresh
    response = self.client.post(self.wizard_url, self.wizard_step_data[3])
  File "/home/kmtracey/django/trunk/django/test/client.py", line 455, in post
    response = super(Client, self).post(path, data=data, content_type=content_type, **extra)
  File "/home/kmtracey/django/trunk/django/test/client.py", line 256, in post
    return self.request(**r)
  File "/home/kmtracey/django/trunk/django/core/handlers/base.py", line 111, in get_response
    response = callback(request, *callback_args, **callback_kwargs)
  File "/home/kmtracey/django/trunk/django/views/generic/base.py", line 47, in view
    return self.dispatch(request, *args, **kwargs)
  File "/home/kmtracey/django/trunk/django/contrib/formtools/wizard/views.py", line 219, in dispatch
    response = super(WizardView, self).dispatch(request, *args, **kwargs)
  File "/home/kmtracey/django/trunk/django/views/generic/base.py", line 68, in dispatch
    return handler(request, *args, **kwargs)
  File "/home/kmtracey/django/trunk/django/contrib/formtools/wizard/views.py", line 282, in post
    return self.render_done(form, **kwargs)
  File "/home/kmtracey/django/trunk/django/contrib/formtools/wizard/views.py", line 317, in render_done
    if not form_obj.is_valid():
  File "/home/kmtracey/django/trunk/django/forms/forms.py", line 121, in is_valid
    return self.is_bound and not bool(self.errors)
  File "/home/kmtracey/django/trunk/django/forms/forms.py", line 112, in _get_errors
    self.full_clean()
  File "/home/kmtracey/django/trunk/django/forms/forms.py", line 267, in full_clean
    self._clean_fields()
  File "/home/kmtracey/django/trunk/django/forms/forms.py", line 284, in _clean_fields
    value = field.clean(value)
  File "/home/kmtracey/django/trunk/django/forms/fields.py", line 151, in clean
    value = self.to_python(value)
  File "/home/kmtracey/django/trunk/django/forms/models.py", line 976, in to_python
    value = self.queryset.get(**{key: value})
  File "/home/kmtracey/django/trunk/django/db/models/query.py", line 341, in get
    clone = self.filter(*args, **kwargs)
  File "/home/kmtracey/django/trunk/django/db/models/query.py", line 550, in filter
    return self._filter_or_exclude(False, *args, **kwargs)
  File "/home/kmtracey/django/trunk/django/db/models/query.py", line 568, in _filter_or_exclude
    clone.query.add_q(Q(*args, **kwargs))
  File "/home/kmtracey/django/trunk/django/db/models/sql/query.py", line 1185, in add_q
    can_reuse=used_aliases, force_having=force_having)
  File "/home/kmtracey/django/trunk/django/db/models/sql/query.py", line 1120, in add_filter
    connector)
  File "/home/kmtracey/django/trunk/django/db/models/sql/where.py", line 67, in add
    value = obj.prepare(lookup_type, value)
  File "/home/kmtracey/django/trunk/django/db/models/sql/where.py", line 316, in prepare
    return self.field.get_prep_lookup(lookup_type, value)
  File "/home/kmtracey/django/trunk/django/db/models/fields/__init__.py", line 291, in get_prep_lookup
    return self.get_prep_value(value)
  File "/home/kmtracey/django/trunk/django/db/models/fields/__init__.py", line 488, in get_prep_value
    return int(value)
TypeError: int() argument must be a string or a number, not 'list'

======================================================================
ERROR: test_cleaned_data (django.contrib.formtools.wizard.tests.namedwizardtests.tests.NamedCookieWizardTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/kmtracey/django/trunk/django/contrib/formtools/wizard/tests/namedwizardtests/tests.py", line 188, in test_cleaned_data
    response = self.client.get(response['Location'])
  File "/home/kmtracey/django/trunk/django/test/client.py", line 445, in get
    response = super(Client, self).get(path, data=data, **extra)
  File "/home/kmtracey/django/trunk/django/test/client.py", line 229, in get
    return self.request(**r)
  File "/home/kmtracey/django/trunk/django/core/handlers/base.py", line 111, in get_response
    response = callback(request, *callback_args, **callback_kwargs)
  File "/home/kmtracey/django/trunk/django/views/generic/base.py", line 47, in view
    return self.dispatch(request, *args, **kwargs)
  File "/home/kmtracey/django/trunk/django/contrib/formtools/wizard/views.py", line 219, in dispatch
    response = super(WizardView, self).dispatch(request, *args, **kwargs)
  File "/home/kmtracey/django/trunk/django/views/generic/base.py", line 68, in dispatch
    return handler(request, *args, **kwargs)
  File "/home/kmtracey/django/trunk/django/contrib/formtools/wizard/views.py", line 613, in get
    files=self.storage.get_step_files(last_step)
  File "/home/kmtracey/django/trunk/django/contrib/formtools/wizard/views.py", line 389, in get_form
    return self.form_list[step](**kwargs)
  File "/home/kmtracey/django/trunk/django/forms/formsets.py", line 47, in __init__
    self._construct_forms()
  File "/home/kmtracey/django/trunk/django/forms/formsets.py", line 107, in _construct_forms
    for i in xrange(self.total_form_count()):
  File "/home/kmtracey/django/trunk/django/forms/formsets.py", line 81, in total_form_count
    return self.management_form.cleaned_data[TOTAL_FORM_COUNT]
  File "/home/kmtracey/django/trunk/django/forms/formsets.py", line 68, in _management_form
    raise ValidationError('ManagementForm data is missing or has been tampered with')
ValidationError: [u'ManagementForm data is missing or has been tampered with']

======================================================================
ERROR: test_form_finish (django.contrib.formtools.wizard.tests.namedwizardtests.tests.NamedCookieWizardTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/kmtracey/django/trunk/django/contrib/formtools/wizard/tests/namedwizardtests/tests.py", line 149, in test_form_finish
    response = self.client.get(response['Location'])
  File "/home/kmtracey/django/trunk/django/test/client.py", line 445, in get
    response = super(Client, self).get(path, data=data, **extra)
  File "/home/kmtracey/django/trunk/django/test/client.py", line 229, in get
    return self.request(**r)
  File "/home/kmtracey/django/trunk/django/core/handlers/base.py", line 111, in get_response
    response = callback(request, *callback_args, **callback_kwargs)
  File "/home/kmtracey/django/trunk/django/views/generic/base.py", line 47, in view
    return self.dispatch(request, *args, **kwargs)
  File "/home/kmtracey/django/trunk/django/contrib/formtools/wizard/views.py", line 219, in dispatch
    response = super(WizardView, self).dispatch(request, *args, **kwargs)
  File "/home/kmtracey/django/trunk/django/views/generic/base.py", line 68, in dispatch
    return handler(request, *args, **kwargs)
  File "/home/kmtracey/django/trunk/django/contrib/formtools/wizard/views.py", line 613, in get
    files=self.storage.get_step_files(last_step)
  File "/home/kmtracey/django/trunk/django/contrib/formtools/wizard/views.py", line 389, in get_form
    return self.form_list[step](**kwargs)
  File "/home/kmtracey/django/trunk/django/forms/formsets.py", line 47, in __init__
    self._construct_forms()
  File "/home/kmtracey/django/trunk/django/forms/formsets.py", line 107, in _construct_forms
    for i in xrange(self.total_form_count()):
  File "/home/kmtracey/django/trunk/django/forms/formsets.py", line 81, in total_form_count
    return self.management_form.cleaned_data[TOTAL_FORM_COUNT]
  File "/home/kmtracey/django/trunk/django/forms/formsets.py", line 68, in _management_form
    raise ValidationError('ManagementForm data is missing or has been tampered with')
ValidationError: [u'ManagementForm data is missing or has been tampered with']

----------------------------------------------------------------------
Ran 85 tests in 1.250s

FAILED (errors=5)
Destroying test database for alias 'default' (':memory:')...
Destroying test database for alias 'other' ('other_db')...

Attachments (1)

trac-16393.diff​ (1.3 KB ) - added by steph 13 years ago.
Fix without extra tests.

Download all attachments as: .zip

Change History (9)

comment:1 by Aymeric Augustin, 13 years ago

Component: Uncategorized → Forms
Triage Stage: Unreviewed → Accepted

django.http.QueryDict inherits django.utils.datastructures.MultiValueDict where a custom iteritems is defined.

The changelog of simplejson 2.1.0 (which is the version that comes just after 2.0.9) contains this line:

Encoder changed to use iteritems rather than PyDict_Next in order to support dict subclasses that have a well defined ordering ​http://bugs.python.org/issue6105

This is a plausible explanation for the change in the serialization output.

Also, the implementation with PyDict_Next in <= 2.0.9 can be used only when the C module (speedups) is compiled, and it isn't in the version bundled with Django — it's the pure Python implementation. This would explain why you don't see the bug when simplejson isn't installed.

comment:2 by Jannis Leidel, 13 years ago

I was able to reproduce the error messages and am rather baffled by them, to be honest. Can you elaborate why QueryDict is supposed to be the problem?

comment:3 by Karen Tracey, 13 years ago

This is pdb info from a session where simplejson 2.0.9 was installed. I added breakpoints to the django/contrib/formtools/wizard/storage/cookie.py file, methods load_data and update_response to see what was getting stored in the cookie and what was subsequently loaded:

--> ./runtests.py --settings=testdb.sqlite -v0 formtools.CookieWizardTests.test_cleaned_data
> /home/kmtracey/django/hg-django/django/contrib/formtools/wizard/storage/cookie.py(32)update_response()
-> response.set_signed_cookie(self.prefix, self.encoder.encode(self.data))
(Pdb) self.data
{'step_files': {}, 'step': u'form1', 'extra_data': {}, 'step_data': {}}
(Pdb) c
> /home/kmtracey/django/hg-django/django/contrib/formtools/wizard/storage/cookie.py(27)load_data()
-> return json.loads(data, cls=json.JSONDecoder)
(Pdb) n
--Return--
> /home/kmtracey/django/hg-django/django/contrib/formtools/wizard/storage/cookie.py(27)load_data()->{u'extra_data': {}, u'step': u'form1', u'step_data': {}, u'step_files': {}}
-> return json.loads(data, cls=json.JSONDecoder)

# No problem above, first calls to store/load the cookie data have no data that causes a problem

# Next iteration is where there's a problem. The step_data for form1 is a QueryDict

(Pdb) c
> /home/kmtracey/django/hg-django/django/contrib/formtools/wizard/storage/cookie.py(32)update_response()
-> response.set_signed_cookie(self.prefix, self.encoder.encode(self.data))
(Pdb) self.data
{u'step_files': {u'form1': {}}, u'step': u'form2', u'extra_data': {}, u'step_data': {u'form1': <QueryDict: {u'form1-name': [u'Pony'], u'cookie_contact_wizard-current_step': [u'form1'], u'form1-user': [u'1'], u'form1-thirsty': [u'2']}>}}
(Pdb) self.data['step_data']['form1']
<QueryDict: {u'form1-name': [u'Pony'], u'cookie_contact_wizard-current_step': [u'form1'], u'form1-user': [u'1'], u'form1-thirsty': [u'2']}>
(Pdb) c
> /home/kmtracey/django/hg-django/django/contrib/formtools/wizard/storage/cookie.py(27)load_data()
-> return json.loads(data, cls=json.JSONDecoder)
(Pdb) n
--Return--
> /home/kmtracey/django/hg-django/django/contrib/formtools/wizard/storage/cookie.py(27)load_data()->{u'extra_data': {}, u'step': u'form2', u'step_data': {u'form1': {u'cookie...ent_step': [u'form1'], u'form1-name': [u'Pony'], u'form1-thirsty': [u'2'], u'form1-user': [u'1']}}, u'step_files': {u'form1': {}}}
-> return json.loads(data, cls=json.JSONDecoder)
(Pdb) n

# When step_data for form1 is re-constituted, the values for all the keys are wrapped in lists
# This ultimately triggers errors like: TypeError: int() argument must be a string or a number, not 'list'
# when the wizard code uses the re-constituted step data as data for the form.

> /home/kmtracey/django/hg-django/django/contrib/formtools/wizard/storage/cookie.py(14)__init__()
-> if self.data is None:
(Pdb) self.data['step_data']['form1']
{u'form1-user': [u'1'], u'cookie_contact_wizard-current_step': [u'form1'], u'form1-name': [u'Pony'], u'form1-thirsty': [u'2']}
(Pdb)

This is with simplejson 2.1.6 installed:

--> ./runtests.py --settings=testdb.sqlite -v0 formtools.CookieWizardTests.test_cleaned_data
> /home/kmtracey/django/hg-django/django/contrib/formtools/wizard/storage/cookie.py(32)update_response()
-> response.set_signed_cookie(self.prefix, self.encoder.encode(self.data))
(Pdb) c
> /home/kmtracey/django/hg-django/django/contrib/formtools/wizard/storage/cookie.py(27)load_data()
-> return json.loads(data, cls=json.JSONDecoder)
(Pdb) c
> /home/kmtracey/django/hg-django/django/contrib/formtools/wizard/storage/cookie.py(32)update_response()
-> response.set_signed_cookie(self.prefix, self.encoder.encode(self.data))
(Pdb) self.data
{u'step_files': {u'form1': {}}, u'step': u'form2', u'extra_data': {}, u'step_data': {u'form1': <QueryDict: {u'form1-name': [u'Pony'], u'cookie_contact_wizard-current_step': [u'form1'], u'form1-user': [u'1'], u'form1-thirsty': [u'2']}>}}
(Pdb) self.data['step_data']['form1']
<QueryDict: {u'form1-name': [u'Pony'], u'cookie_contact_wizard-current_step': [u'form1'], u'form1-user': [u'1'], u'form1-thirsty': [u'2']}>
(Pdb) c
> /home/kmtracey/django/hg-django/django/contrib/formtools/wizard/storage/cookie.py(27)load_data()
-> return json.loads(data, cls=json.JSONDecoder)
(Pdb) n
--Return--
> /home/kmtracey/django/hg-django/django/contrib/formtools/wizard/storage/cookie.py(27)load_data()->{u'extra_data': {}, u'step': u'form2', u'step_data': {u'form1': {u'cookie...ent_step': u'form1', u'form1-name': u'Pony', u'form1-thirsty': u'2', u'form1-user': u'1'}}, u'step_files': {u'form1': {}}}
-> return json.loads(data, cls=json.JSONDecoder)
(Pdb) n


# With this level of simplejson, the step_data for form1 does not have its values wrapped in lists

> /home/kmtracey/django/hg-django/django/contrib/formtools/wizard/storage/cookie.py(14)__init__()
-> if self.data is None:
(Pdb) self.data['step_data']['form1']
{u'form1-user': u'1', u'cookie_contact_wizard-current_step': u'form1', u'form1-name': u'Pony', u'form1-thirsty': u'2'}
(Pdb)

Thinking about it a bit more, I do kind of wonder whether the formwizard will correctly handle forms where multi-value keys are expected, since round-tripping through serialization you don't get the QueryDict with its special characteristics back, and the formwizard code seems to be using the de-serialized version for cleaning on the last step...

in reply to:  3 comment:4 by Jannis Leidel, 13 years ago

Replying to kmtracey:

This is pdb info from a session where simplejson 2.0.9 was installed. I added breakpoints to the django/contrib/formtools/wizard/storage/cookie.py file, methods load_data and update_response to see what was getting stored in the cookie and what was subsequently loaded:

--> ./runtests.py --settings=testdb.sqlite -v0 formtools.CookieWizardTests.test_cleaned_data
> /home/kmtracey/django/hg-django/django/contrib/formtools/wizard/storage/cookie.py(32)update_response()
-> response.set_signed_cookie(self.prefix, self.encoder.encode(self.data))
(Pdb) self.data
{'step_files': {}, 'step': u'form1', 'extra_data': {}, 'step_data': {}}
(Pdb) c
> /home/kmtracey/django/hg-django/django/contrib/formtools/wizard/storage/cookie.py(27)load_data()
-> return json.loads(data, cls=json.JSONDecoder)
(Pdb) n
--Return--
> /home/kmtracey/django/hg-django/django/contrib/formtools/wizard/storage/cookie.py(27)load_data()->{u'extra_data': {}, u'step': u'form1', u'step_data': {}, u'step_files': {}}
-> return json.loads(data, cls=json.JSONDecoder)

# No problem above, first calls to store/load the cookie data have no data that causes a problem

# Next iteration is where there's a problem. The step_data for form1 is a QueryDict

(Pdb) c
> /home/kmtracey/django/hg-django/django/contrib/formtools/wizard/storage/cookie.py(32)update_response()
-> response.set_signed_cookie(self.prefix, self.encoder.encode(self.data))
(Pdb) self.data
{u'step_files': {u'form1': {}}, u'step': u'form2', u'extra_data': {}, u'step_data': {u'form1': <QueryDict: {u'form1-name': [u'Pony'], u'cookie_contact_wizard-current_step': [u'form1'], u'form1-user': [u'1'], u'form1-thirsty': [u'2']}>}}
(Pdb) self.data['step_data']['form1']
<QueryDict: {u'form1-name': [u'Pony'], u'cookie_contact_wizard-current_step': [u'form1'], u'form1-user': [u'1'], u'form1-thirsty': [u'2']}>
(Pdb) c
> /home/kmtracey/django/hg-django/django/contrib/formtools/wizard/storage/cookie.py(27)load_data()
-> return json.loads(data, cls=json.JSONDecoder)
(Pdb) n
--Return--
> /home/kmtracey/django/hg-django/django/contrib/formtools/wizard/storage/cookie.py(27)load_data()->{u'extra_data': {}, u'step': u'form2', u'step_data': {u'form1': {u'cookie...ent_step': [u'form1'], u'form1-name': [u'Pony'], u'form1-thirsty': [u'2'], u'form1-user': [u'1']}}, u'step_files': {u'form1': {}}}
-> return json.loads(data, cls=json.JSONDecoder)
(Pdb) n

# When step_data for form1 is re-constituted, the values for all the keys are wrapped in lists
# This ultimately triggers errors like: TypeError: int() argument must be a string or a number, not 'list'
# when the wizard code uses the re-constituted step data as data for the form.

> /home/kmtracey/django/hg-django/django/contrib/formtools/wizard/storage/cookie.py(14)__init__()
-> if self.data is None:
(Pdb) self.data['step_data']['form1']
{u'form1-user': [u'1'], u'cookie_contact_wizard-current_step': [u'form1'], u'form1-name': [u'Pony'], u'form1-thirsty': [u'2']}
(Pdb)

This is with simplejson 2.1.6 installed:

--> ./runtests.py --settings=testdb.sqlite -v0 formtools.CookieWizardTests.test_cleaned_data
> /home/kmtracey/django/hg-django/django/contrib/formtools/wizard/storage/cookie.py(32)update_response()
-> response.set_signed_cookie(self.prefix, self.encoder.encode(self.data))
(Pdb) c
> /home/kmtracey/django/hg-django/django/contrib/formtools/wizard/storage/cookie.py(27)load_data()
-> return json.loads(data, cls=json.JSONDecoder)
(Pdb) c
> /home/kmtracey/django/hg-django/django/contrib/formtools/wizard/storage/cookie.py(32)update_response()
-> response.set_signed_cookie(self.prefix, self.encoder.encode(self.data))
(Pdb) self.data
{u'step_files': {u'form1': {}}, u'step': u'form2', u'extra_data': {}, u'step_data': {u'form1': <QueryDict: {u'form1-name': [u'Pony'], u'cookie_contact_wizard-current_step': [u'form1'], u'form1-user': [u'1'], u'form1-thirsty': [u'2']}>}}
(Pdb) self.data['step_data']['form1']
<QueryDict: {u'form1-name': [u'Pony'], u'cookie_contact_wizard-current_step': [u'form1'], u'form1-user': [u'1'], u'form1-thirsty': [u'2']}>
(Pdb) c
> /home/kmtracey/django/hg-django/django/contrib/formtools/wizard/storage/cookie.py(27)load_data()
-> return json.loads(data, cls=json.JSONDecoder)
(Pdb) n
--Return--
> /home/kmtracey/django/hg-django/django/contrib/formtools/wizard/storage/cookie.py(27)load_data()->{u'extra_data': {}, u'step': u'form2', u'step_data': {u'form1': {u'cookie...ent_step': u'form1', u'form1-name': u'Pony', u'form1-thirsty': u'2', u'form1-user': u'1'}}, u'step_files': {u'form1': {}}}
-> return json.loads(data, cls=json.JSONDecoder)
(Pdb) n


# With this level of simplejson, the step_data for form1 does not have its values wrapped in lists

> /home/kmtracey/django/hg-django/django/contrib/formtools/wizard/storage/cookie.py(14)__init__()
-> if self.data is None:
(Pdb) self.data['step_data']['form1']
{u'form1-user': u'1', u'cookie_contact_wizard-current_step': u'form1', u'form1-name': u'Pony', u'form1-thirsty': u'2'}
(Pdb)

Thinking about it a bit more, I do kind of wonder whether the formwizard will correctly handle forms where multi-value keys are expected, since round-tripping through serialization you don't get the QueryDict with its special characteristics back, and the formwizard code seems to be using the de-serialized version for cleaning on the last step...

Huh, that is indeed a good question which I have to pass along to Stephan who wrote the new wizard initially..

comment:5 by Jannis Leidel, 13 years ago

Component: Forms → contrib.formtools

by steph, 13 years ago

Attachment: trac-16393.diff​ added

Fix without extra tests.

comment:6 by steph, 13 years ago

The patch fixes the failing tests but doesn't add new tests for the exact problem. I tested this change with simplejson 2.0.9 and simplejson 2.1.6.

Last edited 13 years ago by steph (previous) (diff)

comment:7 by Karen Tracey, 13 years ago

Owner: changed from nobody to Karen Tracey

comment:8 by Alex Gaynor, 13 years ago

Resolution: → fixed
Status: new → closed

In [17014]:

Fixed #16393 -- FormWizard's cookie storage backend now works with all versions of simplejson and the standard library json module.

Note: See TracTickets for help on using tickets.
Back to Top