BaseModelFormSet.validate_unique() raises unhashable type error for unique fields with unhashable types

[Hanne Moa]

Given a User based on AbstractUser and a model:

class DestinationConfig(model.Model):
    class Meta:
        constraints = [models.UniqueConstraint(fields=["user", "settings"], name="unique_destination_per_user")]

    user = models.ForeignKey(User, on_delete=models.CASCADE)
    settings = models.JSONField()

Admin like so:

from django.contrib import admin
from django.contrib.auth.admin import UserAdmin as BaseUserAdmin

class DestinationConfigInline(admin.TabularInline):
    model = DestinationConfig
    ordering = ["media", "label"]
    extra = 1

    def get_formset(self, request, obj=None, **kwargs):
        self.parent_obj = obj
        return super(DestinationConfigInline, self).get_formset(request, obj, **kwargs)

    def get_queryset(self, request):
        qs = super(DestinationConfigInline, self).get_queryset(request)
        return qs.filter(user=self.parent_obj)

class UserAdmin(BaseUserAdmin):
    inlines = [DestinationConfigInline]

admin.site.register(User, UserAdmin)

Then attempting to save a change to a specific user leads to a traceback like this:

Environment:


Request Method: POST
Request URL: https://ACENSORED.DOMAIN/admin/APP_auth/user/65/change/

Django Version: 5.0.6
Python Version: 3.10.12
Installed Applications:
['channels',
 'django.contrib.admin',
 'django.contrib.auth',
 'django.contrib.contenttypes',
 'django.contrib.sessions',
 'django.contrib.messages',
 'django.contrib.staticfiles',
 'corsheaders',
 'social_django',
 'rest_framework',
 'rest_framework.authtoken',
 'drf_spectacular',
 'django_filters',
 'phonenumber_field',
 'argus.auth',
 'argus.incident',
 'argus.ws',
 'argus.notificationprofile',
 'argus.dev']
Installed Middleware:
['django.middleware.security.SecurityMiddleware',
 'corsheaders.middleware.CorsMiddleware',
 'whitenoise.middleware.WhiteNoiseMiddleware',
 'django.contrib.sessions.middleware.SessionMiddleware',
 'django.middleware.common.CommonMiddleware',
 'django.middleware.csrf.CsrfViewMiddleware',
 'django.contrib.auth.middleware.AuthenticationMiddleware',
 'django.contrib.messages.middleware.MessageMiddleware',
 'django.middleware.clickjacking.XFrameOptionsMiddleware',
 'social_django.middleware.SocialAuthExceptionMiddleware',
 'django.contrib.auth.middleware.RemoteUserMiddleware']

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/django/core/handlers/exception.py", line 55, in inner
    response = get_response(request)
  File "/usr/local/lib/python3.10/site-packages/django/core/handlers/base.py", line 197, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/local/lib/python3.10/site-packages/django/contrib/admin/options.py", line 688, in wrapper
    return self.admin_site.admin_view(view)(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/django/utils/decorators.py", line 134, in _wrapper_view
    response = view_func(request, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/django/views/decorators/cache.py", line 62, in _wrapper_view_func
    response = view_func(request, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/django/contrib/admin/sites.py", line 242, in inner
    return view(request, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/django/contrib/admin/options.py", line 1889, in change_view
    return self.changeform_view(request, object_id, form_url, extra_context)
  File "/usr/local/lib/python3.10/site-packages/django/utils/decorators.py", line 46, in _wrapper
    return bound_method(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/django/utils/decorators.py", line 134, in _wrapper_view
    response = view_func(request, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/django/contrib/admin/options.py", line 1747, in changeform_view
    return self._changeform_view(request, object_id, form_url, extra_context)
  File "/usr/local/lib/python3.10/site-packages/django/contrib/admin/options.py", line 1797, in _changeform_view
    if all_valid(formsets) and form_validated:
  File "/usr/local/lib/python3.10/site-packages/django/forms/formsets.py", line 579, in all_valid
    return all([formset.is_valid() for formset in formsets])
  File "/usr/local/lib/python3.10/site-packages/django/forms/formsets.py", line 579, in <listcomp>
    return all([formset.is_valid() for formset in formsets])
  File "/usr/local/lib/python3.10/site-packages/django/forms/formsets.py", line 384, in is_valid
    self.errors
  File "/usr/local/lib/python3.10/site-packages/django/forms/formsets.py", line 366, in errors
    self.full_clean()
  File "/usr/local/lib/python3.10/site-packages/django/forms/formsets.py", line 456, in full_clean
    self.clean()
  File "/usr/local/lib/python3.10/site-packages/django/forms/models.py", line 789, in clean
    self.validate_unique()
  File "/usr/local/lib/python3.10/site-packages/django/forms/models.py", line 831, in validate_unique
    if row_data in seen_data:

Exception Type: TypeError at /admin/APP_auth/user/65/change/
Exception Value: unhashable type: 'dict'

• row_data was ('user', {'email_address': 'CENSORED@ANOTHERCENSORED.DOMAIN', 'html': True})
• seen_data was set().

Is it a design decision that JSONFields cannot be unique/in UniqueConstraints?

• If yes, can it be documented, preferably with a note as to what to do if you *do* want a unique JSONField?
• If no, can validate_unique be changed to work in this instance?

This has also been tested on:

• Django Version: 4.2.11
• Python Version: 3.10.14

[Sarah Boyce]

This is different to #23964 but could be fixed as part of #23964.

Is it a design decision that JSONFields cannot be unique/in UniqueConstraints?

No, looks like you have uncovered a bug specific to model formsets 👍 thank you for the report

[Simon Charette]

The problem happens to manifests itself for JSONField with UniqueConstraint but it happens for any field with a non-hashable value that has a unique constraint defined on it.

e.g.

class FooBar(models.Model):
    settings = models.JSONField(unique=True)

Would exhibit the same problem and the same could be said of HStoreField or other custom fields dealing with dict, set, and other non-hashable data types.

Using django.utils.make_hashable in BaseModelFormSet.validate_unique seems like a potential low-lift solution but even this function can raise a TypeError if dealing with non-hashable value so maybe we're better off silencing these TypeError and let the model level unique constraint validation kick in.

I think the latter would be a better approach because uniqueness on such fields cannot be determined at the Python level. For example, JSONField relies on the jsonb type on Postgres so the ordering of keys is not preserved which means that {"foo": "bar", "bar": "foo"} would be considered equal to {"bar": "foo", "foo": "bar"} but not on MySQL which json type is a basically longtext with JSON validation and preserves insertion order.

[Simon Charette]

I think this has little to do with #23964 (which focuses on Meta.constraints) and is more of an analogous to #26819 (which was for another field using non-hashable types) but for JSONField.

django/forms/models.py

@@ -23 +23 @@
     SelectMultiple,
 )
 from django.utils.choices import BaseChoiceIterator
+from django.utils.hashable import make_hashable
 from django.utils.text import capfirst, get_text_list
 from django.utils.translation import gettext
 from django.utils.translation import gettext_lazy as _
@@ -835 +836 @@
                         d._get_pk_val()
                         if hasattr(d, "_get_pk_val")
                         # Prevent "unhashable type: list" errors later on.
-                        else tuple(d) if isinstance(d, list) else d
+                        else make_hashable(d)
                     )
                     for d in row_data
                 )

The problem IMO is that 06a11ef6ecf324db0a1530b8cca727883698f442 focused on one type on unhashable value instead of the generic problem.

[Sarah Boyce]

Yes I agree, thank you Simon 👍

[Simon Charette]

Marking as easy picking as all that is required here is likely to submit a PR with the change above to use make_hashable and add a test similar to the one added in 06a11ef6ecf324db0a1530b8cca727883698f442 but using a form.JSONField instead.

It might be a good one to pick up if you've never contributed to Django Hanne!

[Madalin Popa]

Created PR: ​https://github.com/django/django/pull/18245

[Sarah Boyce <42296566+sarahboyce@…>]

In d28626e:

Fixed #35488 -- Fixed BaseModelFormSet.validate_unique() crash due to unhashable type.

[Sarah Boyce <42296566+sarahboyce@…>]

In dbd1a8bd:

[5.1.x] Fixed #35488 -- Fixed BaseModelFormSet.validate_unique() crash due to unhashable type.

Backport of d28626ecf8bd340084ed70ff2d88e8dbab001e2c from main.