Resolve lazy objects inside dictionaries when saving JSONFields

[Jacob Walls]

I have a model Notification like this:

class Notification(models.Model):
    message = models.TextField(null=True)
    context = models.JSONField(null=True)

Here is the lazy object...

>>> from django.utils.translation import gettext_lazy as _
>>> lazy = _("lazy")

... that I can use in a TextField ...

>>> models.Notification(message=lazy).save()
>>> 

... but not for the JSONField without string-ifying it myself. Suggesting an implementation that resolves these in one of the methods that prepares values for database insertion. Tested with psycopg2 2.9.9.

>>> models.Notification(context={'lazy': lazy}).save()
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/Users/jwalls/venv311/lib/python3.11/site-packages/django/db/models/base.py", line 814, in save
    self.save_base(
  File "/Users/jwalls/venv311/lib/python3.11/site-packages/django/db/models/base.py", line 877, in save_base
    updated = self._save_table(
              ^^^^^^^^^^^^^^^^^
  File "/Users/jwalls/venv311/lib/python3.11/site-packages/django/db/models/base.py", line 990, in _save_table
    updated = self._do_update(
              ^^^^^^^^^^^^^^^^
  File "/Users/jwalls/venv311/lib/python3.11/site-packages/django/db/models/base.py", line 1054, in _do_update
    return filtered._update(values) > 0
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jwalls/venv311/lib/python3.11/site-packages/django/db/models/query.py", line 1231, in _update
    return query.get_compiler(self.db).execute_sql(CURSOR)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jwalls/venv311/lib/python3.11/site-packages/django/db/models/sql/compiler.py", line 1982, in execute_sql
    cursor = super().execute_sql(result_type)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jwalls/venv311/lib/python3.11/site-packages/django/db/models/sql/compiler.py", line 1560, in execute_sql
    cursor.execute(sql, params)
  File "/Users/jwalls/venv311/lib/python3.11/site-packages/django/db/backends/utils.py", line 102, in execute
    return super().execute(sql, params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jwalls/venv311/lib/python3.11/site-packages/django/db/backends/utils.py", line 67, in execute
    return self._execute_with_wrappers(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jwalls/venv311/lib/python3.11/site-packages/django/db/backends/utils.py", line 80, in _execute_with_wrappers
    return executor(sql, params, many, context)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jwalls/venv311/lib/python3.11/site-packages/django/db/backends/utils.py", line 89, in _execute
    return self.cursor.execute(sql, params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jwalls/venv311/lib/python3.11/site-packages/psycopg2/_json.py", line 78, in getquoted
    s = self.dumps(self.adapted)
        ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jwalls/venv311/lib/python3.11/site-packages/psycopg2/_json.py", line 72, in dumps
    return self._dumps(obj)
           ^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/json/__init__.py", line 231, in dumps
    return _default_encoder.encode(obj)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/json/encoder.py", line 200, in encode
    chunks = self.iterencode(o, _one_shot=True)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/json/encoder.py", line 258, in iterencode
    return _iterencode(o, 0)
           ^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/json/encoder.py", line 180, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type __proxy__ is not JSON serializable

[Mariusz Felisiak]

It's easier to resolve a type for string-based fields, however, JSONField's values can contain different types of data. I don't think it's worth adding extra guess-like complexity for this. We can reconsider this decision if you provide a PoC.

[Jacob Walls]

I was thinking of something like this:

django/db/models/fields/json.py

@@ +1 @@
+from collections.abc import Container
 import json
 
 from django import forms
+from django.conf import settings
 from django.core import checks, exceptions
 from django.db import NotSupportedError, connections, router
 from django.db.models import expressions, lookups
@@ -11 +13 @@
     PostgresOperatorLookup,
     Transform,
 )
+from django.utils.functional import Promise
 from django.utils.translation import gettext_lazy as _
 
 from . import Field
@@ -96 +99 @@
     def get_internal_type(self):
         return "JSONField"
 
+    def get_prep_value(self, value):
+        if settings.USE_I18N:
+            match value:
+                case dict():
+                    value = {
+                        self.get_prep_value(k): self.get_prep_value(v)
+                        for k, v in value.items()
+                    }
+                case str() | Promise():
+                    pass
+                case Container():
+                    value = value.__class__(self.get_prep_value(v) for v in value)
+
+        return super().get_prep_value(value)
+
     def get_db_prep_value(self, value, connection, prepared=False):
         if not prepared:
             value = self.get_prep_value(value)

[Mariusz Felisiak]

Why do you want to put it behind the USE_I18N?

[Jacob Walls]

Yeah, I hadn't yet looked into whether lazy translation objects get resolved when USE_I18N = False. Looks like they do, so I should remove the guard.

[Mariusz Felisiak]

Tentatively accepted.

[Jacob Walls]

So I have something sort of ready to go here, but then I noticed that what I should probably be doing is using DjangoJSONEncoder in the encoder arg of JSONField, which solves my use case. Is that enough? Should that be the default?

[Jacob Walls]

I guess it would still be nice to separate this concern (resolving the promise) from the choice of serializer.

[Jacob Walls]

​PR

[Claude Paroz]

I think using the encoder arg of JSONField is the right answer to this issue. The get_prep_value of your suggested patch would add too much overhead in my opinion.

[Simon Charette]

I completely agree with Claude here.

As a matter of fact DjangoJSONEncoder already works perfectly for this use case so I don't understand why we should treat promises differently ​given we already document using it for such cases.

import json
from django.core.serializers.json import DjangoJSONEncoder
from django.utils.translation import gettext_lazy as _

json.dumps({"key": _("foobar")})
# TypeError: Object of type __proxy__ is not JSON serializable
json.dumps({"key": _("foobar")}, cls=DjangoJSONEncoder)
>>> '{"key": "foobar"}'

[Jacob Walls]

Makes sense. Docs links are very helpful. Sorry for not using a support channel to check my understanding first!