Django 3.1.2 JSONField handling error with an Postgres Foreign Data Wrapper-based model

[Shaheed Haque]

Over at ​https://groups.google.com/g/django-users/c/Sr1TDVRpCLE, it was suggested that I post an issue here...

I have a Django model working fine under Django 3.0 (i.e. "Django<3.1") which looks like this:

    class Job(models.Model):
        id = models.CharField(max_length=36, primary_key=True)
        queue = models.CharField(max_length=40)
        args = JSONField()
        kwargs = JSONField()
        type = models.CharField(max_length=80)
        ...

        class Meta:
            managed = False  # <------   The table is implemented as a Postgres FDW wrapper.
            db_table = 'jobs'

I am testing the update to Django 3.1.2 and hit an error in executing this line:

jobs = list(models.Job.objects.filter(queue='celery', state='scheduled'))

The error is as follows from pytest (i.e. stack trace with local variables too):

_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/local/lib/python3.8/dist-packages/django/db/models/query.py:287: in __iter__
   self._fetch_all()
       self       = <QuerySet []>
/usr/local/lib/python3.8/dist-packages/django/db/models/query.py:1308: in _fetch_all
   self._result_cache = list(self._iterable_class(self))
       self       = <QuerySet []>
/usr/local/lib/python3.8/dist-packages/django/db/models/query.py:70: in __iter__
   for row in compiler.results_iter(results):
       annotation_col_map = {}
       compiler   = <django.db.models.sql.compiler.SQLCompiler object at 0x7f8685e49160>
       db         = 'fdw'
       init_list  = ['id', 'queue', 'args', 'kwargs', 'type', 'state', ...]
       klass_info = {'model': <class 'paiyroll.models.batch.Job'>, 'select_fields': [0, 1, 2, 3, 4, 5, ...]}
       known_related_objects = []
       model_cls  = <class 'paiyroll.models.batch.Job'>
       model_fields_end = 9
       model_fields_start = 0
       queryset   = <QuerySet []>
       related_populators = []
       results    = [[('8f4ab6b0-914f-4a75-972d-febbe55011fc', 'celery', ['paiyroll.tasks', 'call_to_string', 'call_to_string', [], {}], {}, 'paiyroll.tasks.function_run', 'scheduled', ...)]]
       select     = [(Col(jobs, paiyroll.Job.id), ('"jobs"."id"', []), None), (Col(jobs, paiyroll.Job.queue), ('"jobs"."queue"', []), None..., paiyroll.Job.type), ('"jobs"."type"', []), None), (Col(jobs, paiyroll.Job.state), ('"jobs"."state"', []), None), ...]
       select_fields = [0, 1, 2, 3, 4, 5, ...]
       self       = <django.db.models.query.ModelIterable object at 0x7f86836f3040>
/usr/local/lib/python3.8/dist-packages/django/db/models/sql/compiler.py:1100: in apply_converters
   value = converter(value, expression, connection)
       connection = <django.db.backends.postgresql.base.DatabaseWrapper object at 0x7f869a321670>
       converter  = <bound method JSONField.from_db_value of <django.contrib.postgres.fields.jsonb.JSONField: args>>
       converters = [(2, ([<bound method JSONField.from_db_value of <django.contrib.postgres.fields.jsonb.JSONField: args>>], Col(jobs, pa...NField.from_db_value of <django.contrib.postgres.fields.jsonb.JSONField: details>>], Col(jobs, paiyroll.Job.details)))]
       convs      = [<bound method JSONField.from_db_value of <django.contrib.postgres.fields.jsonb.JSONField: args>>]
       expression = Col(jobs, paiyroll.Job.args)
       pos        = 2
       row        = ['8f4ab6b0-914f-4a75-972d-febbe55011fc', 'celery', ['paiyroll.tasks', 'call_to_string', 'call_to_string', [], {}], {}, 'paiyroll.tasks.function_run', 'scheduled', ...]
       rows       = <itertools.chain object at 0x7f8683ae7520>
       self       = <django.db.models.sql.compiler.SQLCompiler object at 0x7f8685e49160>
       value      = ['paiyroll.tasks', 'call_to_string', 'call_to_string', [], {}]
/usr/local/lib/python3.8/dist-packages/django/db/models/fields/json.py:74: in from_db_value
   return json.loads(value, cls=self.decoder)
       connection = <django.db.backends.postgresql.base.DatabaseWrapper object at 0x7f869a321670>
       expression = Col(jobs, paiyroll.Job.args)
       self       = <django.contrib.postgres.fields.jsonb.JSONField: args>
       value      = ['paiyroll.tasks', 'call_to_string', 'call_to_string', [], {}]
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

s = ['paiyroll.tasks', 'call_to_string', 'call_to_string', [], {}], cls = None
object_hook = None, parse_float = None, parse_int = None, parse_constant = None
object_pairs_hook = None, kw = {}

   def loads(s, *, cls=None, object_hook=None, parse_float=None,
           parse_int=None, parse_constant=None, object_pairs_hook=None, **kw):
       """Deserialize ``s`` (a ``str``, ``bytes`` or ``bytearray`` instance
       containing a JSON document) to a Python object.

       ``object_hook`` is an optional function that will be called with the
       result of any object literal decode (a ``dict``). The return value of
       ``object_hook`` will be used instead of the ``dict``. This feature
       can be used to implement custom decoders (e.g. JSON-RPC class hinting).

       ``object_pairs_hook`` is an optional function that will be called with the
       result of any object literal decoded with an ordered list of pairs.  The
       return value of ``object_pairs_hook`` will be used instead of the ``dict``.
       This feature can be used to implement custom decoders.  If ``object_hook``
       is also defined, the ``object_pairs_hook`` takes priority.

       ``parse_float``, if specified, will be called with the string
       of every JSON float to be decoded. By default this is equivalent to
       float(num_str). This can be used to use another datatype or parser
       for JSON floats (e.g. decimal.Decimal).

       ``parse_int``, if specified, will be called with the string
       of every JSON int to be decoded. By default this is equivalent to
       int(num_str). This can be used to use another datatype or parser
       for JSON integers (e.g. float).

       ``parse_constant``, if specified, will be called with one of the
       following strings: -Infinity, Infinity, NaN.
       This can be used to raise an exception if invalid JSON numbers
       are encountered.

       To use a custom ``JSONDecoder`` subclass, specify it with the ``cls``
       kwarg; otherwise ``JSONDecoder`` is used.

       The ``encoding`` argument is ignored and deprecated since Python 3.1.
       """
       if isinstance(s, str):
           if s.startswith('\ufeff'):
               raise JSONDecodeError("Unexpected UTF-8 BOM (decode using utf-8-sig)",
                                     s, 0)
       else:
           if not isinstance(s, (bytes, bytearray)):
>               raise TypeError(f'the JSON object must be str, bytes or bytearray, '
                               f'not {s.__class__.__name__}')
E               TypeError: the JSON object must be str, bytes or bytearray, not list

cls        = None
kw         = {}
object_hook = None
object_pairs_hook = None
parse_constant = None
parse_float = None
parse_int  = None
s          = ['paiyroll.tasks', 'call_to_string', 'call_to_string', [], {}]

As you can perhaps see from the section for /usr/local/lib/python3.8/dist-packages/django/db/models/fields/json.py:74, the value being written into the JSONField called "args" is a Python list, (shown as "s" on the last line of the traceback-with-values). I am aware of documented changes around serializers but did not think that affected me; however, given that I am using an FDW to return the data, is it that I should now be serialising returned data into a string?

Any pointers appreciated.

Thanks, Shaheed

[Mariusz Felisiak]

It looks that you use json instead of jsonb datatype in your database, which is not supported.

Duplicate of #31973.