Opened 6 years ago

Closed 6 years ago

Last modified 5 years ago

#27675 closed Bug (needsinfo)

Django postgres JSONField encoding

Reported by: Hisham Waleed Karam Owned by: nobody
Component: contrib.postgres Version: 1.10
Severity: Normal Keywords: models JSONField
Cc: Triage Stage: Unreviewed
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

i have JSONField in a model this field store json in 'utf8' but when i am trying to get the value of this field it escaped

model field:

details = JSONField(blank=True, default={})

database row:
http://i.imgur.com/GVKprEK.png

code :

from stores.models import Item
i=Item.objects.all()[0]
i.details

output in console and django admin: i want django to show details without escaping

{u'\u0633\u064a\u0628': u'\u0634\u0633\u064a'}

Change History (8)

comment:1 Changed 6 years ago by Tim Graham

Component: Database layer (models, ORM)contrib.postgres

Can you provide a test case that demonstrates the issue?

comment:2 Changed 6 years ago by Tim Graham

Resolution: needsinfo
Status: newclosed

I can't reproduce the issue given the information provided.

comment:3 Changed 6 years ago by ErikW

Resolution: needsinfo
Status: closednew

I have the same thing happening to me.

I have a model using a JSONField (from django.contrib.postgres.fields). When I examine the field from an ipython shell (also happening when interacting with the field from views), it is returning the contents as a string instead of a dict (ie '{}'). I checked table in Postgres, and it is in fact a jsonb column. Not sure why it's not deserializing the field.

If I save the model, it then escapes the field again which adds the additional backslashes mentioned by Hisham above.

This behavior only started after upgrading this codebase from Django 1.9 to 1.10.5 -- the field worked correctly in 1.9.x.

    from django.contrib.postgres.fields import JSONField
    
    class CRMOnboarding(models.Model):
        data = JSONField(default=dict)

IPython Shell:

    In 13:58:31 [6]: x.crmonboarding_set.all().last()
    Out 13:58:31 [6]: <CRMOnboarding: CRM - bb69b886-25d5-4d59-a4d6-23bebc91af70>
    
    In 13:58:34 [7]: crm=_
    
    In 13:58:36 [8]: crm.data
    Out 13:58:36 [8]: '{"fields": {"dob": "null", "vin": "null", "city": "city", "email": "email", "phone": "phone", "state": "state", "Domain": "null", "Handle": "null", "Status": "null", "zip_code": "zip_code", "address_1": "address_1", "address_2": "address_2", "auto_fuel": "null", "auto_make": "null", "auto_size": "null", "auto_type": "null", "auto_year": "null", "last_name": "last_name", "auto_based": "null", "auto_class": "null", "auto_model": "null", "auto_style": "null", "first_name": "first_name", "Syntax good": "null", "ahh_num_cars": "null", "Invalid Reason": "null", "auto_make_model": "null", "confidence_code": "null", "Catch All Domain": "null", "auto_manufacturer": "null", "auto_purchase_date": "null", "auto_purchase_type": "null"}}'
    
    In 13:58:54 [9]: data=_
    
    In 13:58:57 [10]: crm.save()
    
    In 13:58:59 [11]: x.crmonboarding_set.all().last()
    Out 13:58:59 [11]: <CRMOnboarding: CRM - bb69b886-25d5-4d59-a4d6-23bebc91af70>
    
    In 13:59:01 [12]: crm=_
    
    In 13:59:02 [13]: crm.data
    Out 13:59:02 [13]: '"{\\"fields\\": {\\"dob\\": \\"null\\", \\"vin\\": \\"null\\", \\"city\\": \\"city\\", \\"email\\": \\"email\\", \\"phone\\": \\"phone\\", \\"state\\": \\"state\\", \\"Domain\\": \\"null\\", \\"Handle\\": \\"null\\", \\"Status\\": \\"null\\", \\"zip_code\\": \\"zip_code\\", \\"address_1\\": \\"address_1\\", \\"address_2\\": \\"address_2\\", \\"auto_fuel\\": \\"null\\", \\"auto_make\\": \\"null\\", \\"auto_size\\": \\"null\\", \\"auto_type\\": \\"null\\", \\"auto_year\\": \\"null\\", \\"last_name\\": \\"last_name\\", \\"auto_based\\": \\"null\\", \\"auto_class\\": \\"null\\", \\"auto_model\\": \\"null\\", \\"auto_style\\": \\"null\\", \\"first_name\\": \\"first_name\\", \\"Syntax good\\": \\"null\\", \\"ahh_num_cars\\": \\"null\\", \\"Invalid Reason\\": \\"null\\", \\"auto_make_model\\": \\"null\\", \\"confidence_code\\": \\"null\\", \\"Catch All Domain\\": \\"null\\", \\"auto_manufacturer\\": \\"null\\", \\"auto_purchase_date\\": \\"null\\", \\"auto_purchase_type\\": \\"null\\"}}"'
    
    In 13:59:26 [14]: import json
    
    In 13:59:29 [15]: json.loads(data)
    Out 13:59:29 [15]:
    {u'fields': {u'Catch All Domain': u'null',
      u'Domain': u'null',
      u'Handle': u'null',
      u'Invalid Reason': u'null',
      u'Status': u'null',
      u'Syntax good': u'null',
      u'address_1': u'address_1',
      u'address_2': u'address_2',
      u'ahh_num_cars': u'null',
      u'auto_based': u'null',
      u'auto_class': u'null',
      u'auto_fuel': u'null',
      u'auto_make': u'null',
      u'auto_make_model': u'null',
      u'auto_manufacturer': u'null',
      u'auto_model': u'null',
      u'auto_purchase_date': u'null',
      u'auto_purchase_type': u'null',
      u'auto_size': u'null',
      u'auto_style': u'null',
      u'auto_type': u'null',
      u'auto_year': u'null',
      u'city': u'city',
      u'confidence_code': u'null',
      u'dob': u'null',
      u'email': u'email',
      u'first_name': u'first_name',
      u'last_name': u'last_name',
      u'phone': u'phone',
      u'state': u'state',
      u'vin': u'null',
      u'zip_code': u'zip_code'}}
    
    In 13:59:33 [16]: crm.data
    Out 13:59:33 [16]: '"{\\"fields\\": {\\"dob\\": \\"null\\", \\"vin\\": \\"null\\", \\"city\\": \\"city\\", \\"email\\": \\"email\\", \\"phone\\": \\"phone\\", \\"state\\": \\"state\\", \\"Domain\\": \\"null\\", \\"Handle\\": \\"null\\", \\"Status\\": \\"null\\", \\"zip_code\\": \\"zip_code\\", \\"address_1\\": \\"address_1\\", \\"address_2\\": \\"address_2\\", \\"auto_fuel\\": \\"null\\", \\"auto_make\\": \\"null\\", \\"auto_size\\": \\"null\\", \\"auto_type\\": \\"null\\", \\"auto_year\\": \\"null\\", \\"last_name\\": \\"last_name\\", \\"auto_based\\": \\"null\\", \\"auto_class\\": \\"null\\", \\"auto_model\\": \\"null\\", \\"auto_style\\": \\"null\\", \\"first_name\\": \\"first_name\\", \\"Syntax good\\": \\"null\\", \\"ahh_num_cars\\": \\"null\\", \\"Invalid Reason\\": \\"null\\", \\"auto_make_model\\": \\"null\\", \\"confidence_code\\": \\"null\\", \\"Catch All Domain\\": \\"null\\", \\"auto_manufacturer\\": \\"null\\", \\"auto_purchase_date\\": \\"null\\", \\"auto_purchase_type\\": \\"null\\"}}"'
    
    In 13:59:38 [17]: crm.data = json.loads(data)
    
    In 13:59:42 [19]: crm.data
    Out 13:59:42 [19]:
    {u'fields': {u'Catch All Domain': u'null',
      u'Domain': u'null',
      u'Handle': u'null',
      u'Invalid Reason': u'null',
      u'Status': u'null',
      u'Syntax good': u'null',
      u'address_1': u'address_1',
      u'address_2': u'address_2',
      u'ahh_num_cars': u'null',
      u'auto_based': u'null',
      u'auto_class': u'null',
      u'auto_fuel': u'null',
      u'auto_make': u'null',
      u'auto_make_model': u'null',
      u'auto_manufacturer': u'null',
      u'auto_model': u'null',
      u'auto_purchase_date': u'null',
      u'auto_purchase_type': u'null',
      u'auto_size': u'null',
      u'auto_style': u'null',
      u'auto_type': u'null',
      u'auto_year': u'null',
      u'city': u'city',
      u'confidence_code': u'null',
      u'dob': u'null',
      u'email': u'email',
      u'first_name': u'first_name',
      u'last_name': u'last_name',
      u'phone': u'phone',
      u'state': u'state',
      u'vin': u'null',
      u'zip_code': u'zip_code'}}
    
    In 13:59:46 [20]: crm.save()
    
    In 13:59:48 [21]: x.crmonboarding_set.all().last()
    Out 13:59:48 [21]: <CRMOnboarding: CRM - bb69b886-25d5-4d59-a4d6-23bebc91af70>
    
    In 13:59:50 [22]: crm=_
    
    In 13:59:52 [23]: crm.data
    Out 13:59:52 [23]: '{"fields": {"dob": "null", "vin": "null", "city": "city", "email": "email", "phone": "phone", "state": "state", "Domain": "null", "Handle": "null", "Status": "null", "zip_code": "zip_code", "address_1": "address_1", "address_2": "address_2", "auto_fuel": "null", "auto_make": "null", "auto_size": "null", "auto_type": "null", "auto_year": "null", "last_name": "last_name", "auto_based": "null", "auto_class": "null", "auto_model": "null", "auto_style": "null", "first_name": "first_name", "Syntax good": "null", "ahh_num_cars": "null", "Invalid Reason": "null", "auto_make_model": "null", "confidence_code": "null", "Catch All Domain": "null", "auto_manufacturer": "null", "auto_purchase_date": "null", "auto_purchase_type": "null"}}'

comment:4 Changed 6 years ago by Tim Graham

Could you please provide a test case for Django's test suite?

comment:5 Changed 6 years ago by Tim Graham

Resolution: needsinfo
Status: newclosed

In particular, it's unclear why crm.data is coming back as a string rather than as a dict. Is it possible your data is corrupted?

comment:6 Changed 5 years ago by grapemix

I got the same problem too. I am very sure the bug is not caused by data corruption. I double check with the db table type and the raw data in postgres db. The interesting thing is the data return dict type in django-admin.py shell environment, but the data will return string in the runserver and testcase environment.

Another experiment I did is that converting the string data to dict by json.loads() and save to db in the runserver environment. The data type of the JSONField is dict, but if I fetch the model again, the data type of the JSONField become string.

So please reopen the ticket. This bug gives us lots of trouble.

Thanks.

comment:7 Changed 5 years ago by Tim Graham

Steps to reproduce the issue are missing. You can reopen the ticket if you provide a test case to reproduce the issue.

comment:8 Changed 5 years ago by grapemix

After debugging like crazy for a few days, we got some good news.

Short story:
Django's JSONField cannot be mixed with django-jsonfield's JSONField to use. See: https://bitbucket.org/schinckel/django-jsonfield/issues/57/cannot-use-in-the-same-project-as-djangos, not even just in migration files. After removing all migrations file and run the make migrations cmd, everything is working.

Long story:
We had tried running Django's std postgres testcase and our testcase seperately and mix and matched to try to reproduce the problem. Since we used django-jsonfield's JSONField before and we would like to switch to Django's JSONField at once, we only keep django-jsonfield's JSONField at migrations file to 'transit'. That's why the test result is so in-consistence. After removing all migrations file and run the make migrations cmd, everything is working.

Since we only tested on our dev environment, we are not sure the transition in the prod. If we have not screamed in the following days, that means the transition in the prod is good.
Thanks everyone for your time.

Note: See TracTickets for help on using tickets.
Back to Top