Opened 14 years ago

Closed 4 years ago

Last modified 3 years ago

#12990 closed New feature (fixed)

Add JSONField model field

Reported by: paltman Owned by: Sage Abdullah
Component: Database layer (models, ORM) Version: dev
Severity: Normal Keywords:
Cc: paltman@…, self.anderson@…, t.django@…, slav0nic0@…, NicoEchaniz, dan.fairs@…, hv@…, ShawnMilo, Torsten Bronger, antonis+djangoproject.com@…, Mariusz Felisiak, Giacomo Graziosi Triage Stage: Ready for checkin
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

I have been using this for over a year now and find it quite useful. I posted it about it the other day (http://paltman.com/2010/feb/25/how-to-store-arbitrary-data-in-a-django-model/) and then thought it might be generally useful as an out of the box field type so I thought I'd submit a patch in case it is deemed "generally useful" enough to be considered for inclusion in django proper.

Attachments (1)

json_field.diff (4.8 KB ) - added by paltman 14 years ago.
JSON Field Patch

Download all attachments as: .zip

Change History (69)

by paltman, 14 years ago

Attachment: json_field.diff added

JSON Field Patch

comment:1 by Alex Gaynor, 14 years ago

milestone: 1.2
Triage Stage: UnreviewedDesign decision needed

I'm -1 on this, I think it's out of scope for Django core, and easily accomplished outside. Regardless I'm removing the 1.2 milestone, 1.2 is in feature freeze.

comment:2 by floguy, 14 years ago

+1 on this. There are many implementations of this floating around, and most of them are wrong. It's a common-enough pattern that Django should provide the canonical implementation.

comment:3 by Marcelo Ramos, 14 years ago

+1 If XMLField is in the core then JSONField deserves to be in too. Also I'm tempted to propose the removal of XMLField, you all know that
the use of XML as a format to store arbitrary data has dramatically decreased in favour of JSON.

in reply to:  3 ; comment:4 by Alex Gaynor, 14 years ago

Replying to floguy:

+1 on this. There are many implementations of this floating around, and most of them are wrong. It's a common-enough pattern that Django should provide the canonical implementation.

How are most of them wrong? It's pretty trivial (there's actually an implementation in the tests).

Replying to marcelor:

+1 If XMLField is in the core then JSONField deserves to be in too. Also I'm tempted to propose the removal of XMLField, you all know that
the use of XML as a format to store arbitrary data has dramatically decreased in favour of JSON.

XMLField at this point does almost nothing (doesn't even do validation at this point).

comment:5 by paltman, 14 years ago

Cc: paltman@… added

comment:7 by Alex Gaynor, 14 years ago

The fact that people don't know how to organize code externally cannot act as an argument in favor of including everything in django's core.

comment:8 by floguy, 14 years ago

The proliferation of implementations proves that there's significant interest in the feature. Since each of the implementations are slightly different, it makes it difficult to rely on the behavior of any one implementation. Also, some of the implementations behave incorrectly when using dumpdata/loaddata. Including it in core would solve all of these problems, as well as provide much lower friction for a new user who does not know which implementation to choose.

comment:9 by Peter Bengtsson, 14 years ago

+1 on this. I'm not sure what it means in terms of exports/dumps/fixtures etc.

@Alex: the reason we do Django is very much because it's a convenient frameworks that helps getting things done quickly.

comment:10 by anonymous, 14 years ago

Cc: self.anderson@… added

comment:11 by anonymous, 14 years ago

Cc: t.django@… added

comment:12 by Sergey Maranchuk, 14 years ago

Cc: slav0nic0@… added

comment:13 by Jeff Bauer, 14 years ago

+1 me too, vote. I'd really like to see this added to Django. A single implementation of JSONField would be an improvement over the current state.

comment:14 by NicoEchaniz, 13 years ago

Cc: NicoEchaniz added

comment:15 by Łukasz Rekucki, 13 years ago

Needs tests: set
Patch needs improvement: set

The patch has some tests, but their coverage is very limited. Some things that need testing:

  • get_*_json/set_*_json hooks
  • does this work properly with dumpdata/loaddata in all formats ?

Having an option to override the encoder/decoder without having to subclass the field would be nice (like in some other implementations).

A major argument for me against having this in core, is that you can't include South support (at least atm, we'll see what happens in 1.4), so I would still prefer a 3rd party implementation that does.

Last edited 13 years ago by Łukasz Rekucki (previous) (diff)

comment:16 by Dan Fairs, 13 years ago

Cc: dan.fairs@… added

comment:17 by Luke Plant, 13 years ago

Type: New feature

comment:18 by Luke Plant, 13 years ago

Severity: Normal

comment:19 by joshg, 13 years ago

I just wanted to say, I would like this to be built into the core. Thanks!

comment:20 by Mark Jones, 13 years ago

Easy pickings: unset

+1 Considering all the versions of this floating out there and all the patches due to them not being quite right. Putting this is to Django seems to be the right way to do it.

comment:21 by Russell Keith-Magee, 13 years ago

For the record - I'm -0 on this. For my money, if you're putting JSON in a database field (and expecting it to be searchable etc), you're doing it wrong. I won't stand in the way if someone else in the core team feels particularly enthused, but I'm not rushing myself.

Also, the previous argument about XMLField has been rendered invalid -- we removed XMLField in 1.3.

comment:22 by Jannis Leidel, 13 years ago

Agreed, I think projects like https://github.com/jordanm/django-hstore show that there are db specific fields that are better suited for storing JSON.

comment:23 by sorl, 13 years ago

I think the usecase for something like this should be simply serialized data in a field, not searchable and not just a mapping to a dict. Given that, we might as well use pickle, but then we might loose data portability. If you want to build an efficient relational db centric application you probably want to use something like this and I think many people realize that but the many implementations floating around makes it hard for them to decide which one to use. If Django wants to advocate for that kind of database design then perhaps it is a good idea to include a serialization field. I have a very simple lazier approach, it can't serialize strings but it only deserializes on attr access, https://github.com/aino/django-cerial/blob/master/cerial/fields.py

comment:24 by Thomas Güttler, 13 years ago

Cc: hv@… added

comment:25 by ShawnMilo, 13 years ago

Cc: ShawnMilo added

I'd like to see this as well, because we use a JSON field frequently. I think the argument against that searching JSON in a queryset is poor design is a straw-man argument. I haven't heard anyone ask for a JSON field with this in mind. It's just convenient to store serialized data to avoid unnecessary auxiliary storage and proliferation of extra fields and models.

Having a canonical JSON field is better than people implementing their own, in my opinion, because it makes it provides consistency across projects without needing to copy/paste code into each new project.

Last edited 13 years ago by ShawnMilo (previous) (diff)

in reply to:  4 comment:26 by Fredde, 13 years ago

Replying to Alex:

How are most of them wrong? It's pretty trivial (there's actually an implementation in the tests).

The JSONField included in the in the django tests ([source:django/trunk/tests/modeltests/field_subclassing/fields.py#L56]) is actually not working properly eather. For example, when trying to store a empty list [] or dict {} the JSONField.to_python will convert it to None:

#python
>>> class A(models.Model):
	json_field = JSONField()

>>> a = A()
>>> a.json_field = []
>>> print a.json_fied
None

I don't know how to fix this. From the documentation the following values should be handled by the to_python method:

  • An instance of the correct type (e.g., Hand in our ongoing example).
  • A string (e.g., from a deserializer).
  • Whatever the database returns for the column type you're using.

Hovever, there is no way to know if the value is a "correct type" a serialized representation or the value from the database. For example value = '[1, 2, 3]'. It could eather come from an attribute assignment, or from the database, but it should only be deserialized if it's from the database.

comment:27 by Phillip Marshall, 13 years ago

Cc: wizpig64@… added
UI/UX: unset

in reply to:  21 comment:28 by anonymous, 13 years ago

Replying to russellm:

For the record - I'm -0 on this. For my money, if you're putting JSON in a database field (and expecting it to be searchable etc), you're doing it wrong.

You cannot say that generally. The answer is as always, it depends. There are perfectly valid uses of this, for example, storing a little (dynamic) metadata without setting up a NoSQL database.

comment:29 by Russell Keith-Magee, 13 years ago

If you're arguing for JSONField as a way of avoiding setting up a NoSQL store, then you've just proved my point. If you need a non-relational store, *use a non-relational store*. Yes, there may be reasons why this isn't possible for political or business reasons, but that doesn't change the fact that:

  • There is a better way to solve the problem
  • Django shouldn't be encouraging people to hack around the root cause of problems
  • The absence of a JSONField in Django core doesn't prevent the community from developing a canonical implementation that can be used as a third-party library.
  • Putting JSONField in core wouldn't affect the stability or reliability of that third party implementation -- it just means the core team are the only people who can maintain it.

comment:30 by Thomas Güttler, 13 years ago

By setting up a NoSQL store you loose the transactional atomic behavior: commit all or nothing. AFAIK two-phase commit is not supported by django. I think a JSONField inside django would be good. I don't want to query the JSON value. I just use JSONField like cache that never expires.

in reply to:  30 comment:31 by Lee S, 13 years ago

I would love to see a canonical version of JSON fields included.

I routinely find storing non-structured, non-searchable data in the database to be extremely useful in many use cases. Setting up a NoSQL database is often overkill. The real need is to store the odd JSON field in an otherwise traditional SQL model. The example cited above, storing flexible metadata is a common one -- for example, key/value pairs indicating user preferences.

I'm at the point where I have to try one of the many implementations listed above and cross my fingers that it's not broken.

comment:32 by Aymeric Augustin, 13 years ago

I've recently rolled my own implementation of this.

I use it in a logging table where I want to store an event_kind (ex : "USER_LOGIN") + a datetime + a bunch of parameters (ex : username, remote IP) that depend on the kind of event. I want to be able to add arbitrary parameters at any point. Normalizing parameters in a separate table (e.g. storing event_id, key, value in another table for each parameter) seems overkill — I don't want to add a model just for this.

So I think there are some real-life use cases for storing a JSON-serialized dict in a database field. Since it's moderately difficult to get right, +0 for adding it in Django.

comment:33 by radbrad182+djangoproject@…, 13 years ago

JSON fields are also a great way to store basic, dynamic, typed data. Being able to store Key Value data in the DB would be useful:

class Key(models.Model):
   user = models.ForeignKey(User)
   name = models.CharField()
   value = models.JSONField()

user.key_set.create(name="an_integer", value=4)
user.key_set.create(name="a_list", value=['a','b','c'])

Normally to get this kind of basic data typing you would need to store another field for value_type, and then parse manually.

Arguing that this should be stored in a K/V store is not ideal as it adds dependency on another datastore type, and you lose out on user.key_set.get(name='something').

comment:34 by Torsten Bronger, 12 years ago

Cc: Torsten Bronger added

comment:35 by Riccardo Di Virgilio, 12 years ago

Version: 1.2-beta1.2-alpha

this is my proposition to for the field.

class JsonField(models.Field): 
    __metaclass__ = models.SubfieldBase 
    serialize_to_string = True 
    def get_internal_type(self): 
        return "TextField" 
    def value_to_string(self, obj): 
        return self.get_prep_value(self._get_val_from_obj(obj)) 
    def get_prep_value(self, value): 
        if value: 
            stream = StringIO.StringIO() 
            simplejson.dump(value, stream, cls=DjangoJSONEncoder) 
            value = stream.getvalue() 
            stream.close() 
            return value 
        return None 
    def to_python(self, value): 
        if isinstance(value, (str, unicode)): 
            value = StringIO.StringIO(value) 
            return simplejson.load(value) 
        return value 

maybe there is no need to to create a class
JSONDateEncoder(json.JSONEncoder) because there is DjangoJSONEncoder
already.

then it would be great to implement a lazy translation object for
DjangoJSONEncoder to store ugettext_lazy objects.

for me there is no need to create set_%s_json and get_%s_json method
because the field should handle it directly, and we should use the
to_python method, like all other fields in django.

for example the datetime field directly push a datetime object to the
model (using to_python) and transform it to a string when save
method is called

we should use the same logic here and encode a json object to string
only when we call the save method, no need for a get_%s_json in the
field api.

comment:36 by David Foster, 12 years ago

Cc: davidfstr@… added

comment:37 by anonymous, 12 years ago

What's the best third-party alternative out there? It seems like this is going to drag on for a while in a religious discussion of whether it should be included or not. Just add it and let people decide whether and how to use it.

comment:38 by Aymeric Augustin, 12 years ago

django-extensions is a fairly popular project that provides an implementation of JSONField.

comment:39 by Aymeric Augustin, 12 years ago

Resolution: wontfix
Status: newclosed

This idea was discussed and vetoed by one of our BDFLs on #django-social. Here are the relevant logs, edited for readability.

2012-01-20 21:18:10 <jacobkm> (...) we've explicitly decided *not* to include (...) JSONField

2012-01-20 21:21:58 <jacobkm> (...) I'm not saying there's no good reason for it, I'm saying the bad reasons outweigh the good and that's why it shouldn't ship with Django.

2012-01-20 21:22:11 <jacobkm> Under the Don't Give Users Loaded Guns Aimed At Feet principle.

comment:40 by gabewb@…, 11 years ago

It is pretty frustrating to google around for this, finally land here, and see that it's been vetoed by the principle "Don't Give Users Loaded Guns Aimed At Feet." That is weak principle. When it prevents people from using guns in dangerous ways, great, fine. But sometimes people really need to use a gun pointed in the downward direction, and if you don't give it to them, they do exactly what they did --- they make their own guns, and those guns all have their own sights, triggers, and safety mechanisms, and many of them don't work quite right or are liable to jam or backfire, and a newbie has neither the time nor the ability to pick the right one.

In this case, it's a lot better to include JSONField with a warning label than to keep it out entirely.

Incidentally, I found my way here because JSONField was the default custom field listed in https://docs.djangoproject.com/en/dev/ref/models/fields/

comment:41 by Luke Plant, 11 years ago

The WONTFIX will remain unless you bring this up on Django-devs.

However, one further reason why it is a bad idea is the existence of hstore in Postgres.

This is a database specific solution, and is a much better solution than storing JSON in a text field for many applications. Providing a built-in JSONField discourages people from finding better solutions.

Also Googling "django JSONField" gives you several solutions that are easily installable using PyPI, and work fine for most people, so I don't see how it is an issue. It's extremely easy to get hold of a working JSONField if you want one.

comment:42 by Anssi Kääriäinen, 11 years ago

For the record I think there are many valid use cases for storing JSON in the database. Also, PostgreSQL 9.2 already has JSON typed field, 9.3 might make it nicely searchable. If the JSON improvements get into 9.3 then it would actually be a better solution than HStore.

But, we likely don't want this field in core. If we are restricted to the functionality offered by all databases, then it basically boils down to "convert to text on save, convert to python on load". Which isn't all that great.

My vote is on improving core support for custom fields. If one can write a json field that just works outside of core, then "pip install django-jsonfield" seems like a good way to add this feature...

comment:43 by anonymous, 11 years ago

How disappointing. In my case, I want to store user-specific settings provided by the UI guys which are used purely for layout, styles, etc... I don't want to have to maintain normalised tables for a constantly-updating data structure which is only ever stored/retrieved as a whole.

I keep finding this in Django (and Python in general) - great ideas get vetoed because "we don't dare trust a developer to use tools properly".

comment:44 by David Foster, 11 years ago

Cc: davidfstr@… removed

comment:45 by Phillip Marshall, 11 years ago

Cc: wizpig64@… removed

comment:46 by tom.gruner@…, 11 years ago

If you "Don't Give Users Loaded Guns Aimed At Feet ", how do you ever expect them to dance ( ;

comment:47 by akanouras, 10 years ago

Cc: antonis+djangoproject.com@… added

comment:48 by Zach Borboa, 8 years ago

JSONField for Postgres is now available in Django 1.9

https://docs.djangoproject.com/en/1.9/ref/contrib/postgres/fields/#jsonfield

comment:49 by Tim Graham, 5 years ago

Has patch: unset
Needs tests: unset
Patch needs improvement: unset
Resolution: wontfixfixed
Summary: New Field Type: JSONFieldAdd JSONField model field
Triage Stage: Design decision neededAccepted
Version: 1.2-alphamaster

Reopening as per ticket:29821#comment:1.

comment:50 by Simon Charette, 5 years ago

Resolution: fixed
Status: closednew

I believe the next step is to come up with a compatibility table of all supported database types and which lookups we want to support.

comment:51 by Mariusz Felisiak, 5 years ago

Cc: Mariusz Felisiak added

comment:52 by Sage Abdullah, 5 years ago

Owner: changed from nobody to Sage Abdullah
Status: newassigned

I will be working on this ticket during Google Summer of Code 2019.

comment:53 by Sage Abdullah, 5 years ago

Has patch: set
Needs documentation: set
Patch needs improvement: set

PR.

Please note that it's still a draft and I haven't implemented custom lookups and transforms. More info on the PR's description.

comment:54 by Sage Abdullah, 5 years ago

Needs documentation: unset
Patch needs improvement: unset

The implementation is pretty much complete now, and docs have also been added.

comment:55 by Giacomo Graziosi, 5 years ago

Cc: Giacomo Graziosi added

comment:56 by Mariusz Felisiak <felisiak.mariusz@…>, 4 years ago

In 6f82df6:

Refs #12990 -- Moved CheckFieldDefaultMixin to the django.db.models.fields.mixins.

comment:57 by Sage Abdullah, 4 years ago

PR.

Moved to a fresh one.

comment:58 by GitHub <noreply@…>, 4 years ago

In 5c24c16e:

Refs #12990 -- Moved PostgresSimpleLookup to the django.db.models.lookups.PostgresOperatorLookup.

comment:59 by Mariusz Felisiak, 4 years ago

Triage Stage: AcceptedReady for checkin

comment:60 by Mariusz Felisiak <felisiak.mariusz@…>, 4 years ago

In f97f71f5:

Refs #12990 -- Bumped mysqlclient requirement to >= 1.4.0.

MySQLdb.constants.FIELD_TYPE.JSON was added in mysqlclient 1.4.0rc2.

comment:61 by Mariusz Felisiak <felisiak.mariusz@…>, 4 years ago

Resolution: fixed
Status: assignedclosed

In 6789ded0:

Fixed #12990, Refs #27694 -- Added JSONField model field.

Thanks to Adam Johnson, Carlton Gibson, Mariusz Felisiak, and Raphael
Michel for mentoring this Google Summer of Code 2019 project and
everyone else who helped with the patch.

Special thanks to Mads Jensen, Nick Pope, and Simon Charette for
extensive reviews.

Co-authored-by: Mariusz Felisiak <felisiak.mariusz@…>

comment:62 by GitHub <noreply@…>, 4 years ago

In f59a2b73:

Refs #12990 -- Added DatabaseFeatures.has_json_operators.

CockroachDB also has them.

comment:63 by Mariusz Felisiak <felisiak.mariusz@…>, 4 years ago

In 5480fab2:

[3.1.x] Refs #12990 -- Added DatabaseFeatures.has_json_operators.

CockroachDB also has them.
Backport of f59a2b730685fc62c5cb44101f54faf8921d9bc7 from master

comment:64 by Mariusz Felisiak <felisiak.mariusz@…>, 4 years ago

In 5d4b9c1c:

Refs #12990 -- Added example to JSONField release notes.

comment:65 by Mariusz Felisiak <felisiak.mariusz@…>, 4 years ago

In 7598ba01:

[3.1.x] Refs #12990 -- Added example to JSONField release notes.

Backport of 5d4b9c1cab03f0d057f0c7751862df0302c65cf9 from master

comment:66 by Mariusz Felisiak <felisiak.mariusz@…>, 3 years ago

In 7cb5712:

Refs #12990 -- Removed django.contrib.postgres.fields.JSONField per deprecation timeline.

comment:67 by Mariusz Felisiak <felisiak.mariusz@…>, 3 years ago

In 8fdb5a65:

Refs #12990 -- Removed django.contrib.postgres.fields.jsonb.KeyTransform/KeyTextTransform.

Per deprecation timeline.

comment:68 by Mariusz Felisiak <felisiak.mariusz@…>, 3 years ago

In 2dd6a83:

Refs #12990 -- Removed django.contrib.postgres.forms.JSONField per deprecation timeline.

Note: See TracTickets for help on using tickets.
Back to Top