Code

Opened 4 years ago

Closed 2 years ago

Last modified 9 months ago

#12990 closed New feature (wontfix)

New Field Type: JSONField

Reported by: paltman Owned by: nobody
Component: Database layer (models, ORM) Version: 1.2-alpha
Severity: Normal Keywords:
Cc: paltman@…, self.anderson@…, t.django@…, slav0nic0@…, NicoEchaniz, dan.fairs@…, hv@…, ShawnMilo, bronger Triage Stage: Design decision needed
Has patch: yes Needs documentation: no
Needs tests: yes Patch needs improvement: yes
Easy pickings: no UI/UX: no

Description

I have been using this for over a year now and find it quite useful. I posted it about it the other day (http://paltman.com/2010/feb/25/how-to-store-arbitrary-data-in-a-django-model/) and then thought it might be generally useful as an out of the box field type so I thought I'd submit a patch in case it is deemed "generally useful" enough to be considered for inclusion in django proper.

Attachments (1)

json_field.diff (4.8 KB) - added by paltman 4 years ago.
JSON Field Patch

Download all attachments as: .zip

Change History (47)

Changed 4 years ago by paltman

JSON Field Patch

comment:1 Changed 4 years ago by Alex

  • milestone 1.2 deleted
  • Needs documentation unset
  • Needs tests unset
  • Patch needs improvement unset
  • Triage Stage changed from Unreviewed to Design decision needed

I'm -1 on this, I think it's out of scope for Django core, and easily accomplished outside. Regardless I'm removing the 1.2 milestone, 1.2 is in feature freeze.

comment:2 Changed 4 years ago by floguy

+1 on this. There are many implementations of this floating around, and most of them are wrong. It's a common-enough pattern that Django should provide the canonical implementation.

comment:3 follow-up: Changed 4 years ago by marcelor

+1 If XMLField is in the core then JSONField deserves to be in too. Also I'm tempted to propose the removal of XMLField, you all know that
the use of XML as a format to store arbitrary data has dramatically decreased in favour of JSON.

comment:4 in reply to: ↑ 3 ; follow-up: Changed 4 years ago by Alex

Replying to floguy:

+1 on this. There are many implementations of this floating around, and most of them are wrong. It's a common-enough pattern that Django should provide the canonical implementation.

How are most of them wrong? It's pretty trivial (there's actually an implementation in the tests).

Replying to marcelor:

+1 If XMLField is in the core then JSONField deserves to be in too. Also I'm tempted to propose the removal of XMLField, you all know that
the use of XML as a format to store arbitrary data has dramatically decreased in favour of JSON.

XMLField at this point does almost nothing (doesn't even do validation at this point).

comment:5 Changed 4 years ago by paltman

  • Cc paltman@… added

comment:7 Changed 4 years ago by Alex

The fact that people don't know how to organize code externally cannot act as an argument in favor of including everything in django's core.

comment:8 Changed 4 years ago by floguy

The proliferation of implementations proves that there's significant interest in the feature. Since each of the implementations are slightly different, it makes it difficult to rely on the behavior of any one implementation. Also, some of the implementations behave incorrectly when using dumpdata/loaddata. Including it in core would solve all of these problems, as well as provide much lower friction for a new user who does not know which implementation to choose.

comment:9 Changed 4 years ago by peterbe

+1 on this. I'm not sure what it means in terms of exports/dumps/fixtures etc.

@Alex: the reason we do Django is very much because it's a convenient frameworks that helps getting things done quickly.

comment:10 Changed 4 years ago by anonymous

  • Cc self.anderson@… added

comment:11 Changed 4 years ago by anonymous

  • Cc t.django@… added

comment:12 Changed 4 years ago by slav0nic

  • Cc slav0nic0@… added

comment:13 Changed 4 years ago by rubic

+1 me too, vote. I'd really like to see this added to Django. A single implementation of JSONField would be an improvement over the current state.

comment:14 Changed 3 years ago by NicoEchaniz

  • Cc NicoEchaniz added

comment:15 Changed 3 years ago by lrekucki

  • Needs tests set
  • Patch needs improvement set

The patch has some tests, but their coverage is very limited. Some things that need testing:

  • get_*_json/set_*_json hooks
  • does this work properly with dumpdata/loaddata in all formats ?

Having an option to override the encoder/decoder without having to subclass the field would be nice (like in some other implementations).

A major argument for me against having this in core, is that you can't include South support (at least atm, we'll see what happens in 1.4), so I would still prefer a 3rd party implementation that does.

Last edited 3 years ago by lrekucki (previous) (diff)

comment:16 Changed 3 years ago by danfairs

  • Cc dan.fairs@… added

comment:17 Changed 3 years ago by lukeplant

  • Type set to New feature

comment:18 Changed 3 years ago by lukeplant

  • Severity set to Normal

comment:19 Changed 3 years ago by joshg

I just wanted to say, I would like this to be built into the core. Thanks!

comment:20 Changed 3 years ago by mark0978

  • Easy pickings unset

+1 Considering all the versions of this floating out there and all the patches due to them not being quite right. Putting this is to Django seems to be the right way to do it.

comment:21 follow-up: Changed 3 years ago by russellm

For the record - I'm -0 on this. For my money, if you're putting JSON in a database field (and expecting it to be searchable etc), you're doing it wrong. I won't stand in the way if someone else in the core team feels particularly enthused, but I'm not rushing myself.

Also, the previous argument about XMLField has been rendered invalid -- we removed XMLField in 1.3.

comment:22 Changed 3 years ago by jezdez

Agreed, I think projects like https://github.com/jordanm/django-hstore show that there are db specific fields that are better suited for storing JSON.

comment:23 Changed 3 years ago by sorl

I think the usecase for something like this should be simply serialized data in a field, not searchable and not just a mapping to a dict. Given that, we might as well use pickle, but then we might loose data portability. If you want to build an efficient relational db centric application you probably want to use something like this and I think many people realize that but the many implementations floating around makes it hard for them to decide which one to use. If Django wants to advocate for that kind of database design then perhaps it is a good idea to include a serialization field. I have a very simple lazier approach, it can't serialize strings but it only deserializes on attr access, https://github.com/aino/django-cerial/blob/master/cerial/fields.py

comment:24 Changed 3 years ago by guettli

  • Cc hv@… added

comment:25 Changed 3 years ago by ShawnMilo

  • Cc ShawnMilo added

I'd like to see this as well, because we use a JSON field frequently. I think the argument against that searching JSON in a queryset is poor design is a straw-man argument. I haven't heard anyone ask for a JSON field with this in mind. It's just convenient to store serialized data to avoid unnecessary auxiliary storage and proliferation of extra fields and models.

Having a canonical JSON field is better than people implementing their own, in my opinion, because it makes it provides consistency across projects without needing to copy/paste code into each new project.

Last edited 3 years ago by ShawnMilo (previous) (diff)

comment:26 in reply to: ↑ 4 Changed 3 years ago by Fredde

Replying to Alex:

How are most of them wrong? It's pretty trivial (there's actually an implementation in the tests).

The JSONField included in the in the django tests ([source:django/trunk/tests/modeltests/field_subclassing/fields.py#L56]) is actually not working properly eather. For example, when trying to store a empty list [] or dict {} the JSONField.to_python will convert it to None:

#python
>>> class A(models.Model):
	json_field = JSONField()

>>> a = A()
>>> a.json_field = []
>>> print a.json_fied
None

I don't know how to fix this. From the documentation the following values should be handled by the to_python method:

  • An instance of the correct type (e.g., Hand in our ongoing example).
  • A string (e.g., from a deserializer).
  • Whatever the database returns for the column type you're using.

Hovever, there is no way to know if the value is a "correct type" a serialized representation or the value from the database. For example value = '[1, 2, 3]'. It could eather come from an attribute assignment, or from the database, but it should only be deserialized if it's from the database.

comment:27 Changed 3 years ago by wizpig64

  • Cc wizpig64@… added
  • UI/UX unset

comment:28 in reply to: ↑ 21 Changed 3 years ago by anonymous

Replying to russellm:

For the record - I'm -0 on this. For my money, if you're putting JSON in a database field (and expecting it to be searchable etc), you're doing it wrong.

You cannot say that generally. The answer is as always, it depends. There are perfectly valid uses of this, for example, storing a little (dynamic) metadata without setting up a NoSQL database.

comment:29 Changed 3 years ago by russellm

If you're arguing for JSONField as a way of avoiding setting up a NoSQL store, then you've just proved my point. If you need a non-relational store, *use a non-relational store*. Yes, there may be reasons why this isn't possible for political or business reasons, but that doesn't change the fact that:

  • There is a better way to solve the problem
  • Django shouldn't be encouraging people to hack around the root cause of problems
  • The absence of a JSONField in Django core doesn't prevent the community from developing a canonical implementation that can be used as a third-party library.
  • Putting JSONField in core wouldn't affect the stability or reliability of that third party implementation -- it just means the core team are the only people who can maintain it.

comment:30 follow-up: Changed 3 years ago by guettli

By setting up a NoSQL store you loose the transactional atomic behavior: commit all or nothing. AFAIK two-phase commit is not supported by django. I think a JSONField inside django would be good. I don't want to query the JSON value. I just use JSONField like cache that never expires.

comment:31 in reply to: ↑ 30 Changed 3 years ago by Lee S

I would love to see a canonical version of JSON fields included.

I routinely find storing non-structured, non-searchable data in the database to be extremely useful in many use cases. Setting up a NoSQL database is often overkill. The real need is to store the odd JSON field in an otherwise traditional SQL model. The example cited above, storing flexible metadata is a common one -- for example, key/value pairs indicating user preferences.

I'm at the point where I have to try one of the many implementations listed above and cross my fingers that it's not broken.

comment:32 Changed 3 years ago by aaugustin

I've recently rolled my own implementation of this.

I use it in a logging table where I want to store an event_kind (ex : "USER_LOGIN") + a datetime + a bunch of parameters (ex : username, remote IP) that depend on the kind of event. I want to be able to add arbitrary parameters at any point. Normalizing parameters in a separate table (e.g. storing event_id, key, value in another table for each parameter) seems overkill — I don't want to add a model just for this.

So I think there are some real-life use cases for storing a JSON-serialized dict in a database field. Since it's moderately difficult to get right, +0 for adding it in Django.

comment:33 Changed 3 years ago by radbrad182+djangoproject@…

JSON fields are also a great way to store basic, dynamic, typed data. Being able to store Key Value data in the DB would be useful:

class Key(models.Model):
   user = models.ForeignKey(User)
   name = models.CharField()
   value = models.JSONField()

user.key_set.create(name="an_integer", value=4)
user.key_set.create(name="a_list", value=['a','b','c'])

Normally to get this kind of basic data typing you would need to store another field for value_type, and then parse manually.

Arguing that this should be stored in a K/V store is not ideal as it adds dependency on another datastore type, and you lose out on user.key_set.get(name='something').

comment:34 Changed 2 years ago by bronger

  • Cc bronger added

comment:35 Changed 2 years ago by riccardodivirgilio

  • Version changed from 1.2-beta to 1.2-alpha

this is my proposition to for the field.

class JsonField(models.Field): 
    __metaclass__ = models.SubfieldBase 
    serialize_to_string = True 
    def get_internal_type(self): 
        return "TextField" 
    def value_to_string(self, obj): 
        return self.get_prep_value(self._get_val_from_obj(obj)) 
    def get_prep_value(self, value): 
        if value: 
            stream = StringIO.StringIO() 
            simplejson.dump(value, stream, cls=DjangoJSONEncoder) 
            value = stream.getvalue() 
            stream.close() 
            return value 
        return None 
    def to_python(self, value): 
        if isinstance(value, (str, unicode)): 
            value = StringIO.StringIO(value) 
            return simplejson.load(value) 
        return value 

maybe there is no need to to create a class
JSONDateEncoder(json.JSONEncoder) because there is DjangoJSONEncoder
already.

then it would be great to implement a lazy translation object for
DjangoJSONEncoder to store ugettext_lazy objects.

for me there is no need to create set_%s_json and get_%s_json method
because the field should handle it directly, and we should use the
to_python method, like all other fields in django.

for example the datetime field directly push a datetime object to the
model (using to_python) and transform it to a string when save
method is called

we should use the same logic here and encode a json object to string
only when we call the save method, no need for a get_%s_json in the
field api.

comment:36 Changed 2 years ago by davidfstr

  • Cc davidfstr@… added

comment:37 Changed 2 years ago by anonymous

What's the best third-party alternative out there? It seems like this is going to drag on for a while in a religious discussion of whether it should be included or not. Just add it and let people decide whether and how to use it.

comment:38 Changed 2 years ago by aaugustin

django-extensions is a fairly popular project that provides an implementation of JSONField.

comment:39 Changed 2 years ago by aaugustin

  • Resolution set to wontfix
  • Status changed from new to closed

This idea was discussed and vetoed by one of our BDFLs on #django-social. Here are the relevant logs, edited for readability.

2012-01-20 21:18:10 <jacobkm> (...) we've explicitly decided *not* to include (...) JSONField

2012-01-20 21:21:58 <jacobkm> (...) I'm not saying there's no good reason for it, I'm saying the bad reasons outweigh the good and that's why it shouldn't ship with Django.

2012-01-20 21:22:11 <jacobkm> Under the Don't Give Users Loaded Guns Aimed At Feet principle.

comment:40 Changed 15 months ago by gabewb@…

It is pretty frustrating to google around for this, finally land here, and see that it's been vetoed by the principle "Don't Give Users Loaded Guns Aimed At Feet." That is weak principle. When it prevents people from using guns in dangerous ways, great, fine. But sometimes people really need to use a gun pointed in the downward direction, and if you don't give it to them, they do exactly what they did --- they make their own guns, and those guns all have their own sights, triggers, and safety mechanisms, and many of them don't work quite right or are liable to jam or backfire, and a newbie has neither the time nor the ability to pick the right one.

In this case, it's a lot better to include JSONField with a warning label than to keep it out entirely.

Incidentally, I found my way here because JSONField was the default custom field listed in https://docs.djangoproject.com/en/dev/ref/models/fields/

comment:41 Changed 15 months ago by lukeplant

The WONTFIX will remain unless you bring this up on Django-devs.

However, one further reason why it is a bad idea is the existence of hstore in Postgres.

This is a database specific solution, and is a much better solution than storing JSON in a text field for many applications. Providing a built-in JSONField discourages people from finding better solutions.

Also Googling "django JSONField" gives you several solutions that are easily installable using PyPI, and work fine for most people, so I don't see how it is an issue. It's extremely easy to get hold of a working JSONField if you want one.

comment:42 Changed 15 months ago by akaariai

For the record I think there are many valid use cases for storing JSON in the database. Also, PostgreSQL 9.2 already has JSON typed field, 9.3 might make it nicely searchable. If the JSON improvements get into 9.3 then it would actually be a better solution than HStore.

But, we likely don't want this field in core. If we are restricted to the functionality offered by all databases, then it basically boils down to "convert to text on save, convert to python on load". Which isn't all that great.

My vote is on improving core support for custom fields. If one can write a json field that just works outside of core, then "pip install django-jsonfield" seems like a good way to add this feature...

comment:43 Changed 12 months ago by anonymous

How disappointing. In my case, I want to store user-specific settings provided by the UI guys which are used purely for layout, styles, etc... I don't want to have to maintain normalised tables for a constantly-updating data structure which is only ever stored/retrieved as a whole.

I keep finding this in Django (and Python in general) - great ideas get vetoed because "we don't dare trust a developer to use tools properly".

comment:44 Changed 12 months ago by davidfstr

  • Cc davidfstr@… removed

comment:45 Changed 12 months ago by wizpig64

  • Cc wizpig64@… removed

comment:46 Changed 9 months ago by tom.gruner@…

If you "Don't Give Users Loaded Guns Aimed At Feet ", how do you ever expect them to dance ( ;

Add Comment

Modify Ticket

Change Properties
<Author field>
Action
as closed
as The resolution will be set. Next status will be 'closed'
The resolution will be deleted. Next status will be 'new'
Author


E-mail address and user name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.