Changes between Version 5 and Version 6 of AppEngine


Ignore:
Timestamp:
02/12/2009 04:05:41 AM (7 years ago)
Author:
wkornewald
Comment:

m2m improvements

Legend:

Unmodified
Added
Removed
Modified
  • AppEngine

    v5 v6  
    6363Datastore writes are very expensive. App Engine provides batch operations for saving and deleting lots of model instances at once (no more than 500 entries, though). Django should provide such an API, too, so code can be optimized.
    6464
    65 The API would be most flexible if it worked like a transaction handler where all save() calls within a function call are collected and then committed afterwards. The implementation wouldn't be trivial, though. It requires maintaining a cache of to-be-saved instances, so filter() calls can check the cache.
     65The API would be most flexible if it worked like a transaction handler where all save() calls within a function call are collected and then committed afterwards. The implementation wouldn't be trivial, though. It requires maintaining a cache of to-be-saved instances, so filter() calls can check the cache. Also, when a real transaction starts the cache must be flushed and disabled because in transactions we have to interact with the DB directly in order to lock an entity group.
    6666
    6767There are batch operations for getting lots of model instances by key. This could be emulated with
     
    7777For efficiency it should be possible to retrieve only the pk values of a !ForeignKey or !ManyToManyField without loading the actual entities from the db.
    7878
    79 Many-to-many relations could be emulated with a !ListProperty(db.Key), so you can at least issue simple queries, but this can quickly hit the 5000 index entries limit. The alternative of having an intermediate table is useless if you have to issue queries on the data and due to the query limit you wouldn't be able to retrieve more than 1000 related entities, anyway.
     79Many-to-many relations could be emulated with a !ListProperty(db.Key), so you can at least issue simple queries, but this can quickly hit the 5000 index entries limit. The alternative of having an intermediate table is useless if you have to issue queries on the data and due to the query limit you wouldn't be able to retrieve more than 1000 related entities, anyway (well, that could be worked around with __key__-based sorting, but then you have to create an index and you might hit CPU limits if you check a lot of data in one request).
    8080
    81 The problem with many-to-many relations is that, for example, !ModelForm saves the model instance and and its many-to-many relations in separate steps. With !ListProperty this would cause multiple write operations. Also, depending on where the many-to-many relation is defined the changes could affect multiple models at once. One solution is to use batch operations as described above, but this means that all existing many-to-many code has to be optimized. Since this should be transaction-safe the field would have to be defined on a specific model, so that only one entity is affected when adding multiple relations. This means that Django has to make it easy to add new fields to existing models.
     81The problem with many-to-many relations is that, for example, !ModelForm saves the model instance and and its many-to-many relations in separate steps. With !ListProperty this would cause multiple write operations. Also, depending on where the many-to-many relation is defined the changes could affect multiple models at once. One solution is to use batch operations as described above, but this means that all existing many-to-many code has to be changed to use batch operations. An alternative is to change !ModelForm and all other many-to-many code to allow for setting the !ListProperty before save() is called. Since this should be transaction-safe the field would have to be defined on a specific model, so that only one entity is affected when adding multiple relations. This means that Django has to make it easy to add new fields to existing models (i.e., add a !ManyToManyField to model B, but store the data in the target model A) and it must have knowledge of the storage location of the many-to-many relations since we might not have an intermediate table.
    8282
    8383== Special field types ==
Back to Top