Changes between Version 5 and Version 6 of AppEngine

Timestamp:

Feb 12, 2009, 4:05:41 AM (15 years ago)

Author:

Waldemar Kornewald

Comment:

m2m improvements

AppEngine

@@ -63 +63 @@
 Datastore writes are very expensive. App Engine provides batch operations for saving and deleting lots of model instances at once (no more than 500 entries, though). Django should provide such an API, too, so code can be optimized.
 
-The API would be most flexible if it worked like a transaction handler where all save() calls within a function call are collected and then committed afterwards. The implementation wouldn't be trivial, though. It requires maintaining a cache of to-be-saved instances, so filter() calls can check the cache.
+The API would be most flexible if it worked like a transaction handler where all save() calls within a function call are collected and then committed afterwards. The implementation wouldn't be trivial, though. It requires maintaining a cache of to-be-saved instances, so filter() calls can check the cache. Also, when a real transaction starts the cache must be flushed and disabled because in transactions we have to interact with the DB directly in order to lock an entity group.
 
 There are batch operations for getting lots of model instances by key. This could be emulated with
@@ -77 +77 @@
 For efficiency it should be possible to retrieve only the pk values of a !ForeignKey or !ManyToManyField without loading the actual entities from the db.
 
-Many-to-many relations could be emulated with a !ListProperty(db.Key), so you can at least issue simple queries, but this can quickly hit the 5000 index entries limit. The alternative of having an intermediate table is useless if you have to issue queries on the data and due to the query limit you wouldn't be able to retrieve more than 1000 related entities, anyway.
+Many-to-many relations could be emulated with a !ListProperty(db.Key), so you can at least issue simple queries, but this can quickly hit the 5000 index entries limit. The alternative of having an intermediate table is useless if you have to issue queries on the data and due to the query limit you wouldn't be able to retrieve more than 1000 related entities, anyway (well, that could be worked around with __key__-based sorting, but then you have to create an index and you might hit CPU limits if you check a lot of data in one request).
 
-The problem with many-to-many relations is that, for example, !ModelForm saves the model instance and and its many-to-many relations in separate steps. With !ListProperty this would cause multiple write operations. Also, depending on where the many-to-many relation is defined the changes could affect multiple models at once. One solution is to use batch operations as described above, but this means that all existing many-to-many code has to be optimized. Since this should be transaction-safe the field would have to be defined on a specific model, so that only one entity is affected when adding multiple relations. This means that Django has to make it easy to add new fields to existing models.
+The problem with many-to-many relations is that, for example, !ModelForm saves the model instance and and its many-to-many relations in separate steps. With !ListProperty this would cause multiple write operations. Also, depending on where the many-to-many relation is defined the changes could affect multiple models at once. One solution is to use batch operations as described above, but this means that all existing many-to-many code has to be changed to use batch operations. An alternative is to change !ModelForm and all other many-to-many code to allow for setting the !ListProperty before save() is called. Since this should be transaction-safe the field would have to be defined on a specific model, so that only one entity is affected when adding multiple relations. This means that Django has to make it easy to add new fields to existing models (i.e., add a !ManyToManyField to model B, but store the data in the target model A) and it must have knowledge of the storage location of the many-to-many relations since we might not have an intermediate table.
 
 == Special field types ==