Changes between Version 2 and Version 3 of AppEngine


Ignore:
Timestamp:
02/09/09 02:46:43 (6 years ago)
Author:
wkornewald
Comment:

many-to-many relations need batch operations

Legend:

Unmodified
Added
Removed
Modified
  • AppEngine

    v2 v3  
    5757== Transactions ==
    5858
    59 Django could emulate transactions with the commit_on_success decorator. Manual transaction handling and checkpoints can't be implemented on App Engine, though.
     59Django could emulate transactions with the commit_on_success decorator. Manual transaction handling and checkpoints can't be implemented with App Engine's current API, though. We might ask Google for help. The problem with commit_on_success is that it should run only once, but App Engine's run_in_transaction runs up to three times if an error occurs. The worst that can happen is that someone uses a custom decorator which calls commit_on_success multiple times because this could quickly hit a request limit. Maybe Django should officially change commit_on_success to issue retries?
    6060
    6161== Datastore batch operations ==
     
    6363Datastore writes are very expensive. App Engine provides batch operations for saving and deleting lots of model instances at once (no more than 500 entries, though). Django should provide such an API, too, so code can be optimized.
    6464
    65 The API would be most flexible if it worked like a transaction handler where all save() calls within a function call are collected and then committed afterwards. The implementation wouldn't be trivial, though. It requires maintaining a list of (pre-collected) saved instances, so filter() calls also check the pre-collected list.
     65The API would be most flexible if it worked like a transaction handler where all save() calls within a function call are collected and then committed afterwards. The implementation wouldn't be trivial, though. It requires maintaining a cache of to-be-saved instances, so filter() calls can check the cache.
    6666
    6767There are batch operations for getting lots of model instances by key. This could be emulated with
     
    7575Since JOINs don't work, Django should fall back to client-side JOIN emulation by issuing multiple queries. Of course, this only works with small datasets and it's inefficient, but that can be documented. It can still be a useful feature.
    7676
    77 Many-to-many relations could be emulated with something like a !ListProperty(db.Key), so you can at least issue simple queries, but this can quickly hit the 5000 index entries limit. The alternative of having an intermediate table is useless if you have to issue queries on the data. Anyway, for efficiency it should be possible to retrieve only the pk values without loading the actual entities from the db.
     77For efficiency it should be possible to retrieve only the pk values of a !ForeignKey or !ManyToManyField without loading the actual entities from the db.
     78
     79Many-to-many relations could be emulated with a !ListProperty(db.Key), so you can at least issue simple queries, but this can quickly hit the 5000 index entries limit. The alternative of having an intermediate table is useless if you have to issue queries on the data and due to the query limit you wouldn't be able to retrieve more than 1000 related entities, anyway.
     80
     81The problem with many-to-many relations is that, for example, ModelForm saves the model instance and and its many-to-many relations in separate steps. With !ListProperty this would cause multiple write operations. One solution is to use batch operations as described above, but this at least means that most existing code won't work.
    7882
    7983== Special field types ==
Back to Top