Code

Changes between Version 16 and Version 17 of AppEngine


Ignore:
Timestamp:
03/19/09 09:06:02 (5 years ago)
Author:
wkornewald
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • AppEngine

    v16 v17  
    1212 
    1313At the Django level we need support for: 
    14  * setting an owner model for !ManyToManyFields (i.e., there is no intermediary table and the field's data could be stored on either the model defining the relation or on the other model) and !ModelForm should handle that case efficiently (i.e., try to save the relations together with the model instead of afterwards in save_m2m) 
     14 * setting an owner model for !ManyToManyFields (i.e., there is no intermediary table and the field's data could be stored on either the model defining the !ManyToManyField or on the other model) and !ModelForm should handle that case efficiently (i.e., try to save the relations together with the model instead of afterwards in save_m2m) 
    1515 * !ListField (stores a list of values of a certain type; DB backends could use this internally for !ManyToManyFields without an intermediary table) and [http://code.djangoproject.com/ticket/2417 BinaryField] 
    1616 * batch save() and delete() 
     
    1919 * Permission and !ContentType should be fake models which are stored as a string in a !CharField 
    2020 
    21 == Reminder: Datastore and request limitations == 
    22  
    23 You can't have requests that take longer than 10 seconds and you can't retrieve more than 1000 model instances at once from the datastore. It's also impossible to run more than 30 queries without hitting the 10 sec request limit. 
    24  
    25 A single entity (actually: a whole entity group) can't handle more than 5 writes per second (writes = save or delete). 
    26  
    27 Unique properties can only be emulated via the key_name, but this means their values can't be changed afterwards, so we might have to fall back to non-guaranteed uniqueness. Since you can't issue queries from within a transaction we have the problem that we can't even do a simple check in all cases. Probably we can only rely on checking on the !ModelForm level, then. Alternatively, we have to document that you can't use transactions on models that have unique properties apart from the !PrimaryKey (which can be emulated with the key_name and thus gives us a 100% uniqueness guarantee because it can be used in a transaction). 
    28  
    29 An entity may not have more than 5000 index entries. 
    30  
    31 All query filter rules are connected via the AND operator. The OR operator is not supported. 
    32  
    33 Transactions can only run on a single entity group and you can't run queries within a transaction. 
    34  
    35 Also not supported: 
    36  * JOINs (could be done manually for small datasets) 
    37  * sub-queries (ditto) 
    38  * DISTINCT queries (i.e., no queryset.dates(), etc.) 
    39  * referential integrity 
    40  
    4121== Emulation of SQL features == 
    4222 
    4323If we emulate certain SQL features, it must be possible to detect when something gets emulated. This is important because you might mistakenly develop code that doesn't scale when your database grows beyond a certain size. In settings.py we could have a flag which specifies whether to throw warnings for emulated features (ENABLE_DB_EMULATION_WARNINGS?). By default, warnings should be enabled, so you never miss potential sources of problems. 
    4424 
     25Alternatively, one could have to activate emulation by calling a special function (Model.objects.with_emulation().filter(...)). That's more explicit and less error-prone. 
     26 
    4527== Schemas == 
    4628 
    47 Since tables are flexible and don't have schema definitions running "manage.py syncdb" shouldn't be necessary. 
     29Since tables are flexible and don't have schema definitions running "manage.py syncdb" shouldn't be necessary. It can still be supported by emulating a remote DB connection. 
    4830 
    4931== Indexes == 
     
    5537== Keys, key_name, key id, parents == 
    5638 
    57 Django should always assign a key_name to each newly created entity instead of letting App Engine choose a key id, so data can be exported from and imported into the datastore more easily and migrations to other providers become less problematic. A transaction can be used to ensure that no existing entity with the generated key_name gets overwritten. 
     39In general, the "pk" field should never be assumed to be a number. If an entity uses a key_name the application should continue to work. 
    5840 
    59 An interface to the underlying key and key id should be provided, too, but it shouldn't be recommended due to its problems. 
     41The key_name and parent could be emulated with a !CharField(primary_key=True) that automatically prefixes the given string with a character, internally. A special function could be provided for encoding a primary_key that consists of parents and a key_name or key id. This API would allow for reducing the key_name and parent into a single pk property and staying compatible with existing Django code which wouldn't work if we had separate pk and parent properties. 
    6042 
    61 The key_name and parent could be emulated with a !CharField(primary_key=True) that automatically prefixes the given string with a character, internally. If you only need a key_name it's sufficient to specify a string. If you want to also specify a parent you could create a special encoded string by passing the parent and (optionally) the desired key_name to that function and then passing the result to the !CharField. This API would allow for reducing the key_name and parent into a single pk property and staying compatible with existing Django code which wouldn't work if we had separate pk and parent properties. 
    62  
    63 The pk property should return an url-safe string that contains the key_name without the safety prefix (i.e., the value of the !CharField(primary_key=True)) and the parent pk. This is more portable than using the str(Key) because it doesn't contain the model and app name. Moreover, in case you specified a pk manually , the URLs will be much nicer. Even if you don't specify a key_name the URL is still shorter than the str(Key) version. 
     43The pk property should return an url-safe string that contains the key_name without the safety prefix (i.e., the value of the !CharField(primary_key=True)) and the parent pk. This is more portable than using the str(Key) because it doesn't contain the model and appid. Moreover, in case you specified a pk manually, the URLs will be much nicer. Even if you don't specify a key_name the URL is still shorter than the str(Key) version. 
    6444 
    6545Queries should support ancestor conditions. 
    6646 
    67 Every model should provide these properties: key, key_name, key_id, parent, parent_key 
     47Every model should provide properties for: key, key_name, key_id, parent, parent_key 
    6848 
    6949== Transactions == 
    7050 
    71 Django could emulate transactions with the commit_on_success decorator. Manual transaction handling and checkpoints can't be implemented with App Engine's current API, though. We might ask Google for help. The problem with commit_on_success is that it should run only once, but App Engine's run_in_transaction runs up to three times if an error occurs. The worst that can happen is that someone uses a custom decorator which calls commit_on_success multiple times because this could quickly hit a request limit. Maybe Django should officially change commit_on_success to issue retries? 
     51Django could emulate transactions with the commit_on_success decorator. Manual transaction handling and checkpoints can't be implemented with App Engine's current API, though. We might ask Google for help. The problem with commit_on_success is that it should run only once, but App Engine requires that it runs multiple times if an error occurs. The worst that can happen is that someone uses a custom decorator which calls commit_on_success multiple times because this could quickly hit a request limit. Maybe Django should officially change commit_on_success to issue retries? 
    7252 
    7353== Datastore batch operations ==