This wiki page collects design principles and best-practice which would be useful for an App Engine port of Django.

The DB backend itself and other App Engine code will be kept as a separate project/repository: ​http://bitbucket.org/gumptioncom/django-db-app-engine/

Related tickets: #10192, #10355

Parent wiki page: NonSqlBackends

Porting Django to App Engine: What's needed/different?

In order to understand this proposal you must have read the ​App Engine documentation. In order to simplify the port it might be possible to reuse a few parts from ​app-engine-patch.

The following might also apply to other cloud hosts which provide special database and communication interfaces.

Summary

At the Django level we need support for:

• setting an owner model for ManyToManyFields (i.e., there is no intermediary table and the field's data could be stored on either the model defining the ManyToManyField or on the other model) and ModelForm should handle that case efficiently (i.e., try to save the relations together with the model instead of afterwards in save_m2m)
• ListField (stores a list of values of a certain type; DB backends could use this internally for ManyToManyFields without an intermediary table) and ​BinaryField
• batch save() and delete() on instances
• email backends
• running Django from a zip package
• Permission and ContentType should be fake models which are stored as a string in a CharField

Indexes

Queries with inequality filters or sort orders need special index rules, so Django features like the admin interface should have a fall-back mode in which you can't sort query results because the developer can hardly define all possible index rules, especially if the searched property is a list property (in which case you need multiple index rules for all possible numbers of search terms).

Possibly, when App Engine gets full-text search support there could be a fall-back to (or preference for?) running complex queries on the full-text index.

Keys, key_name, key id, parents

In order to stay compatible with normal Django code the id should be emulated with an AutoField and the key_name with a CharField(primary_key=True).

Parents are a special App Engine feature which can't be mapped transparently to Django features, so they could be simulated in two different ways:

• With a separate gae_parent field that is automatically added to every model or (if this isn't possible: that must be added manually). The pk itself only contains the id or key_name (without the prefix) and thus stays clean and doesn't have to be encoded in a special format. A special function allows for constructing parent paths: make_parent_path(ModelA, 23, ModelB, 'kname'). This function returns a human-readable string (e.g.: "ModelA|23|ModelB|kname") representing the path
• With a special GAEKeyField(primary_key=True) that holds an App Engine db.Key object or its str() value.

Some applications might mix the use of ids and key_names within the same model (does this only affect existing GAE apps?). This could be emulated cleanly with a GAEKeyField, too. Of course, you lose portability in this case.

In order to simplify porting existing applications we could also add transparent prefixing of key names (since they ​originally couldn't start with a digit). This could be configurable per model, but it shouldn't be part of the actual model definition because it's only useful for old applications.

TODO: Queries should somehow support ancestor conditions.

Transactions

App Engine supports running a whole function in a transaction via db.run_in_transaction(func, args...). Manual transaction handling and checkpoints can't be implemented with App Engine's current API. We might ask Google for help.

Django could emulate App Engine transactions with the commit_on_success decorator, but it doesn't really do exactly the same thing. App Engine's transactions are much more like a "SELECT FOR UPDATE" (which isn't supported by Django, yet). We might need a new transaction decorator for this.

Datastore batch operations

Certain batch operations can already be emulated with the existing API:

Getting lots of model instances by key:

Blog.objects.all().filter(pk__in=[key1, key2, ...])

Deleting a set of rows by key:

Blog.objects.all().filter(pk__in=[key1, key2, ...]).delete()

Changing existing rows:

Blog.objects.all().filter(pk__in=[key1, key2, ...]).update(title='My Blog')

What we also need is a way to batch-create/update model instances. Also, batch-deletes should work with model instances and reuse the model instances when sending signals (currently, they're re-fetched which is inefficient).

Model relations and JOINs

Since JOINs don't work, Django should fall back to client-side JOIN emulation by issuing multiple queries. Of course, this only works with small datasets and it's inefficient, but that can be documented. It can still be a useful feature.

Many-to-many relations could be emulated with a ListProperty(db.Key), so you can at least issue simple queries, but this can quickly hit the 5000 index entries limit. The alternative of having an intermediate table is useless if you have to issue queries on the data and due to the query limit you wouldn't be able to retrieve more than 1000 related entities, anyway (well, that could be worked around with key-based sorting, but then you have to create an index and you might hit CPU limits if you check a lot of data in one request).

The problem with many-to-many relations is that, for example, ModelForm saves the model instance and and its many-to-many relations in separate steps. With ListProperty this would cause multiple write operations. Also, depending on where the many-to-many relation is defined the changes could affect multiple models at once. One solution is to use batch operations as described above, but this means that all existing many-to-many code has to be changed to use batch operations. An alternative is to change ModelForm and all other many-to-many code to allow for setting the ListProperty before save() is called.

Since this should be transaction-safe the field would have to be defined on a specific model, so that only one entity is affected when adding multiple relations. This means that Django has to make it easy to add new fields to existing models (i.e., add a ManyToManyField to model B, but store the data in the target model A) and it must have knowledge of the storage location of the many-to-many relations since we might not have an intermediate table.

Special field types

The following field types have to be ported to Django:

• ListProperty
• BlobProperty (see ​BinaryField)

Zipimport

Django should work from within a zip package. This means at least extending find_commands(), so manage.py commands can work (app-engine-patch ​already does this). The media files and templates could be exported from the zip file (like it's currently done in app-engine-patch) if that is more efficient.

manage.py commands

Not all manage.py commands should be available on App Engine (e.g., the SQL-related commands). This could probably be detected at runtime based on the DB backend's capabilities. Some commands like "runserver" have to be replaced. This could possibly be done by adding an app to INSTALLED_APPS which redefines a few commands.

We also need an "official" deployment command to emulate "appcfg.py update" and similar commands for other cloud hosts.

Email support

In order to support email functionality it must be possible to provide email backends which handle the actual sending process. App Engine has a special ​Mail API. See #10355

File uploads

The file upload handling code should never assume that it has access to the file system. Instead, it should be supported that the file gets uploaded indirectly into the datastore (e.g., via POST to S3 and then Django just gets notified when the upload is finished). This means that imports of file system functions should be deferred as much as possible.

Permissions and content types

In order to prevent many inefficient JOINs the Permission and ContentType models should be replaced with dynamically generated fake model instances. Since we can retrieve the list of defined models at runtime we can easily generate all instances for those two models at runtime, too. Internally, they could be stored as a simple string (e.g., 'user.can_add') and converted into fake models when the field is accessed. This might require creating a FakeModelField for holding this kind of model. Alternatively, the backend itself could take care of efficiently faking those two models at a lower level (including JOIN emulation).