Provide a way to define a model without being registered into the app registry / Get rid of get_registered_model

[Mitar]

Provide a way to define a model without being registered into AppConfig. This is useful when you are creating dynamic models to be returned from querysets which have dynamic virtual fields (for example, created from data which can have arbitrary virtual fields stored). You do not want all those models to be stored in AppCache.

[Aymeric Augustin]

I know I'm insisting heavily, but can you formulate this feature request in terms of effects on the public API of the app registry, or any other public API of Django?

For example:

• "I would like app_config.get_model(xxx) to return None when I've created a model in this way."
• "I would like Django not to crash miserably with this stack trace when I've created a model in that way."

A request on the internal implementation of the app registry doesn't make sense because only public APIs are guaranteed to keep working as documented across versions. Why would you care that the model is or isn't referenced somewhere in the app registry, provided it's never returned or used by any API? In fact, if this feature gets implemented, it's likely that Django will register this model with some sort of "hidden" flag.

[Simon Charette]

I'm also curious to know why you'd need such a feature? There was no way of doing this with the legacy django.db.models.loading AFAIK.

Wouldn't making a django.apps.registry.Apps no-op subclass and specifying it as your DynamicModel.Meta.apps work?

class Void(django.apps.registry.Apps):
    def register_model(self, app_label, model):
        pass

void = Void()

Meta = type('Meta', (), {'apps': void})
MyDynamicModel = type('MyDynamicModel', (django.db.models.Model,), {'Meta': Meta})

[Aymeric Augustin]

Setting Model._meta.auto_created most likely does exactly what the OP wants, but since they didn't specify, I can't be sure.

​https://github.com/django/django/blob/98b52ae2/django/apps/registry.py#L222

[Aymeric Augustin]

Model._deferred might be even more suitable in that case, since it's what Django does for its own dynamic models.

Of course it would be better to provide a formal API.

[Aymeric Augustin]

Maybe the goal is to avoid leaking memory when creating a large number of such models?

[Mitar]

I am just copying from #21698, to make things more clear here. So my comment how I see that unregistering could help:

So there are two ways to allow making custom subclasses of models which should not be registered. One is to somehow allow some configuration in Meta (#21682), another is to call current metaclass which adds it, but be able to remove it from registry immediately afterwards (a bad side-effect is that registration signal has been already made).

I didn't know about "no-op subclass" approach. This seems interesting idea.

Maybe the goal is to avoid leaking memory when creating a large number of such models?

Yes. So what we are doing is developing support for dynamic schema. Users can add dynamically fields from the many hop away models. In a way we make proxy fields to things you would need multiple hops and we do one big join instead in the background to not have to fetch them again and again as you are hopping around. (Those hops can cross content type relations as well.) (Which mean you can change concrete implementation of a model, but the rest of your code can still work as it is referencing them through registered names.) We add those fields as virtual fields into base model and create a dummy proxy model to keep them, when we return a new queryset for them. So the syntax is something like:

queryset = models.Node.objects.regpoint('config').registry_fields(
    name='core.general#name',
    type='core.type#type',
    router_id='core.routerid#router_id',
    project='core.project#project.name',
).regpoint('monitoring').registry_fields(
    last_seen='core.general#last_seen',
    network_status='core.status#network',
    monitored_status='core.status#monitored',
    health_status='core.status#health',
    peers='network.routing.topology#link_count',
).order_by('type')

So, after you fetch those additional fields, returned queryset contains a dynamically created proxy model to models.Node, with name and others added as virtual fields. As this proxy model is useful only for this particular queryset, we do not want to register it globally. core.status is a registered name for a particular registry point, which can be implemented by any model which announces that it is implementing it. So a concrete related model can change (be extended by an app), but your queryset does not have to change.

All this is a bit involved and we do not yet have good documentation, but you can check ​this description and our ​current implantation of creating a dynamic proxy model and unregistered immediately afterwards is here. We do not want that such models pile up.

I know this is not public API, but it would still be good to have some clear way to: or not register a model when created, or be able to remove it from the registry. As you are planing on rewriting AppConfig, I just wanted to raise a point that some internal API (which I understand can change at any time) would be useful here.

[Aymeric Augustin]

So we're talking of hacking the internals heavily here ;-)

The implementation of Apps has become very straightforward if you ignore all the deprecated code. In the short term I'd just go for del apps.all_models[app_label][model_name] or del apps.get_app_config(app_label).models[model_name], followed by apps.clear_cache(). The first one is marginally faster, the second one is marginally less likely to change. Make sure model_name is lowercase.

Yesterday I tried to stop storing deferred models with the app registry. It would have solved your use case nicely -- what you're doing looks a lot like deferred models. Sadly it didn't work.

This request has larger ramifications. Currently ModelBase.new always returns a class registered in the app registry. It's quite a hack, and it's done for bad reasons that no longer exist since Django 1.4 (new project layout). I'm going to keep the ticket open with the following scope: get rid of get_registered_model, and then figure out what the registration API becomes.

[Mitar]

So we're talking of hacking the internals heavily here ;-)

Yes. :-) But from such hacking new use cases emerge. :-)

Currently ModelBase.new always returns a class registered in the app registry.

Yes. This is why we register, and then try to unregister. So maybe then #21698 is a better approach. The issue there is that you still send a signal when a new model is registered, despite then removing it almost immediately. Maybe an alternative would be that API could provide a way to register a model without sending a signal, and then to unregister. But this is just a hack for current approach with get_registered_model.

[Aymeric Augustin]

ModelBase.__new__ cannot return a class previously registered in the app registry any more.

get_registered_model is still used in ModelSignal.connect and in add_lazy_relation. In both cases it triggers different code paths depending on whether the target model was already created and its class_prepared signal was sent.

[Aymeric Augustin]

mitar, how should relations (foreign keys, one to one, many to many) between such a model and other models behave?

[Jernej Kos]

mitar, how should relations (foreign keys, one to one, many to many) between such a model and other models behave?

Do you see any specific problems here?

[Aymeric Augustin]

Yes. If you have multiple variants of a model with relations to another model, the reverse relations conflict on the other model.

[John Cronan]

Hi! This is a very old ticket, but I've tested this same approach in 3.2 and it still works. I'm doing a form-builder type of app, and I decided to use dynamic models.

I get the warning message about "reloading models is not advised as it can lead to inconsistencies." In my case, I've ensured that the dynamic models don't use any foreign keys (except between each other).

Considering the scope, "figure out what the registration API becomes": just as someone using this capability, I'd suggest that there could be a way to suppress the warning when you know that the side effects are not harmful. True unregistering, where the side effects with related models and such are removed automatically, would be cool but sounds hard.