Opened 12 months ago

Closed 12 months ago

Last modified 12 months ago

#34985 closed Bug (fixed)

Migrations raise AppRegistryNotReady when GeneratedField references incorrect fields.

Reported by: Paolo Melchiorre Owned by: Mariusz Felisiak
Component: Database layer (models, ORM) Version: 5.0
Severity: Release blocker Keywords: field, database, generated, output_field
Cc: Lily Foote Triage Stage: Accepted
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

Note: In my experiments, I am sure I have encountered similar errors even in cases that are simpler to reproduce, without models with multi-table inheritance to clarify, but in this case, I cannot find the example code of the field generated to reproduce it, so I report below the example that certainly allows you to reproduce the error.


I tried to test the behavior of the generated fields in the case of multi-table inheritance.

I was expecting an error since the documentation specifies that:

The expressions should be deterministic and only reference fields within the model (in the same database table).

The generation of migrations of two inheritance models occurs successfully (without generated fields)

from django.db import models

class Person(models.Model):
    first_name = models.CharField(max_length=150)
    last_name = models.CharField(max_length=150)

class Profile(Person):
    email = models.EmailField()
$ python3 -m manage makemigrations
Migrations for 'samples':
  samples/migrations/0010_person_profile.py
    - Create model Person
    - Create model Profile
$ python3 -m manage sqlmigrate samples 0010
BEGIN;
--
-- Create model Person
--
CREATE TABLE "samples_person" ("id" integer NOT NULL PRIMARY KEY AUTOINCREMENT, "first_name" varchar(150) NOT NULL, "last_name" varchar(150) NOT NULL);
--
-- Create model Profile
--
CREATE TABLE "samples_profile" ("person_ptr_id" bigint NOT NULL PRIMARY KEY REFERENCES "samples_person" ("id") DEFERRABLE INITIALLY DEFERRED, "email" varchar(254) NOT NULL);
COMMIT;

Instead, if in the second model, I create a generated field that references the fields in the first model I get a very unclear error that perhaps we could make more clear.

from django.db import models
from django.db.models import Value
from django.db.models.functions import (
    Concat,
)

class Person(models.Model):
    first_name = models.CharField(max_length=150)
    last_name = models.CharField(max_length=150)

class Profile(Person):
    email = models.EmailField()
    display_name = models.GeneratedField(
        expression=Concat("first_name", Value(" "), "last_name", Value(" "), "email",),
        db_persist=True,
        output_field=models.TextField(),
    )
$ python3 -m manage makemigrations
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/home/paulox/Projects/generatedfields/manage.py", line 22, in <module>
    main()
  File "/home/paulox/Projects/generatedfields/manage.py", line 18, in main
    execute_from_command_line(sys.argv)
  File "/home/paulox/.virtualenvs/generatedfields/lib/python3.12/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File "/home/paulox/.virtualenvs/generatedfields/lib/python3.12/site-packages/django/core/management/__init__.py", line 416, in execute
    django.setup()
  File "/home/paulox/.virtualenvs/generatedfields/lib/python3.12/site-packages/django/__init__.py", line 24, in setup
    apps.populate(settings.INSTALLED_APPS)
  File "/home/paulox/.virtualenvs/generatedfields/lib/python3.12/site-packages/django/apps/registry.py", line 116, in populate
    app_config.import_models()
  File "/home/paulox/.virtualenvs/generatedfields/lib/python3.12/site-packages/django/apps/config.py", line 269, in import_models
    self.models_module = import_module(models_module_name)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/importlib/__init__.py", line 90, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1381, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1354, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1325, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 929, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 994, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "/home/paulox/Projects/generatedfields/samples/models.py", line 233, in <module>
    class Profile(User):
  File "/home/paulox/.virtualenvs/generatedfields/lib/python3.12/site-packages/django/db/models/base.py", line 194, in __new__
    new_class.add_to_class(obj_name, obj)
  File "/home/paulox/.virtualenvs/generatedfields/lib/python3.12/site-packages/django/db/models/base.py", line 371, in add_to_class
    value.contribute_to_class(cls, name)
  File "/home/paulox/.virtualenvs/generatedfields/lib/python3.12/site-packages/django/db/models/fields/generated.py", line 51, in contribute_to_class
    self._resolved_expression = self.expression.resolve_expression(
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/paulox/.virtualenvs/generatedfields/lib/python3.12/site-packages/django/db/models/expressions.py", line 975, in resolve_expression
    c.source_expressions[pos] = arg.resolve_expression(
                                ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/paulox/.virtualenvs/generatedfields/lib/python3.12/site-packages/django/db/models/expressions.py", line 975, in resolve_expression
    c.source_expressions[pos] = arg.resolve_expression(
                                ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/paulox/.virtualenvs/generatedfields/lib/python3.12/site-packages/django/db/models/expressions.py", line 854, in resolve_expression
    return query.resolve_ref(self.name, allow_joins, reuse, summarize)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/paulox/.virtualenvs/generatedfields/lib/python3.12/site-packages/django/db/models/sql/query.py", line 2001, in resolve_ref
    join_info = self.setup_joins(
                ^^^^^^^^^^^^^^^^^
  File "/home/paulox/.virtualenvs/generatedfields/lib/python3.12/site-packages/django/db/models/sql/query.py", line 1854, in setup_joins
    path, final_field, targets, rest = self.names_to_path(
                                       ^^^^^^^^^^^^^^^^^^^
  File "/home/paulox/.virtualenvs/generatedfields/lib/python3.12/site-packages/django/db/models/sql/query.py", line 1754, in names_to_path
    *get_field_names_from_opts(opts),
     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/paulox/.virtualenvs/generatedfields/lib/python3.12/site-packages/django/db/models/sql/query.py", line 63, in get_field_names_from_opts
    (f.name, f.attname) if f.concrete else (f.name,) for f in opts.get_fields()
                                                              ^^^^^^^^^^^^^^^^^
  File "/home/paulox/.virtualenvs/generatedfields/lib/python3.12/site-packages/django/db/models/options.py", line 858, in get_fields
    return self._get_fields(
           ^^^^^^^^^^^^^^^^^
  File "/home/paulox/.virtualenvs/generatedfields/lib/python3.12/site-packages/django/db/models/options.py", line 931, in _get_fields
    all_fields = self._relation_tree
                 ^^^^^^^^^^^^^^^^^^^
  File "/home/paulox/.virtualenvs/generatedfields/lib/python3.12/site-packages/django/utils/functional.py", line 47, in __get__
    res = instance.__dict__[self.name] = self.func(instance)
                                         ^^^^^^^^^^^^^^^^^^^
  File "/home/paulox/.virtualenvs/generatedfields/lib/python3.12/site-packages/django/db/models/options.py", line 831, in _relation_tree
    return self._populate_directed_relation_graph()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/paulox/.virtualenvs/generatedfields/lib/python3.12/site-packages/django/db/models/options.py", line 798, in _populate_directed_relation_graph
    all_models = self.apps.get_models(include_auto_created=True)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/paulox/.virtualenvs/generatedfields/lib/python3.12/site-packages/django/apps/registry.py", line 181, in get_models
    self.check_models_ready()
  File "/home/paulox/.virtualenvs/generatedfields/lib/python3.12/site-packages/django/apps/registry.py", line 143, in check_models_ready
    raise AppRegistryNotReady("Models aren't loaded yet.")
django.core.exceptions.AppRegistryNotReady: Models aren't loaded yet.

Change History (7)

comment:1 by Mariusz Felisiak, 12 months ago

Cc: Lily Foote added
Severity: NormalRelease blocker
Summary: Confusing migration failures in GeneratedFieldMigrations raise AppRegistryNotReady when GeneratedField references incorrect fields.
Triage Stage: UnreviewedAccepted

Thanks for the report. We could add a small check for GeneratedField references, e.g.

  • django/db/models/base.py

    diff --git a/django/db/models/base.py b/django/db/models/base.py
    index 64d54380da..62ce34c4f8 100644
    a b class Model(AltersData, metaclass=ModelBase):  
    16011601                *cls._check_constraints(databases),
    16021602                *cls._check_default_pk(),
    16031603                *cls._check_db_table_comment(databases),
     1604                *cls._check_generated_fields(databases),
    16041605            ]
    16051606
    16061607        return errors
    class Model(AltersData, metaclass=ModelBase):  
    19571958                errors.extend(cls._check_local_fields(fields, "unique_together"))
    19581959            return errors
    19591960
     1961    @classmethod
     1962    def _check_generated_fields(cls, databases):
     1963        errors = []
     1964        for f in cls._meta.local_fields:
     1965            if not f.generated:
     1966                continue
     1967            references = {
     1968                ref[0] for ref in cls._get_expr_references(f.expression)
     1969            }
     1970            errors.extend(cls._check_local_fields(references, f.name))
     1971        return errors
     1972
    19601973    @classmethod
    19611974    def _check_indexes(cls, databases):
    19621975        """Check fields, names, and conditions of indexes."""

Unfortunately, this would still required catching exceptions when resolving GeneratedField expressions.

in reply to:  1 comment:2 by Mariusz Felisiak, 12 months ago

Unfortunately, this would still required catching exceptions when resolving GeneratedField expressions.

Or we can move resolving an expression to the generated_sql() which is not the end of the world:

  • django/db/models/fields/generated.py

    diff --git a/django/db/models/fields/generated.py b/django/db/models/fields/generated.py
    index 9a73b7fe37..95d19582de 100644
    a b class GeneratedField(Field):  
    1313    db_returning = True
    1414
    1515    _query = None
    16     _resolved_expression = None
    1716    output_field = None
    1817
    1918    def __init__(self, *, expression, output_field, db_persist=None, **kwargs):
    class GeneratedField(Field):  
    4847        super().contribute_to_class(*args, **kwargs)
    4948
    5049        self._query = Query(model=self.model, alias_cols=False)
    51         self._resolved_expression = self.expression.resolve_expression(
    52             self._query, allow_joins=False
    53         )
    5450        # Register lookups from the output_field class.
    5551        for lookup_name, lookup in self.output_field.get_class_lookups().items():
    5652            self.register_lookup(lookup, lookup_name=lookup_name)
    class GeneratedField(Field):  
    5955        compiler = connection.ops.compiler("SQLCompiler")(
    6056            self._query, connection=connection, using=None
    6157        )
    62         return compiler.compile(self._resolved_expression)
     58        resolved_expression = self.expression.resolve_expression(
     59            self._query, allow_joins=False
     60        )
     61        return compiler.compile(resolved_expression)
    6362
    6463    def check(self, **kwargs):
    6564        databases = kwargs.get("databases") or []

comment:3 by Mariusz Felisiak, 12 months ago

Owner: changed from nobody to Mariusz Felisiak
Status: newassigned

comment:4 by Mariusz Felisiak, 12 months ago

On reconsideration, system check is not necessary. makemigrations doesn't crash with the proposed diff, and migrate raises:

django.core.exceptions.FieldError: Joined field references are not permitted in this query

which is fine, IMO.

comment:5 by Mariusz Felisiak, 12 months ago

Has patch: set

comment:6 by GitHub <noreply@…>, 12 months ago

Resolution: fixed
Status: assignedclosed

In 101a85a:

Fixed #34985 -- Fixed GeneratedFields.contribute_to_class() crash when apps are not populated.

Thanks Paolo Melchiorre for the report.

Regression in f333e3513e8bdf5ffeb6eeb63021c230082e6f95.

comment:7 by Mariusz Felisiak <felisiak.mariusz@…>, 12 months ago

In 48eebdc:

[5.0.x] Fixed #34985 -- Fixed GeneratedFields.contribute_to_class() crash when apps are not populated.

Thanks Paolo Melchiorre for the report.

Regression in f333e3513e8bdf5ffeb6eeb63021c230082e6f95.
Backport of 101a85a5a06585ba16ecb25860146d034a8a55ec from main

Note: See TracTickets for help on using tickets.
Back to Top