Opened 4 years ago

Closed 2 years ago

#19399 closed Cleanup/optimization (fixed)

Do not recreate RelatedObject instances

Reported by: Krzysztof Jurewicz Owned by: Krzysztof Jurewicz
Component: Database layer (models, ORM) Version: master
Severity: Normal Keywords:
Cc: Krzysztof Jurewicz, sun.void@… Triage Stage: Accepted
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no


When generating accessor names for fields in django/core/management/ , new RelatedObject instances are created. Using existing ones retrieved from fields will make the code cleaner and more flexible, allowing some modifications of RelatedObject instances used by third party apps.

Change History (18)

comment:1 Changed 4 years ago by Krzysztof Jurewicz

Has patch: set

I've created a pull request:

comment:2 Changed 4 years ago by Anssi Kääriäinen

Triage Stage: UnreviewedAccepted

Seems valid - I assume all tests pass?

comment:3 Changed 4 years ago by Krzysztof Jurewicz


Ran 4915 tests in 431.403s

OK (skipped=135, expected failures=7)

comment:4 Changed 4 years ago by Ramiro Morales

Needs tests: set

comment:5 Changed 4 years ago by Krzysztof Jurewicz

Component: Core (Management commands)Database layer (models, ORM)
Summary: Do not recreate RelatedObject instances in validationDo not recreate RelatedObject instances

I do agree that some tests are worth adding for this ticket. Meanwhile, I've located other places in code where RelatedObject was being recreated and made a similar change to them. This caused test_inline_formsets_with_multi_table_inheritance from model_tests.model_formset.ModelFormSetTest to fail, but since the reason was only the change of generated formset's prefix (which shouldn't be harmful by itself), I've resolved the problem by changing the test.

comment:6 Changed 4 years ago by Krzysztof Jurewicz

Needs tests: unset

I've added tests and updated the pull request (commit hash is now c2028f6bfcde2db5681b).

Last edited 4 years ago by Krzysztof Jurewicz (previous) (diff)

comment:7 Changed 4 years ago by Nick Sandford

Triage Stage: AcceptedReady for checkin

All tests pass on sqlite. Patch looks good, marking RFC.

comment:8 Changed 4 years ago by Anssi Kääriäinen

I took a look at the patch. It looks otherwise good, but I can't see why the change in the model_formsets is correct. Not saying it isn't, I just can't see why it is. On a quick look it seems we are really changing semantics here - the form uses the plain fk to Book, while the user asked for AlternateBook.

It seems this change has (small) potential for breaking existing code, so there should at least be a good reason to do so. My initial feeling is to not commit this part of the code.

comment:9 Changed 4 years ago by Anssi Kääriäinen

Actually, if you have two subclasses of a Book and you try to use them in inlineformset_factory in the same view, wouldn't the ID names conflict when they didn't before? On this ground I think we need to not do the form changes.

comment:10 Changed 4 years ago by Krzysztof Jurewicz

The user asks for AlternateBook when creating inline formset, but foreign key is created on Book. Author class doesn't know anything about AlternateBook class being tied to it. So if we ask what will happen if we have two subclasses of Book with inline formsets, then this situation is roughly equivalent to creating two inline formsets for Book class, and similar problems will occur. If there is really separate logic tied to AlternateBook, maybe separate foreign key should be created, but these are just my loose thoughts, not strict reasoning.

If we however decide to preserve AlternateBook formset prefixes, I will argue to change the method of generating them. Current implementation doesn't use RelatedObject from AlternateBook, which was created for Book class and returns 'book-' prefixes. Instead of it another RelatedObject is created to obtain different prefix, but the original one remains inside AlternateBook class. This is tricky and may be misleading — if we already have one RelatedObject inside AlternateBook class, then we should stick to it.

I've updated the pull request to fix a typo reported in comment. Commit hash is now 6cfb8a8a506b2c.

comment:11 Changed 4 years ago by Anssi Kääriäinen <akaariai@…>

In 3647c0a49a2f4535b8a9aba40e662743e4d53e76:

Avoided unnecessary recreation of RelatedObjects

Refs #19399. Thanks to Track alias KJ for the patch.

comment:12 Changed 4 years ago by Anssi Kääriäinen

Easy pickings: unset
Has patch: unset
Triage Stage: Ready for checkinAccepted

I committed the validation and caching changes, without the tests.

Personally, I don't think the tests add that much value. They test 3 specific places for not creating RelatedObject instances, but we have still the rest of codebase uncovered. It is impossible to test that RelatedObjects aren't recreated anywhere in the codebase by testing specific places for that. In addition, if we ever decide to alter the underlying implementations, the tests would be noise errors. (I am currently wondering why we need both field.rel and field.related. Seems strange...).

I also removed the self.rel_name line from the forms/ It is dead-code.

I don't feel comfortable doing the alternatebook name change. It *will* break existing public-API using code, at least tests, as we change what values can be passed in the request.POST. Those using public APIs are more important than those who want to hack with the internals.

So, leaving this open if you want to tackle the AlternateBook issue. It doesn't seem to be an easy one to tackle.

comment:13 Changed 4 years ago by anonymous

Commit 3647c0a49a2f4535b8a9aba40e662743e4d53e76 broke django-taggit with the following traceback:

  File ".../django/django/core/management/", line 277, in validate
    num_errors = get_validation_errors(s, app)
  File ".../django/django/core/management/", line 169, in get_validation_errors
    for r in rel_opts.get_all_related_many_to_many_objects():
  File ".../django/django/db/models/", line 437, in get_all_related_many_to_many_objects
    cache = self._fill_related_many_to_many_cache()
  File ".../django/django/db/models/", line 470, in _fill_related_many_to_many_cache
    cache[f.related] = None
AttributeError: 'TaggableManager' object has no attribute 'related'

So, it seems that it was possible for 3rd-party fields which emulate ManyToManyField to have .rel property, but don't have .related property

comment:14 Changed 4 years ago by Anssi Kääriäinen

I will still try to investigate the related fields API and see if it was just accidental that django-taggit worked before. Still, my gut feeling is to revert the commit - the idea for the commit was to make life easier for 3rd party apps, yet we break other 3rd party apps.

comment:15 Changed 4 years ago by Anssi Kääriäinen

And, after some digging, I think the right resolution is to leave the commit in, and django-taggit should include this fix into TaggableManager:

     def post_through_setup(self, cls):
+        self.related = RelatedObject(cls, self.model, self)

I am very confused about the whole related field setup. We have these three things:

   - RelatedField (which isn't a Field subclass)
   - RelatedObject (telling us what the relation is) - found from field.related
   - ManyToOneRel (as far as I can see, telling us what the relation is) - found from field.rel

At least the last two seem like they could be merged. Maybe we could get totally rid of the last two. OTOH this is a change that will break 3rd party apps for no other reason than code clarity, so I don't feel like doing anything about this. Still, if somebody knows why the setup is like this a comment wouldn't hurt...

If it seems the commit for this ticket breaks more stuff, I am willing to consider reverting it. For now, it seems a relation field should have both field.rel and field.related.

comment:16 Changed 4 years ago by Alex Gaynor

For what it's worth django-taggit is definitely using non-public APIs, so don't feel obligated to fix it.

comment:17 Changed 4 years ago by Alexey Boriskin

Cc: sun.void@… added

comment:18 Changed 2 years ago by Tim Graham

Resolution: fixed
Status: newclosed

As there hasn't been any followup from the reporter regarding the "AlternateBook issue" described in comment 12, I'm going to close this for now. Please open a new ticket if any issues remain.

Note: See TracTickets for help on using tickets.
Back to Top