Add ModelAdmin.autocomplete_fields support for ForeignKeys that use to_field

[Jonathan Nye]

Hi

I have encountered an issue where I have specified a 'to_field' in my foreign key and the autocomplete widget tries to enter the primary key of the related model instead. This means I cannot submit the form and I get the "Select a valid choice. That choice is not one of the available choices." error.

In the AutocompleteJsonView:

def get(self, request, *args, **kwargs):
        """
        Return a JsonResponse with search results of the form:
        {
            results: [{id: "123" text: "foo"}],
            pagination: {more: true}
        }
       ....
       ....
       return JsonResponse({
            'results': [
                {'id': str(obj.pk), 'text': str(obj)}
                for obj in context['object_list']
            ],
            'pagination': {'more': context['page_obj'].has_next()},
        })

Is there a way to replace the id manually when saving the form?

Thanks.

[Tim Graham]

I think the correct solution is to modify AutocompleteJsonView to return the to_field value instead of the primary key. To prevent improper data leakage, ModelAdmin.to_field_allowed() should be called.

[Johannes Maron]

Hi Jonathan,

very good catch. This is certainly a bug. Custom to_fields should be supported.

Do you want to fix this yourself or should I?

Best
-Joe

[Jonathan Nye]

Replying to Johannes Hoppe:

Hi Jonathan,

very good catch. This is certainly a bug. Custom to_fields should be supported.

Do you want to fix this yourself or should I?

Best
-Joe

Hi Joe,

I can give it a shot. I haven't contributed to Django before so I would appreciate some pointers to get me looking in the right direction.

What would need to be updated?

Regards,
Jonathan

[Johannes Maron]

Hey Jonathan,

well that sounds like the perfect opportunity to get into contributing to Django.

Of course the is a contributing guide, but it long. I would recommend to read at least the section for new contributors.
​https://docs.djangoproject.com/en/dev/internals/contributing/

Anyhow, this is how I would go about this:
Write your test first. You should write a test, that exploits the bug you found and fails. You will need to add one at the end anyways, if you start with it actually fixing the issue will become a lot easier. You might want to add the test here tests.admin_views.test_autocomplete_view.AutocompleteJsonViewTests.

Once that is out of the way there is this line in django.contrib.admin.views.autocomplete.AutocompleteJsonView#get. It returns a JsonResponse that includes the obj.pk at some point. What it should return is the value of the to_field. This is where things might become tricky. Since there is only once view per model and probably multiple other models having foreign keys to that one, you will need to get the to_field from the relation or pass it as a GET parameter. In the latter case you should definitely validate the field to avoid leakage as Tim said earlier.

Anyhow, just get started and let me know if you hit an obstacle and need help :)

Best
-Joe

[Basu Dubey]

I wish to take this up.

[Johannes Maron]

Cool, let me know when you have patch. I am happy to review it :)

[Kirill Fries-Raevskiy]

Is there any progress?

[Constantino Schillebeeckx]

I've put together a ​bug fix for this.

Note I've been able to update all unit tests and create some new ones, however I haven't been able to leverage to_field_allowed to prevent data leaks. I've tried to implement it (see ​here), however I can't get it to play nicely with the unit tests. When uncommented, the id_field isn't properly being considered as to_field_allowed. I'm not familiar with this function, so could use some help troubleshooting.

[Johannes Maron]

Replying to Constantino Schillebeeckx:

I've put together a ​bug fix for this.

Note I've been able to update all unit tests and create some new ones, however I haven't been able to leverage to_field_allowed to prevent data leaks. I've tried to implement it (see ​here), however I can't get it to play nicely with the unit tests. When uncommented, the id_field isn't properly being considered as to_field_allowed. I'm not familiar with this function, so could use some help troubleshooting.

Hey there! Thanks for the work. It is a little hard to review since you only posted a branch and I don't know what commits to review. Could you maybe assign yourself and open up a pull-request to the Django repo? I will gladly do the first review.

Best
-Joe

[Constantino Schillebeeckx]

Replying to Johannes Hoppe:

Hey there! Thanks for the work. It is a little hard to review since you only posted a branch and I don't know what commits to review. Could you maybe assign yourself and open up a pull-request to the Django repo? I will gladly do the first review.

All set: ​https://github.com/django/django/pull/10517

[Johannes Maron]

This has been inactive and I stumbled upon a solution while fixing another issue, I am assigning this to me.

[Tyler Schwartz]

This work went stale, so I re-forked, upmerged from master, and resolved the merge conflict and isort/flake8 issues. This new PR should be ready for review/acceptance.

​New PR

[Johannes Maron]

Hi schwtyl,

thanks for your commitment. I must admit, I did push this issue aside a bit in favor of some other patches I have been working on.

Sadly, you did base your work on GH@10494 not on GH@11026, therefore my critique stays the same. It will only solve this issue but make it harder to solve #29138 which is equally important.

Might I also suggest next time, to comment on the ticket first, before fixing someone else's patch. I might save you some trouble.

Best
-Joe

[Tyler Schwartz]

Joe,

Constantino developed this patch for the project that I succeeded him on, so I'm hardly a random bystander. Everytime our project builds for the past year, it pulls it from this GitHub commit, and I'd love to see that eventually go back to using an official Django release again. I'm not sure how I'd know about 11026, since you didn't link it here. I don't quite know that you mean, because this issue *is* #29138 - perhaps you mean #29010? The patch fixes this issue - can you provide feedback as to what can be done for a fix to be accepted? What specifically *needs improvement*?

Thanks,

Tyler

[Johannes Maron]

Hey Tyler,

thanks for your swift response. I see, I wasn't ware of your background, but also didn't mean to offend you. I my comment really was only trying to save you some effort. I know first hand that the Django development process can be lengthy and demotivating.

Anyhow, I see where you are coming from, and yes, you shouldn't absolutely use a mainstream Django version, to avoid security vulnerabilities.
Might I suggest to use django-select2 until then? I developed and maintain that package, it was the blue print for Django's autocomplete field implementation, which I implemented too. It does work slightly different, but it will at least allow you to jump to a stable Django release.

Regarding the patch: As the initial developer of this feature I have an elevated interest to solve all bugs, not just this one. That aside – IMHO – the best solution actually happens to fix both problems.

I will do a review of your PR, to give you a bit better feeling of the work that would need to be done on the PR to fix only this issue. I'll let you decide, if you still think it's worth the effort given that all changes would need to be revered with the fix for #29138.

Best
-Joe

[Johannes Maron]

Scratch everything I said. I had another look at #29138. It will require a lot of work and probably some mailing list discussion. Therefore, I do believe it's worth the effort to fix them separately even if it involves revering some parts later.

I already did a first review of your patch. I'll give it another go tomorrow. I believe we can push this over the finish line.

[Tyler Schwartz]

Thanks Joe!

I appreciate the background you were able to fill in, and I wasn't aware that you were so involved in the implementation of Django Admin's autocomplete feature. I am new to Django development, though I am interested in helping take this forward if I can be of help, even if it takes a while. I don't have any emotional attachment to this current patch, other than having used it in production for this past year (yikes!) - so if you think a different direction is preferable, I won't be offended. I'll take a look at your feedback shortly.

Regardless, you make a good point - I am working to transition our code back to a stable Django release (I inherited this setup when I started earlier this year).

-Tyler

[Manel Clos]

We have been using this in Django 2.1.5. As the to_field contains a code, it is a valid integer, and no error is produced. This issue should be marked as DATA LOSS and be urgently addressed.

[Johannes Maron]

Since we have a working patch, that's been reviewed a bunch, I'm snatching this ticket.

[GitHub <noreply@…>]

In 3071660a:

Fixed #29010, Fixed #29138 -- Added limit_choices_to and to_field support to autocomplete fields.

• Fixed #29010 -- Added limit_choices_to support to autocomplete fields.
• Fixed #29138 -- Allowed autocomplete fields to target a custom to_field rather than the PK.

[Carlton Gibson <carlton.gibson@…>]

In 0b120f5:

Refs #29138 -- Added extra tests for autocomplete FK to_field variations.

[007]

AutocompleteMixin is incompatible in django 3.1 and 3.2. and there is no tip like RemovedInDjango32Warning in django 3.1. It's not user friendly.

[Johannes Maron]

Hi 007,

Thank you for reaching out. I understand your frustration. However, the AutocompleteMixin was never documented, which means it is not considered a public API. Private APIs are not subject to deprecation. That is the risk you take, when using a private API.

All that being said, is there anything I could be of help with? If you have a 3rd party package, that needs assistance, I would be delighted to help.

Best,
Joe

[007]

Replying to Johannes Maron:

Hi 007,

Thank you for reaching out. I understand your frustration. However, the AutocompleteMixin was never documented, which means it is not considered a public API. Private APIs are not subject to deprecation. That is the risk you take, when using a private API.

All that being said, is there anything I could be of help with? If you have a 3rd party package, that needs assistance, I would be delighted to help.

Best,
Joe

In my package to achieve a similar function of this submission. I'm going to try to re implement it in the way of 3.2, and compatible with earlier versions

​https://github.com/007gzs/django-cool/blob/f13d52b571eb77c02b062523bc1d79cdeed31b5a/cool/admin/widgets.py#L184-L234

​https://github.com/007gzs/django-cool/blob/f13d52b571eb77c02b062523bc1d79cdeed31b5a/cool/admin/views.py#L14-L76

[007]

I have change my package in ​https://github.com/007gzs/django-cool/commit/63b0ea6021ea50ad40a2d5d0af6ae9c6cbcdab0f

[Johannes Maron]

I am happy you found a solution. And this should be the last big change on this feature for a while.