Lookup using F() on a non-existing model field doesn't throw an error

[Vidya Rani D G]

Fetching the value of a non-existing field on a model using F() object does not throw an error.

Instead, it is just giving the FK value of the instance.

Please find below a minimal test case to reproduce this issue.

# lineitem/models.py 

from django.db import models


class Account(models.Model):
    REGULAR = 'R'
    GOLD = 'G'
    PLATINUM = 'P'
    ACCOUNT_TYPE_CHOICES = (
        (REGULAR, 'Regular'),
        (GOLD, 'Gold'),
        (PLATINUM, 'Platinum'),
    )

    account_type = models.CharField(
        max_length=1,
        choices=ACCOUNT_TYPE_CHOICES,
        default=REGULAR,
    )


class Client(models.Model):
    name = models.CharField(max_length=50)
    registered_on = models.DateField()
    account_type = models.ForeignKey(Account, on_delete=models.CASCADE)

$ python manage.py shell

In [1]: from datetime import date, timedelta

In [2]: from models import Client, Account

In [3]: Client.objects.create(
   ...:     name='Jane Doe',
   ...:     account_type=Account.objects.create(account_type=Account.REGULAR),
   ...:     registered_on=date.today() - timedelta(days=36)
   ...: )
   ...: Client.objects.create(
   ...:     name='James Smith',
   ...:     account_type=Account.objects.create(account_type=Account.GOLD),
   ...:     registered_on=date.today() - timedelta(days=5)
   ...: )
   ...: Client.objects.create(
   ...:     name='Jack Black',
   ...:     account_type=Account.objects.create(account_type=Account.PLATINUM),
   ...:     registered_on=date.today() - timedelta(days=10 * 365)
   ...: )
   ...: 
   ...: 
Out[3]: <Client: Client object (6)>

In [4]: from django.db.models import F

In [5]: Client.objects.values(client_name=F('name'), account_name=F('account_type__name'))

Out[5]: <QuerySet [{'client_name': 'Jane Doe', 'account_name': 2}, {'client_name': 'James Smith', 'account_name': 3}, {'client_name': 'Jack Black', 'account_name': 4}, {'client_name': 'Jane Doe', 'account_name': 5}, {'client_name': 'James Smith', 'account_name': 6}, {'client_name': 'Jack Black', 'account_name': 7}]>

In [6]: print(queryset.query)
SELECT "lineitem_client"."name" AS "client_name", "lineitem_client"."account_type_id" AS "account_name" FROM "lineitem_client"

[Matthew Schinckel]

I tried this in Django 1.11, and it raises the expected exception.

[Matthew Schinckel]

Git bisect tells me it's:

2162f0983de0dfe2178531638ce7ea56f54dd4e7 is the first bad commit
commit 2162f0983de0dfe2178531638ce7ea56f54dd4e7
Author: Matthew Wilkes <git@matthewwilkes.name>
Date:   Sun Jun 18 16:53:40 2017 +0100

    Fixed #24747 -- Allowed transforms in QuerySet.order_by() and distinct(*fields).

[Matthew Schinckel]

​https://github.com/django/django/commit/2162f0983de0dfe2178531638ce7ea56f54dd4e7

[Matthew Schinckel]

So, does it think it's a transform?

[Curtis Maloney]

Even if it does, it should surely raise an error? I don't know of a transform called "name", do you?

[Vidya Rani D G]

I tried this in Django 2.0 and it raises an exception.

[Matthew Schinckel]

Replying to Vidya Rani D G:

I tried this in Django 2.0 and it raises an exception.

Okay, that's weird, because the test I wrote passed under 2.0

[Alexander Holmbäck]

Added a test expecting FieldError when F() contains a join that ends with something other than a field or transform (test fails in 2.1a1 and above): ​https://github.com/alexh546/django/tree/ticket_29727

Edit: Patch is now included. I want to make sure that transforms work as expected so I'll look over the current tests and see if I should add some that combine joins and transforms.

[Alexander Holmbäck]

Here's my take

In django.db.models.sql.Query:

setup_joins holds on to FieldException raised by names_to_path so that JoinInfo.transform_function can reraise if it doesn't find a satisfying transform.

It seems like resolve_ref doesn't need the return value from transform_function (unlike add_fields), so it didn't call it, which accidentally led FieldError to never be reraised.

When adding a call to transform_function from resolve_ref, all tests passes. Note how I assume targets to not be empty, that may be bad. Also, I haven't confirmed that there are tests for when joins and transforms are mixed, which could be relevant but also not.

[Carlton Gibson]

Hi Alexander. Can you open ​a PR from your branch to master? Thanks.

[Alexander Holmbäck]

Done: ​PR

[Mariusz Felisiak]

IMO it should be marked as a "Release Blocker", because it is a regression in the 2.1 release.

[Tim Graham <timograham@…>]

In f315d042:

Fixed #29727 -- Made nonexistent joins in F() raise FieldError.

Regression in 2162f0983de0dfe2178531638ce7ea56f54dd4e7.

[Tim Graham <timograham@…>]

In bd5ce059:

[2.1.x] Fixed #29727 -- Made nonexistent joins in F() raise FieldError.

Regression in 2162f0983de0dfe2178531638ce7ea56f54dd4e7.

Backport of f315d0423a09dfe20dd4d4f6a0eb11fc8e45a665 from master