Django

Code

Ticket #7047 (closed: fixed)

Opened 5 months ago

Last modified 4 months ago

filter() with Q spanning ManyToMany relations return wrong results

Reported by: Waldemar Kornewald Assigned to: nobody
Milestone: Component: Database wrapper
Version: queryset-refactor Keywords:
Cc: Triage Stage: Accepted
Has patch: 0 Needs documentation: 0
Needs tests: 0 Patch needs improvement: 0

Description

I'll try to simplify our situation, so the code samples have nothing to do with our actual models. Currently, we use sqlite with Python 2.5 and Django queryset-refactor (trunk has similar problems, though).

We have models similar to these:

class Tag(Model):
    name = CharField(max_length=100)
    hits = ManyToManyField(Article, related_name='tags')

class Category(Model):
    name = CharField(max_length=100)
    hits = ManyToManyField(Article, related_name='categories')

In our DB we have an Article object that has two tags ('taga' and 'tagb') and two categories ('cata' and 'catb').

Then we run the following query (find an Article that has both 'taga' and 'tagb' in the tags and/or categories):

Article.objects.filter(
    Q(categories__name__startswith='taga') \
    | \
    Q(tags__name__startswith='taga') \
    ,
    Q(categories__name__startswith='tagb') \
    | \
    Q(tags__name__startswith='tagb') \
    ).distinct()

The query will not find the article (although it should). Now, if we swap the attribute accessing order (i.e., replace Q(categories__...) | Q(tags__...) with Q(tags__...) | Q(categories__...)) it'll work correctly.

In other words: Only the first attribute (categories__...) will allow for matching multiple times whereas the second attribute (tags__...) only allows for matching once. Note, this actually seems to only depend on the attribute order of the *last* Q group (in the snippet it's the Q pair for 'tagb').

Other example queries that work correctly with the above code snippet:

  • 'cata' and 'catb'
  • 'taga' and 'cata'
  • 'cata' and 'taga' and 'catb'

Attachments

Change History

04/20/08 20:09:24 changed by mtredinnick

  • needs_better_patch changed.
  • needs_tests changed.
  • version changed from SVN to queryset-refactor.
  • keywords deleted.
  • needs_docs changed.
  • stage changed from Unreviewed to Accepted.

Fixed version and keywords, since report is against the branch code.

04/24/08 11:35:03 changed by mtredinnick

Whilst this example does demonstrate that Django isn't generating quite the correct SQL, your sample filter there is expected to return no results. Ignoring the category bits (which obviously won't match anything), it's asking for a tag that has a name that startswith 'taga' and also startswith 'tagb'. That is always the empty set.

If you're wanting -- and I suspect you are -- to filter on tags that start with 'taga' plus (other) tags that start with 'tagb', you need two filter calls. See the db-api.txt documentation, in the section labelled "Spanning multi-valued relationships" (only in queryset-refactor) for an explanation of the difference and why it's necessary.

Leaving the ticket open, since it shows a way to create the incorrect SQL, but just noting that you'll have to change your filter in any case (the correct way doesn't work properly in this example at the moment, either, so that's why it's a bug).

04/24/08 15:50:54 changed by Waldemar Kornewald

Then, I misunderstood the explanation in QuerysetRefactorBranch. I thought that filter().filter() would try to find tags that start with both strings at the same time. But I have the impression that even the filter(..., ...) version isn't really correct because, depending on the order, I am actually able to get the desired results. According to your explanation my query should *always* return an empty set, but it doesn't.

04/25/08 12:08:29 changed by mtredinnick

  • status changed from new to closed.
  • resolution set to fixed.

(In [7462]) queryset-refactor: Fixed some bugs in the multi-valued filtering behaviour introduced in [7317]. It was failing in a couple of different ways on some complex Q() combinations.

Fixed #7047

04/25/08 17:05:35 changed by Waldemar Kornewald

Yay! It works! Thank you *so* much!


Add/Change #7047 (filter() with Q spanning ManyToMany relations return wrong results)




Change Properties
Action