Filter by Count annotated inside Subquery

[MrFus10n]

I want to get authors and annotate minimal count of books in category if it is greater than three.

Book and Author models and are not connected with ForeignKey fields (this is abstract and simplified, there is a reason):

Author(models.Model):
    name = models.CharField(max_length=250)

Book(models.Model):
    author_name = models.CharField(max_length=250)
    book_category = models.CharField(max_length=250)

Here is simplest query I can get to reproduce:

(Author.objects
 .annotate(min_valuable_count=Subquery(
    Book.objects
        .filter(author_name=OuterRef('name'))
        .annotate(cnt=Count('book_category'))
        .filter(cnt__gt=3)
        .order_by('cnt')
        .values('cnt')[:1],
    output_field=models.IntegerField()
)))

And I get an error:

psycopg2.ProgrammingError: missing FROM-clause entry for table "U0"
LINE 1: ... "core_author" GROUP BY "core_author"."id", "U0"."id" ...
                                                       ^

Here is SQL:

SELECT "core_author"."id", "core_author"."name", (
    SELECT COUNT(U0."book_category") AS "cnt" 
    FROM "core_book" U0 WHERE U0."id" = ("core_author"."chat_id") 
    GROUP BY U0."id" HAVING COUNT(U0."book_category") > 3 
    ORDER BY "cnt" ASC  LIMIT 1) 
AS "min_valuable_count" 
FROM "core_author" 
GROUP BY "core_author"."id", "U0"."id"

If I remove line .filter(cnt__gt=3), last GROUP BY disappears and query stops raising error:

SELECT "core_author"."id", "core_author"."name", (
    SELECT COUNT(U0."book_category") AS "cnt" 
    FROM "core_book" U0 WHERE U0."id" = ("core_author"."chat_id") 
    GROUP BY U0."id" HAVING COUNT(U0."book_category") > 3 
    ORDER BY "cnt" ASC  LIMIT 1) 
AS "min_valuable_count" 
FROM "core_author" 

Is there any way to remove GROUP BY in outer query without removing .filter(cnt__gt=3) in subquery?