Opened 2 months ago

Closed 2 months ago

Last modified 2 months ago

#30769 closed Bug (fixed)

Access JSON key crashes when using annotated field from subquery.

Reported by: Tim Kleinschmidt Owned by: Simon Charette
Component: Database layer (models, ORM) Version: 1.11
Severity: Release blocker Keywords:
Cc: Simon Charette Triage Stage: Accepted
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description (last modified by Tim Kleinschmidt)


TypeError: can only concatenate tuple (not "list") to tuple
  File "django/core/handlers/exception.py", line 41, in inner
    response = get_response(request)
  File "django/core/handlers/base.py", line 187, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "django/core/handlers/base.py", line 185, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "django/contrib/auth/decorators.py", line 23, in _wrapped_view
    return view_func(request, *args, **kwargs)
  File "apps/dashboard/companies/employee_list.py", line 395, in company_employee_view
    RequestConfig(request, paginate={'per_page': 25}).configure(table)
  File "django_tables2/config.py", line 59, in configure
    table.paginate(**kwargs)
  File "django_tables2/tables.py", line 526, in paginate
    self.page = self.paginator.page(page)
  File "django/core/paginator.py", line 57, in page
    number = self.validate_number(number)
  File "django/core/paginator.py", line 46, in validate_number
    if number > self.num_pages:
  File "django/utils/functional.py", line 35, in __get__
    res = instance.__dict__[self.name] = self.func(instance)
  File "django/core/paginator.py", line 91, in num_pages
    if self.count == 0 and not self.allow_empty_first_page:
  File "django/utils/functional.py", line 35, in __get__
    res = instance.__dict__[self.name] = self.func(instance)
  File "django/core/paginator.py", line 84, in count
    return len(self.object_list)
  File "django_tables2/rows.py", line 341, in __len__
    length = len(self.data)
  File "django_tables2/data.py", line 140, in __len__
    self._length = self.data.count()
  File "django/db/models/query.py", line 364, in count
    return self.query.get_count(using=self.db)
  File "django/db/models/sql/query.py", line 499, in get_count
    number = obj.get_aggregation(using, ['__count'])['__count']
  File "django/db/models/sql/query.py", line 463, in get_aggregation
    outer_query.add_subquery(inner_query, using)
  File "django/db/models/sql/subqueries.py", line 209, in add_subquery
    self.subquery, self.sub_params = query.get_compiler(using).as_sql(with_col_aliases=True)
  File "django/db/models/sql/compiler.py", line 441, in as_sql
    where, w_params = self.compile(self.where) if self.where is not None else ("", [])
  File "django/db/models/sql/compiler.py", line 373, in compile
    sql, params = node.as_sql(self, self.connection)
  File "django/db/models/sql/where.py", line 79, in as_sql
    sql, params = compiler.compile(child)
  File "django/db/models/sql/compiler.py", line 373, in compile
    sql, params = node.as_sql(self, self.connection)
  File "django/db/models/sql/where.py", line 79, in as_sql
    sql, params = compiler.compile(child)
  File "django/db/models/sql/compiler.py", line 373, in compile
    sql, params = node.as_sql(self, self.connection)
  File "django/db/models/lookups.py", line 169, in as_sql
    lhs_sql, params = self.process_lhs(compiler, connection)
  File "django/db/models/lookups.py", line 160, in process_lhs
    compiler, connection, lhs)
  File "django/db/models/lookups.py", line 82, in process_lhs
    return compiler.compile(lhs)
  File "django/db/models/sql/compiler.py", line 373, in compile
    sql, params = node.as_sql(self, self.connection)
  File "django/contrib/postgres/fields/jsonb.py", line 110, in as_sql
    return '(%s %s %%s)' % (lhs, self.operator), params + [lookup]
AttributeError: 'BoundRows' object has no attribute 'count'
  File "django/core/paginator.py", line 79, in count
    return self.object_list.count()

Code which leads to the error

# Base Query Construct
auth_query = UserAuth.objects.filter(user_id=OuterRef('user_id')).order_by(
    '-received_at')[:1]
base_employees = (
    base_employees.annotate(auth_received_at=Subquery(auth_query.values('received_at')),
                            auth_response=Subquery(auth_query.values('response'))))


# Q object building
q = Q()
if IS_AUTH_REJECTED in auth_status:
    q |= Q(auth_response__REJECTED='N')

base_employees.filter(q)

Workaround

# base query construct
auth_query = UserAuth.objects.annotate(
    has_access=KeyTextTransform('REJECTED', 'response')
).filter(user_id=OuterRef('user_id')).order_by('-received_at')[:1]

base_employees = base_employees.annotate(
    auth_received_at=Subquery(auth_query.values('received_at')),
    has_access=Subquery(auth_query.values('has_access'), output_field=CharField()))

# q object creation
q = Q()
if IS_AUTH_REJECTED in auth_status:
    q |= Q(has_access='N')

base_employees.filter(q)

Attachments (1)

test-30769.diff (1.2 KB) - added by felixxm 2 months ago.
Regression test.

Download all attachments as: .zip

Change History (10)

comment:1 Changed 2 months ago by Tim Kleinschmidt

Description: modified (diff)

comment:2 Changed 2 months ago by felixxm

Cc: Simon Charette added
Component: UncategorizedDatabase layer (models, ORM)
Severity: NormalRelease blocker
Summary: Access Json Key over annotated field in subqueryAccess JSON key crashes when using annotated field from subquery.
Triage Stage: UnreviewedAccepted
Type: UncategorizedBug

Thank for this report.

Reproduced at 9a2a12d415e6d224797b0779c5af03deb8c45b0a.
Regression in 7deeabc7c7526786df6894429ce89a9c4b614086.

Changed 2 months ago by felixxm

Attachment: test-30769.diff added

Regression test.

comment:3 Changed 2 months ago by Simon Charette

Owner: changed from nobody to Simon Charette
Status: newassigned

comment:4 Changed 2 months ago by Simon Charette

Has patch: set

comment:5 Changed 2 months ago by Mariusz Felisiak <felisiak.mariusz@…>

Resolution: fixed
Status: assignedclosed

In 6c3dfba:

Fixed #30769 -- Fixed a crash when filtering against a subquery JSON/HStoreField annotation.

This was a regression introduced by 7deeabc7c7526786df6894429ce89a9c4b614086
to address CVE-2019-14234.

Thanks Tim Kleinschmidt for the report and Mariusz for the tests.

comment:6 Changed 2 months ago by Mariusz Felisiak <felisiak.mariusz@…>

In 574154e:

[3.0.x] Fixed #30769 -- Fixed a crash when filtering against a subquery JSON/HStoreField annotation.

This was a regression introduced by 7deeabc7c7526786df6894429ce89a9c4b614086
to address CVE-2019-14234.

Thanks Tim Kleinschmidt for the report and Mariusz for the tests.

Backport of 6c3dfba89215fc56fc27ef61829a6fff88be4abb from master

comment:7 Changed 2 months ago by Mariusz Felisiak <felisiak.mariusz@…>

In 7806e45:

[2.2.x] Fixed #30769 -- Fixed a crash when filtering against a subquery JSON/HStoreField annotation.

This was a regression introduced by 7deeabc7c7526786df6894429ce89a9c4b614086
to address CVE-2019-14234.

Thanks Tim Kleinschmidt for the report and Mariusz for the tests.

Backport of 6c3dfba89215fc56fc27ef61829a6fff88be4abb from master

comment:8 Changed 2 months ago by Mariusz Felisiak <felisiak.mariusz@…>

In db181f4:

[2.1.x] Fixed #30769 -- Fixed a crash when filtering against a subquery JSON/HStoreField annotation.

This was a regression introduced by 7deeabc7c7526786df6894429ce89a9c4b614086
to address CVE-2019-14234.

Thanks Tim Kleinschmidt for the report and Mariusz for the tests.

Backport of 6c3dfba89215fc56fc27ef61829a6fff88be4abb from master.

comment:9 Changed 2 months ago by Mariusz Felisiak <felisiak.mariusz@…>

In fd39390:

[1.11.x] Fixed #30769 -- Fixed a crash when filtering against a subquery JSON/HStoreField annotation.

This was a regression introduced by 7deeabc7c7526786df6894429ce89a9c4b614086
to address CVE-2019-14234.

Thanks Tim Kleinschmidt for the report and Mariusz for the tests.

Backport of 6c3dfba89215fc56fc27ef61829a6fff88be4abb from master.

Note: See TracTickets for help on using tickets.
Back to Top