Code

Opened 4 years ago

Closed 4 years ago

Last modified 3 years ago

#12429 closed (fixed)

RawQuerySet doesn't call ops.convert_values in the backend

Reported by: ikelly Owned by: nobody
Component: Database layer (models, ORM) Version: master
Severity: Keywords: oracle raw
Cc: mboersma Triage Stage: Accepted
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: UI/UX:

Description

Because RawQuerySet doesn't invoke the sql compiler, the ops.convert_values method in the Oracle backend never gets called. As a result, several values come through in the wrong format -- most notably, TextField values get pulled in as a cx_Oracle LOB object rather than as a Unicode string.

The complication with just adding a call to the method is that when using the other backends, it only ever gets called in an aggregation query. Outside of this context, it breaks. The method needs a closer examination to determine when it should be called and exactly what it should be doing.

Also, we need a test case for this. The existing raw_query tests miss it.

Attachments (2)

t12429-r12905.diff (615 bytes) - added by russellm 4 years ago.
Possible fix for Oracle problems in raw_query
t12429-r12905.2.diff (1.1 KB) - added by russellm 4 years ago.
Second attempt at an Oracle fix

Download all attachments as: .zip

Change History (10)

comment:1 Changed 4 years ago by russellm

  • Needs documentation unset
  • Needs tests unset
  • Patch needs improvement unset
  • Triage Stage changed from Unreviewed to Accepted

comment:2 Changed 4 years ago by gav

We don't necessarily need to call the other backends .convert_values directly. We could still potentially do it through the SQLCompiler on the backend so that it functions equivalently to the main QuerySet class. However, post-1.2 it's been discussed on IRC that the base convert_values method does need a review and update to handle cases outside of aggregates more intelligently.

comment:3 Changed 4 years ago by russellm

  • Resolution set to fixed
  • Status changed from new to closed

(In [12904]) Fixed #12429 -- Ensure that raw queries call resolve_columns if the backend defines it. This ensures (as much as possible) that the model values returned by a raw query match that in normal queries. Thanks to Ian Kelly for the report.

comment:4 Changed 4 years ago by kmtracey

  • Resolution fixed deleted
  • Status changed from closed to reopened

r12904 doesn't quite work for Oracle: the raw_query tests now all result in an error due to the fact that RawQuery does not have a high_mark attribute. From the line of code in the traceback (I only included one, they are all the same) I suspect low_mark is going to be a problem too...

No fixtures found.
testAnnotations (modeltests.raw_query.tests.RawQueryTests) ... ERROR
testDBColumnHandler (modeltests.raw_query.tests.RawQueryTests) ... ERROR
testExtraConversions (modeltests.raw_query.tests.RawQueryTests) ... ERROR
testFkeyRawQuery (modeltests.raw_query.tests.RawQueryTests) ... ERROR
testGetItem (modeltests.raw_query.tests.RawQueryTests) ... ERROR
testInvalidQuery (modeltests.raw_query.tests.RawQueryTests) ... ok
testManyToMany (modeltests.raw_query.tests.RawQueryTests) ... ERROR
testMissingFields (modeltests.raw_query.tests.RawQueryTests) ... ERROR
testMissingFieldsWithoutPK (modeltests.raw_query.tests.RawQueryTests) ... ERROR
testMultipleIterations (modeltests.raw_query.tests.RawQueryTests) ... ERROR
testOrderHandler (modeltests.raw_query.tests.RawQueryTests) ... ERROR
testParams (modeltests.raw_query.tests.RawQueryTests) ... ERROR
testRawQueryLazy (modeltests.raw_query.tests.RawQueryTests) ... ERROR
testSimpleRawQuery (modeltests.raw_query.tests.RawQueryTests) ... ERROR
testTranslations (modeltests.raw_query.tests.RawQueryTests) ... ERROR
testWhiteSpaceQuery (modeltests.raw_query.tests.RawQueryTests) ... ERROR
test_inheritance (modeltests.raw_query.tests.RawQueryTests) ... ERROR

======================================================================
ERROR: testAnnotations (modeltests.raw_query.tests.RawQueryTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\u\kmt\django\trunk\tests\modeltests\raw_query\tests.py", line 170, in testAnnotations
    self.assertSuccessfulRawQuery(Author, query, authors, expected_annotations)
  File "C:\u\kmt\django\trunk\tests\modeltests\raw_query\tests.py", line 17, in assertSuccessfulRawQuery
    results = list(model.objects.raw(query, params=params, translations=translations))
  File "C:\u\kmt\django\trunk\django\db\models\query.py", line 1358, in __iter__
    yield self.transform_results(row)
  File "C:\u\kmt\django\trunk\django\db\models\query.py", line 1422, in transform_results
    values = compiler.resolve_columns(values, fields)
  File "C:\u\kmt\django\trunk\django\db\backends\oracle\compiler.py", line 9, in resolve_columns
    if self.query.high_mark is not None or self.query.low_mark:
AttributeError: 'RawQuery' object has no attribute 'high_mark'

Changed 4 years ago by russellm

Possible fix for Oracle problems in raw_query

comment:5 Changed 4 years ago by russellm

I've just uploaded a possible fix for the oracle problems; if anyone with an Oracle setup could take it for a spin, I'd be much obliged.

comment:6 Changed 4 years ago by kmtracey

With the t12429-r12905 patch applied test_invalid_query in the raw_query tests now works:

testInvalidQuery (modeltests.raw_query.tests.RawQueryTests) ... ok

But the other 16 tests generate an error:

======================================================================
ERROR: test_inheritance (modeltests.raw_query.tests.RawQueryTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\u\kmt\django\trunk\tests\modeltests\raw_query\tests.py", line 218, in test_inheritance
    [o.pk for o in FriendlyAuthor.objects.raw(query)], [f.pk]
  File "C:\u\kmt\django\trunk\django\db\models\query.py", line 1358, in __iter__
    yield self.transform_results(row)
  File "C:\u\kmt\django\trunk\django\db\models\query.py", line 1422, in transform_results
    values = compiler.resolve_columns(values, fields)
  File "C:\u\kmt\django\trunk\django\db\backends\oracle\compiler.py", line 17, in resolve_columns
    values.append(self.query.convert_values(value, field, connection=self.connection))
AttributeError: 'RawQuery' object has no attribute 'convert_values'

Changed 4 years ago by russellm

Second attempt at an Oracle fix

comment:7 Changed 4 years ago by russellm

  • Resolution set to fixed
  • Status changed from reopened to closed

(In [12907]) Fixed #12429 -- Modified RawQuery to provide some facilities required by Oracle. Thanks to Karen Tracey for the testing help.

comment:8 Changed 3 years ago by jacob

  • milestone 1.2 deleted

Milestone 1.2 deleted

Add Comment

Modify Ticket

Change Properties
<Author field>
Action
as closed
as The resolution will be set. Next status will be 'closed'
The resolution will be deleted. Next status will be 'new'
Author


E-mail address and user name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.