Code

Opened 14 months ago

Last modified 11 months ago

#20414 new Bug

Handling of numbers under oracle is slow

Reported by: shai Owned by: shai
Component: Database layer (models, ORM) Version: master
Severity: Normal Keywords: oracle performance
Cc: akaariai, ikelly Triage Stage: Accepted
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: yes
Easy pickings: no UI/UX: no

Description

With current code, for all number columns, the Oracle backend asks cx_Oracle to return the numbers as strings, and then transforms them back to numbers on the way out. All this is done because Oracle supports higher precision (and a wider range of integers) than the Python float type; some values, if they are to be reported accurately, must be returned as decimal.Decimals, and cx_Oracle doesn't apparently do that on its own.

With cx_Oracle 5, though, it is possible to use a much better approach, on a per-column (rather than per-row) basis, selecting the right output type when there is enough information to do so, and making most of the decisions once-per-query instead of once-per-row.

An original patch for this was written by Ian Kelly; I improved it and made a pull-request in the stretch towards 1.5, but there was no ticket for it and so it was never merged. I hope we can finally merge it into 1.6.

There was a little saga of discussions around this (mostly Anssi, Ian and myself, https://groups.google.com/d/topic/django-developers/4BNkJyGez9A/discussion). Some other issues are also mentioned there. But this is the main one.

So now there is a ticket. Things changed, and the pull-request needed updating. The new one is at https://github.com/django/django/pull/1071.

Attachments (0)

Change History (3)

comment:1 Changed 14 months ago by russellm

  • Needs documentation unset
  • Needs tests unset
  • Patch needs improvement unset
  • Triage Stage changed from Unreviewed to Accepted

comment:2 Changed 11 months ago by timo

  • Patch needs improvement set

Newest pull request appears to be here. It at least needs to be updated to reflect the fact that it won't be added in 1.6 as currently documented.

comment:3 Changed 11 months ago by shai

Well, I seemed to have botched that branch -- I need to fix it. Until then, I'll close the PR.

Anyway, later tests I ran showed this to be a lot less promising than I had thought. The reason is that cx_Oracle, internally, does this return-number-as-string-and-transform-to-Python-int dance whenever the database column can hold more than 9 digits; Django's Oracle backend uses number(11) for ints. I'll revisit this when I finally get to meddling with cx_Oracle internals (this is needed mostly for Python 3 support, which is currently quite completely broken).

Add Comment

Modify Ticket

Change Properties
<Author field>
Action
as new
The owner will be changed from shai to anonymous. Next status will be 'assigned'
as The resolution will be set. Next status will be 'closed'
Author


E-mail address and user name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.