Opened 3 years ago

Closed 3 years ago

Last modified 2 years ago

#19606 closed Bug (fixed)

Oracle unicode insert fails.

Reported by: jtiai Owned by: nobody
Component: Database layer (models, ORM) Version: 1.4
Severity: Normal Keywords: oracle orm unicode
Cc: Triage Stage: Accepted
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no


smart_str (which detection selects) does not work.

using force_unicode works (except for connection string).

Depending on what OCI client or instant client 11.2 is used when compiling cx_Oracle causes variation. doesn't work with smart_str while 11.2 does work.

Both can take plain unicode (u'<some unicode stuff here>') when using just cx_Oracle directly without any problems.


If I add manually some unicode to database Django can read it without any problems.

Change History (9)

comment:1 Changed 3 years ago by anonymous

  • Needs documentation unset
  • Needs tests unset
  • Patch needs improvement unset

comment:2 Changed 3 years ago by akaariai

  • Triage Stage changed from Unreviewed to Accepted

I haven't haven't been able to verify this (no here). The report seems well researched in the django-users thread so accepted based on that.

comment:3 Changed 3 years ago by jtiai

Problem is that Django makes incorrect detection what to string encoding system to use:

I did also some source code diving to make sure how this works:

In version 5.0.x cx_Oracle.UNICODE is only defined if code was compiled without flag WITH_UNICODE.

In version 5.1+ cx_Oracle.UNICODE is always defined if Python major version compiled against is < 3.

According to changelog of cx_Oracle:

Changes from 5.0.4 to 5.1

1) Remove support for UNICODE mode and permit Unicode to be passed through in

everywhere a string may be passed in. This means that strings will be
passed through to Oracle using the value of the NLS_LANG environment
variable in Python 3.x as well. Doing this eliminated a bunch of problems
that were discovered by using UNICODE mode and also removed an unnecessary
restriction in Python 2.x that Unicode could not be used in connect strings
or SQL statements, for example.

So behaviour should be: If cx_Oracle version is greater or equal to 5.1 real unicode should be used always. With pre-5.1 current behaviour is sufficient.

comment:4 Changed 3 years ago by Anssi Kääriäinen <akaariai@…>

  • Resolution set to fixed
  • Status changed from new to closed

In d194f290571f7e9dda7d2fd7a6f2b171120f2f14:

Fixed #19606 -- Adjusted cx_Oracle unicode detection.

comment:5 Changed 3 years ago by Anssi Kääriäinen <akaariai@…>

In 457290326fa38297467922d827c630ae2026e88b:

[1.5.x] Fixed #19606 -- Adjusted cx_Oracle unicode detection.

Backpatch of d194f290571f7e9dda7d2fd7a6f2b171120f2f14

comment:6 Changed 3 years ago by akaariai

I felt very uncomfortable about backporting this as I can't actually test the combo this is supposed to fix - that is, cx_oracle 5.1 + instantclient 10.2 as that wont compile on my machine. Based on the patch, cx_oracle 5.0 users should no get any change and I have tested 5.1 + instantclient 11.

I think this falls into the category of patches we can still backpatch to 1.5 at this stage - the combination of cx_oracle 5.1 + instantclient 10.2 is supported, but it doesn't work with unicode values currently.

So, after using too much time trying to tests this, I did backpatch this on the grounds that instanclient 11 works, cx_oracle 5.0 works as always, and the research done by jtiai shows that this is the right thing to do for 5.0 + instantclient 10.

comment:7 Changed 3 years ago by jtiai

According to my research it's all about version of cx_Oracle 5.0.x series does behave have slightly different definition for cx_Oracle.UNICODE than 5.1.x

It has nothing to do with OCI version linked against even I though that was the reason. But I think 1.5 is sufficient at this point.

comment:8 Changed 3 years ago by Anssi Kääriäinen <akaariai@…>

In 457290326fa38297467922d827c630ae2026e88b:

[1.5.x] Fixed #19606 -- Adjusted cx_Oracle unicode detection.

Backpatch of d194f290571f7e9dda7d2fd7a6f2b171120f2f14

comment:9 Changed 2 years ago by anonymous

#20292 is possibly related, at least reverting this change fixes that bug.

Note: See TracTickets for help on using tickets.
Back to Top